Match tokens to words in a dictionary then creates new LexItems for the token lemma and/or synonyms if configured.


Operates On:  Lexical Items with TOKEN and possibly other flags as specified below.

This lemmatization does not use rules.


Generic Configuration Parameters

  • boundaryFlags ( type=string array | optional ) - List of vertex flags that indicate the beginning and end of a text block.
    Tokens to process must be inside two vertices marked with this flag (e.g ["TEXT_BLOCK_SPLIT"])
  • skipFlags ( type=string array | optional ) - Flags to be skipped by this stage.
    Tokens marked with this flag will be ignored by this stage, and no processing will be performed.
  • requiredFlags ( type=string array | optional ) - Lex items flags required by every token to be processed.
    Tokens need to have all of the specified flags in order to be processed.
  • atLeastOneFlag ( type=string array | optional ) - Lex items flags needed by every token to be processed.
    Tokens will need at least one of the flags specified in this array.
  • confidenceAdjustment ( type=double | default=1 | required ) - Adjustment factor to apply to the confidence value of 0.0 to 2.0 from (Applies for every pattern match).
    • 0.0 to < 1.0  decreases confidence value
    • 1.0 confidence value remains the same
    • > 1.0 to  2.0 increases confidence value
  • debug ( type=boolean | default=false | optional ) - Enable all debug log functionality for the stage, if any.
  • enable ( type=boolean | default=true | optional ) - Indicates if the current stage should be consider for the Pipeline Manager
    • Only applies for automatic pipeline building

Configuration Parameters

  • dictionary ( type=string | optional ) - The resource containing the list of words and relationships
    • If no dictionary is provided, a built-in dictionary will be used, based on the languageISO3.
  • include ( type=string array | optional ) - A list of the relationships to include
  • exclude ( type=string array | optional ) - A list of the relationships to exclude
  • languageISO3 ( type=string | optional ) - The language the lemmatize stage should use. The value needs to be one of the ISO 3 letter language codes
    • By default, English is always used unless configured otherwise. At the moment, only English (ENG) and Spanish (SPA) are available.

A default dictionary is available in English. Spanish is supported when parameter languageISO3 is configured properly.

$action.getHelper().renderConfluenceMacro("$codeS$body$codeE")

Example Output

$action.getHelper().renderConfluenceMacro("$codeS$body$codeE")

Output Flags

Lex-Item Flags:

  • LEMMATIZE- All words retrieved will be marked as LEMMATIZE.
  • ALL_LOWER_CASE - All of the characters in the token are lower-case characters.
  • TOKEN - This stage creates a new token.

Vertex Flags:

No vertices are created in this stage

Resource Data

The resource data will be a json file with an array of words in a field named words. This is when the 'dictionary' parameter is used. $action.getHelper().renderConfluenceMacro("$codeS$body$codeE")

When the 'dictionary' parameter is not used, an embedded Wiktionary file will be used. This file is formatted as a 1 entry json per line:

Wiktionary file format
{"confidence":0,"rel":["syn"],"from":"japonés","to":"nipón"}
{"confidence":0,"rel":["syn"],"from":"alemán","to":"germano"}
{"confidence":0,"rel":["syn"],"from":"alemán","to":"tedesco"}
{"confidence":0,"rel":["syn"],"from":"alemán","to":"teutón"}
{"confidence":0,"rel":["syn"],"from":"alemán","to":"gringo"}
{"confidence":0,"rel":["syn"],"from":"mayo","to":"guainica"}
{"confidence":0,"rel":["syn"],"from":"mayo","to":"maisito"}
{"confidence":0,"rel":["syn"],"from":"mayo","to":"mayito"}
{"confidence":0,"rel":["syn"],"from":"mayo","to":"turpial de sureste"}
{"confidence":0,"rel":["syn"],"from":"domingo","to":"paga"}

Relationships

The required fields for each entry are:

  • from - Original word to search for.
    • This field will be eliminated after it is added to the entities of the LexItem.
  • to - Resulting word. 
    • It will be a new LexItem on its own.
  • rel - List of relationships between the original word and the resulting word.
    • List of relationships in the default dictionary:
      • pl - pluralization
      • vf - verb form
      • ob - obsolete
      • syn - synonym
      • alt - alternative
      • wwm - word with meaning (more than one)
      • wnm - word no meaning (no additional meaning)


  • No labels