Looks up sequences of tokens in a dictionary and then tags the sequence with one or more semantic tags as an alternative representation. Typically, these tags represent entities such as {person}, {place}, {company}, etc.  This stage is also known as the "Entity Recognizer".

Operates On:  Lexical Items with TOKEN and possibly other flags as specified below.

Stage is a Recognizer for Saga Solution, and can also be used as part of a manual pipeline or a base pipeline

All possibilities are tagged, including overlaps and sub-patterns, with the expectation that later disambiguation stages will choose which tags are the correct interpretation.


Generic Configuration Parameters

  • boundaryFlags ( type=string array | optional ) - List of vertex flags that indicate the beginning and end of a text block.
    Tokens to process must be inside two vertices marked with this flag (e.g ["TEXT_BLOCK_SPLIT"])
  • skipFlags ( type=string array | optional ) - Flags to be skipped by this stage.
    Tokens marked with this flag will be ignored by this stage, and no processing will be performed.
  • requiredFlags ( type=string array | optional ) - Lex items flags required by every token to be processed.
    Tokens need to have all of the specified flags in order to be processed.
  • atLeastOneFlag ( type=string array | optional ) - Lex items flags needed by every token to be processed.
    Tokens will need at least one of the flags specified in this array.
  • confidenceAdjustment ( type=double | default=1 | required ) - Adjustment factor to apply to the confidence value of 0.0 to 2.0 from (Applies for every pattern match).
    • 0.0 to < 1.0  decreases confidence value
    • 1.0 confidence value remains the same
    • > 1.0 to  2.0 increases confidence value
  • debug ( type=boolean | default=false | optional ) - Enable all debug log functionality for the stage, if any.
  • enable ( type=boolean | default=true | optional ) - Indicates if the current stage should be consider for the Pipeline Manager
    • Only applies for automatic pipeline building

Configuration Parameters

  • dictionary ( type=string | required ) - The dictionary resource that holds the names and that is to be located in the text
    • This is specified as "provider:name" in the standard resource format.
  • ignoreTags ( type=string array | optional ) - Ignore matches with tags specified in the ignoreTags list
  • entity ( type=string | optional ) - Name of the tag which indicates what entities should be process
  • normalizeAccents ( type=string | default=false | optional ) - Removes accents and diacritics and generates a new pattern
  • removeChars ( type=boolean | default=false | optional ) - Indicates if characters should be removed from the pattern using a list creating a new pattern
  • charsList ( type=string | default=_-‿⁀⁔︳︴﹍﹎﹏＿ | optional ) - List of characters to remove from the pattern

In version 1.2.2 this parameter was added:

  • cosineSimThreshold ( type=double | default=0.7 | optional ) - Threshold use to filter when doing vector similarity



$action.getHelper().renderConfluenceMacro("$codeS$body$codeE")

"people-lowercase" resource must be in the format specified below.

Example Output

In the following example, "abraham lincoln" is in the dictionary as a person, "lincoln" as a place,  and "macaroni", "cheese" and "macaroni and cheese" are all specified as foods: $action.getHelper().renderConfluenceMacro("$codeS$body$codeE")

Output Flags

Lex-Item Flags

  • SEMANTIC_TAG - Identifies all lexical items that are semantic tags.
  • ENTITY - Identifies the token as an entity.

Vertex Flags:

No vertices are created in this stage

Resource Data

The dictionary tagger must have an "entity dictionary" (a string to JSON map) which is a list of JSON records, indexed by entity ID. In addition, there may also be a pattern map and a token index.

Entity Dictionary Format

The only file that is absolutely required is the entity dictionary. It is a series of JSON records, typically indexed by entity ID.

Each JSON record represents an entity. The format is as follows: $action.getHelper().renderConfluenceMacro("$codeS$body$codeE")

  • Multiple entries can have the same pattern. If the pattern is matched, then it will be tagged with multiple (ambiguous) entry IDs.
  • Additional fielded data can be added to the record; as needed by downstream processes.

Fields

  • id ( type=string | required ) - An ID normally refering the ID of a database, a document, an API key, not necessary unique
  • tag ( type=string | required ) - Tag which will identify any match in the graph, as an interpretation

    • These will all be added to the interpretation graph with the SEMANTIC_TAG flag.

      Tags are hierarchical representations of the same intent. For example, {city} → {administrative-area} → {geographical-area}

  • patterns ( type=string array | required ) - A list of patterns to match in the content

    • Patterns will be tokenized and there may be multiple variations which can match.

      Currently, tokens are separated on simple white-space and punctuation, and then reduced to lowercase.

  • display ( type=string | optional ) - What to show the user when browsing this entity
  • fields ( type=json | optional ) - Free space to add extra data in any format supported by JSON

  • _id ( type=string | required ) - Identifies the entity by unique ID. This identifier must be unique across all entries (across all dictionaries).

  • confAdjust ( type=boolean | required ) - Adjustment factor to apply to the confidence value of 0.0 to 2.0

    • This is the confidence of the entry, in comparison to all of the other entries. (Essentially, the likelihood that this entity will be randomly encountered.)
    • 0.0 to < 1.0  decreases confidence value
    • 1.0 confidence value remains the same
    • > 1.0 to  2.0 increases confidence value
  • updatedAt ( type=date epoch | required ) - Date in milliseconds of the last time the entry was updated
  • createdAt ( type=date epoch | required ) - Date in milliseconds of the creation time of the entry

Dictionary Index

To improve performance especially for every large databases of entities, the entity dictionary is inverted and indexed.

This currently happens in RAM inside the DictionaryTagger stage. An off-line option for pre-inverting the dictionary will be provided in the future.

  • No labels