Page History

Operates On:  Lexical Items with TOKEN and possibly other flags as specified below.

Configuration Parameters

• Parameter
  summary The selected spellchecking algorithm. default "Levenshtein" name algori

• Parameter
  summary Extra parameters for the selected spellchecking algorithm. name algorithm_params

• dictionary (string, required) -

  Parameter
  summary

  The dictionary resource that holds the names and that is to be located in the text

  .

  name dictionary required true

  • This is specified as "provider:name" in the standard resource format.
• ignoreTags (string array, optional) -

  Parameter
  summary

  Ignore matches with tags specified in the ignoreTags list

  .

  name dontProcessTags type string array

• Parameter
  summary Name of the tag which indicates what entities should be process name entity

  
  
• normalizeAccents (boolean, optional) - 

  Parameter
  summary

  Removes accents and diacritics and generates a new pattern

  . The

  default

  is false.

  false name normalizeAccents

• removeChars (boolean, optional) -

  Parameter
  summary

  Indicates if characters should be removed from the pattern using a list creating a new pattern

  .  The

  default

  is false.

  false name removeChars type boolean

• charList (string, optional) -

  Parameter
  summary

  List of characters to remove from

  the pattern,

  the

  current

  pattern default

  is "_-‿⁀⁔︳︴﹍﹎﹏＿".
  

{
 "type":"DictionaryTagger",
 "dictionary":"dict-provider:people-lowercase"
}

• _-‿⁀⁔︳︴﹍﹎﹏＿ name charsList

• Parameter
  summary Threshold use to filter when doing vector similarity default 0.7 name cosineSimThreshold type double

• Parameter
  summary Activate spellchecking to find entities misspelled. default false name spellchecking type boolean

• Parameter
  summary Activate matching based on Total Match. default false name matchAll type boolean

• Parameter
  summary Optional threshold to match based on coverage of text matched. default 1.0 name matchAllThreshold type double

"algorithm": "Levenshtein"
"algorithm_params": {}
"dictionary": "dict-provider:people-lowercase",
"dontProcessTags": ["color", "currency"],
"normalizeAccents": false,
"removeChars": false,
"charsList": "_-‿⁀⁔︳︴﹍﹎﹏＿"
"spellchecking": false
"cosineSimThreshold": 0.7
"lowercase": true
"matchAll": false
"matchAllThreshold": 1.0

Example Output

In the following example, "abraham lincoln" is in the dictionary as a person, "lincoln" as a place,  and "macaroni", "cheese" and "macaroni and cheese" are all specified as foods:

V--------------[abraham lincoln likes macaroni and cheese]--------------------V
^--[abraham]--V--[lincoln]--V--[likes]--V--[macaroni]--V--[and]--V--[cheese]--^
                           ^--[{place}]--^           ^---[{food}]---^         ^--[{food}]--^
^---------[{person}]--------^           ^----------------[{food}]-------------^

Output Flags

Lex-Item Flags

• SEMANTIC_TAG - Identifies all lexical items that are semantic tags.
• PARTIAL_MATCH - Identifies partial matches of patterns.
• TOTAL_MATCH - Identifies total matches of patterns.
• ENTITY - Identifies the token as an entity.PROCESSED - Placed on all the tokens which composed the semantic tag.
• MISSPELL - Identifies tokens with errors or misspells.

Vertex Flags:

Resource Data

The dictionary tagger must have an "entity dictionary" (a string to JSON map) which is a list of JSON records, indexed by entity ID. In addition, there may also be a pattern map and a token index.

Entity Dictionary Format

The only file that is absolutely required is the entity dictionary. It is a series of JSON records, typically indexed by entity ID.

Each JSON record represents an entity. The format is as follows:

{
  ""_id" : "KGAAJGsBemSwA0nZTLXA",
"id":"Q28260",
  "tagstag":["{city}", "{administrative-area}", "{geography}"],
  
"display": "Lincon"
"patterns":[
    "Lincoln", "Lincoln, Nebraska", "Lincoln, NE"
  ],
"fields": {
  "confidence"coord": [40.813639, -96.702611]
}
"confAdjust": 0.95
  
  . . . additional fields as needed go here . . . 
}

Fields

• Parameter
  summary An ID normally refering the ID of a database, a document, an API key, not necessary unique name id required true

• Parameter
  summary Tag which will identify any match in the graph, as an interpretation name tag required true

• id (required, string) - Identifies the entity by unique ID. This identifier must be unique across all entities (across all dictionaries).
  • Typically, this is an identifier with meaning to the larger application which is using the Language Processing Toolkit.
• tags (required, array of string) - The list of semantic tags that will be added to the interpretation graph whenever any of the patterns are matched.
  

  • These will all be added to the interpretation graph with the SEMANTIC_TAG flag.

    Tip
    Tags

    Typically, multiple tags

    are hierarchical representations of the same intent.

     

    For example, {city} → {administrative-area} → {geographical-area}

• patterns (required, array of string) -

  Parameter
  summary

  A list of patterns to match in the content

  .

  name patterns type string array required true

  • Patterns will be tokenized and there may be multiple variations which can match.

     Currently

    Note

    NOTE:

    Currently, tokens are separated on simple white-space and punctuation, and then reduced to lowercase.

    TODO:  This will need to be improved in the future, perhaps by specifying a pipeline to perform the tokenization and to allow for multiple variations.
• confidence (optional, float) - Specifies the confidence level of the entity, independent of any patterns matched.
  • This is the confidence of the entity, in comparison to all of the other entities. Essentially, the likelihood that this entity will be randomly encountered.

Other, Optional Fields

• display (optional, string) -

  Parameter
  summary

  What to show the user when browsing this entity

  .context (optional, object) - A context vector that can help disambiguate this entity from others with the same pattern.Format TBD, but probably a list of weighted words, phrases and tags.

  name display

  
  

• Parameter
  summary Free space to add extra data in any format supported by JSON name fields type json

Dictionary Index

To improve performance especially for every large databases of entities, the entity dictionary is inverted and indexed.

This currently happens in RAM inside the DictionaryTagger stage. An off-line option for pre-inverting the dictionary will be provided in the future.