Page History

Operates On:  Lexical Items with TOKEN or SEMANTIC_TAG and possibly other flags as specified below.

Configuration Parameters

• patterns (string, required) - The resource which contains the pattern database
  • See below for the format.

{
 "type":"Fragmentation",
 "patterns":"fragmented-provider:patterns"
}

Example Output

Description

V--------------[abraham lincoln likes macaroni and cheese]--------------------V
^--[abraham]--V--[lincoln]--V--[likes]--V--[macaroni]--V--[and]--V--[cheese]--^
              ^---{place}---^           ^----{food}----^         ^---{food}---^
^----------{person}---------^           ^-----------------{food}--------------^

Output Flags

Lex-Item Flags:

• SEMANTIC_TAG - Identifies all lexical items which are semantic tags.
• FRAGMENT- Identifies all lexical items which were created from a fragmentation pattern.
• PROCESSED - Placed on all the tokens which composed the semantic tag.

Vertex Flags:

• none

Resource Data

The resource data will be a database of fragmented patterns, and the resulting semantic tag which they produce.

Resource Format

The only file which is absolutely required is the entity dictionary. It is a series of JSON records, typically indexed by entity ID.

Description of entity:
Entity JSON Format

{
  "id":"Q28260",
  "tags":["{city}", "{administrative-area}", "{geography}"],
  "patterns":[
    "Lincoln", "Lincoln, Nebraska", "Lincoln, NE"
  ],
  "options": {
	"minTokens": 3,
    "maxTokens": 6,
    "combination": true
  }
  "confidence":0.95
  
  . . . additional fields as needed go here . . . 
}

Notes

1. Multiple entities can have the same pattern.
   1. If the pattern is matched, then it will be tagged with multiple (ambiguous) entity IDs.
2. Additional fielded data can be added to the record
   1. As needed by downstream processes.

Fields

• id (required, string) - Identifies the entity by unique ID. This identifier must be unique across all entities (across all dictionaries).
  • Typically this is an identifier with meaning to the larger application which is using the Language Processing Toolkit.
• tags (required, array of string) - The list of semantic tags which will be added to the interpretation graph whenever any of the patterns are matched.
  • These will all be added to the interpretation graph with the SEMANTIC_TAG flag.
  • Typically, multiple tags are hierarchical representations of the same intent. For example, {city} → {administrative-area} → {geographical-area}
• patterns (required, array of string) - A list of patterns to match in the content.
  • Patterns will be tokenized and there may be multiple variations which can match.
    • NOTE:  Currenty, tokens are separated on simple white-space and punctuation, and then reduced to lowercase.
    • TODO:  This will need to be improved in the future, perhaps by specifying a pipeline to perform the tokenization and to allow for multiple variations.
• options (optional, JSON Object) - Object with options applicable for this entity
  • minTokens (optional, int) - Minimum number of tokens the match must contains to be valid, default is the number of tokens each pattern has
  • maxTokens (optional, int) - Maximum number of tokens the match must contains to be valid, default is the number of tokens each pattern has 
  • combination (optional, boolean) - Indicates if the given tokens can be match in any order possible as long all appear in the match, if false, it means the tokens need be in the order provided
• confidence (optional, float) - Specifies the confidence level of the entity, independent of any patterns matched.
  • This is the confidence of the entity, in comparison to all of the other entities. Essentially, the likelihood that this entity will be randomly encountered.

Other, Optional Fields

• display (optional, string) - What to show the user when browsing this entity.