Page History

Operates On:  Lexical Items with TOKEN or SEMANTIC_TAG and

other possible flags as specified below.

Configuration Parameters

• patterns (string, required) - The resource

• that contains the pattern database.
  • See below for the format.

{
 "type":"Fragmentation",
 "patterns":"fragmented-provider:patterns"
}

Example Output

Description

V--------------[abraham lincoln likes macaroni and cheese]--------------------V
^--[abraham]--V--[lincoln]--V--[likes]--V--[macaroni]--V--[and]--V--[cheese]--^
              ^---{place}---^           ^----{food}----^         ^---{food}---^
^----------{person}---------^           ^-----------------{food}--------------^

Output Flags

Lex-Item Flags

• SEMANTIC_TAG - Identifies all lexical items

• that are semantic tags.
• FRAGMENT- Identifies all lexical items

• that were created from a fragmentation pattern.
• PROCESSED - Placed on all the tokens

• that compose the semantic tag.

Resource Data

The resource data

is a database of fragmented patterns, and the resulting semantic

tags they produce.

Resource Format

The only required file

is

the entity dictionary. It is a series of JSON records, typically indexed by entity ID.

Description of entity

Entity JSON Format

{
  "id":"Q28260",
  "tags":["{city}", "{administrative-area}", "{geography}"],
  "patterns":[
    "Lincoln", "Lincoln, Nebraska", "Lincoln, NE"
  ],
  "options": {
	"minTokens": 3,
    "maxTokens": 6,
    "combination": true
  }
  "confidence":0.95
  
  . . . additional fields as needed go here . . . 
}

Notes

1. Multiple entities can have the same pattern.
   • If the pattern is matched, then it will be tagged with multiple (ambiguous) entity IDs.
2. Additional fielded data can be added to the record.
   • As needed by downstream processes.

Fields

• id (required, string) - Identifies the entity by unique ID. This identifier must be unique across all entities (across all dictionaries).
  • Typically, this is an identifier with meaning to the larger application

• • that uses the Language Processing Toolkit.
• tags (required, array of string) - The list of semantic tags

• to be added to the interpretation graph whenever any of the patterns are matched.
  • These will

• • be added to the interpretation graph with the SEMANTIC_TAG flag.
  • Typically, multiple tags are hierarchical representations of the same intent. For example, {city} → {administrative-area} → {geographical-area}
• patterns (required, array of string) - A list of patterns to match in the content.
  • Patterns will be tokenized and

• • multiple variations

• • may match.
    NOTE:  Currenty, tokens are separated on simple white-space and punctuation, and then reduced to lowercase.
    TODO:

• •  This will need to be improved in the future, perhaps by specifying a pipeline to perform the tokenization and to allow for multiple variations.
• options (optional, JSON Object) - Object with options applicable for this entity
  • minTokens (optional, int) - Minimum number of tokens the match must contains to be valid

• • . The default is the number of tokens contained in each pattern

• • .
  • maxTokens (optional, int) - Maximum number of tokens the match must

• • contain to be valid

• • . The default is the number of tokens

• • contained in each pattern. 
  • combination (optional, boolean) - Indicates if the given tokens can be

• • matched in any order

• • as long as all appear in the match

• • . If false,

• • the tokens

• • must be in the order provided.
• confidence (optional, float) - Specifies the confidence level of the entity, independent of any patterns matched.
  • This is the confidence of the entity, in comparison to all of the other entities. Essentially, the likelihood that this entity will be randomly encountered.

Other, Optional Fields

• display (optional, string) - What to show the user when browsing this entity.