Page History

Excerpt
This stage works in a similar way to the Dictionary Tagger stage in the sense that looks up sequences of tokens in a dictionary to match the text being processed. The difference is that it will also include in the matching text N tokens to the right and/or left of the original matched text.

Operates On: Lexical Items with TOKEN and possibly other flags as specified below.

Saga_is_recognizer

V-----------------[call 333-4444 for pizza]------------------V
^------[call]------V-----[333-4444]------V-[for]-V--[pizza]--^
^-[{_leftTokens_}]-^---[{phonenumber}]---^-[{_rightTokens_}]-^
^-[{_matchedTokens_}]-^
^----------------------[{pizza_phone}]-----------------------^

Include Page

	Generic Configuration Parameters
	Generic Configuration Parameters

Configuration Parameters

Parameter
summary When true, the recognizer will create internal tags to tag the left part, the matched text and the right part
default false
name groupTokens
type boolean
Parameter
summary Internal Tag name for the tokens at the left of the matched text
default _leftTokens_
name leftGroupTagName
Parameter
summary Internal Tag name for the matched text portion
default _matchedTokens_
name matchGroupTagName
Parameter
summary Internal Tag name for the tokens at the right of the matched text
default _rightTokens_
name rightGroupTagName

Saga_config_stage

boundaryFlags	text block split

    "type": "TokenMatcherRecognizer",
    "dictionary": "dict-provider:token_matcher_patterns",
	"groupTokens": true,
	"leftGroupTagName": "_leftTokens_",
	"matchGroupTagName": "_matchedTokens_",
	"rightGroupTagName": "_rightTokens_",
    "boundaryFlags": ["TEXT_BLOCK_SPLIT"],
    "requiredFlags": ["TOKEN", "SEMANTIC_TAG"],
    "skipFlags": : ["SKIP"],
    "atLeastOneFlag": [],
    "confidenceAdjustment": 1,
    "debug": false,
    "enable": true

Example Output

Saga_graph

 V-----------------[call 333-4444 for pizza]------------------V 
 ^------[call]------V-----[333-4444]------V-[for]-V--[pizza]--^ 
 ^-[{_leftTokens_}]-^---[{phonenumber}]---^-[{_rightTokens_}]-^ 
                    ^-[{_matchedTokens_}]-^                     
 ^----------------------[{pizza_phone}]-----------------------^

Output Flags

Lex-Item Flags:

SEMANTIC_TAG - Identifies all lexical items which are semantic tags.
TOKEN_MATCHER - Identifies a semantic tag that covers both the matched text and the tokens to the right and/or left
TOKEN_MATCHER_GROUP - Identifies the semantic tags used to cover the tokens grouped when using the "groupTokens" configuration

Vertex Flags:

Info
No vertices are created in this stage

ALL_PUNCTUATION - Identifies the vertex as all token
The default flag if no "splitFlag" is present.

<splitFlag> - Defines an alternative flag to ALL_PUNCTUATION, if desired (see above)

CHAR_CHANGE - Identifies the vertex as a change between character formats

TEXT_BLOCK_SPLIT - Identifies the vertex as a split between text blocks.

OVERFLOW_SPLIT - Identifies that an entire buffer was read without finding a split between text blocks.
The current maximum size of a text block is 64K characters.
Text blocks larger than this will be arbitrarily split, and the vertex will be marked with "OVERFLOW_SPLIT"\
ALL_WHITESPACE - Identifies that the characters spanned by the vertex are all whitespace characters (spaces, tabs, new-lines, carriage returns, etc.)
Resource Data
Description of resource.
Resource Format
Saga_json
Title Entity Json Format
"_id" : "KGAAJGsBemSwA0nZTLXANlp3kXYBOOvNPbzJXDcQ", "tag": "recipeqVRZi3YBvyJs83wnkfik", "pattern": "("how many"|"how much") {ingredient} {phonenumber}", "confAdjust": 0.95, . . . additional fields as needed go here . . ."options" : { "maxTokensLeft": 1, "maxTokensRight": 1 }
Note
Multiple entries can have the same pattern. If the pattern is matched, then it will be tagged with multiple (ambiguous) entry IDs.
Additional fielded data can be added to the record; as needed by downstream processes.
Fields
Parameter
summary What to show the user when browsing this entity
name display
required true
Parameter
summary Tag which will identify any match in the graph, as an interpretation
name tag
required true
These will all be added to the interpretation graph with the SEMANTIC_TAG flag.
Tip
Tags are hierarchical representations of the same intent. For example, {city} → {administrative-area} → {geographical-area}
Parameter
summary Pattern to match in the content
name pattern
required true
Include Page
Generic Resource Fields
Generic Resource Fields

Page tree

Versions Compared

Old Version 1

New Version 2

Key

Configuration Parameters

Example Output

Output Flags

Lex-Item Flags:

Vertex Flags:

Resource Data

Resource Format

Fields