Matches advanced recursive patterns of tokens and semantic tags. Pattern databases can be very large (millions) of entries.The new patterns can be formed by previously defined tags and text literals. For example, the pattern "{name} likes {product}" where "{name}" and "{product}" are tags and "likes" is the text literal. Since this stage allows for complex entity tagging, is also known as the "Advance Recognizer".

Operates On:  Lexical Items with SEMANTIC_TAG or TOKEN, and possibly other flags as specified below.

Stage is a Recognizer for Saga Solution, and can also be used as part of a manual pipeline or a base pipeline


On this page

Generic Configuration Parameters

  • boundaryFlags ( type=string array | optional ) - List of vertex flags that indicate the beginning and end of a text block.
    Tokens to process must be inside two vertices marked with this flag (e.g ["TEXT_BLOCK_SPLIT"])
  • skipFlags ( type=string array | optional ) - Flags to be skipped by this stage.
    Tokens marked with this flag will be ignored by this stage, and no processing will be performed.
  • requiredFlags ( type=string array | optional ) - Lex items flags required by every token to be processed.
    Tokens need to have all of the specified flags in order to be processed.
  • atLeastOneFlag ( type=string array | optional ) - Lex items flags needed by every token to be processed.
    Tokens will need at least one of the flags specified in this array.
  • confidenceAdjustment ( type=double | default=1 | required ) - Adjustment factor to apply to the confidence value of 0.0 to 2.0 from (Applies for every pattern match).
    • 0.0 to < 1.0  decreases confidence value
    • 1.0 confidence value remains the same
    • > 1.0 to  2.0 increases confidence value
  • debug ( type=boolean | default=false | optional ) - Enable all debug log functionality for the stage, if any.
  • enable ( type=boolean | default=true | optional ) - Indicates if the current stage should be consider for the Pipeline Manager
    • Only applies for automatic pipeline building

Configuration Parameters

  • patterns ( type=string | required ) - The resource which contains the pattern database
  • maxRepeats ( type=integer | default=5 | required ) - Maximum number of repeating elements checked for when matching wilcards * and +.


$action.getHelper().renderConfluenceMacro("$codeS$body$codeE")

Example Output

The following shows sample output from the advanced pattern matcher, which has multiple patterns for the {product} and {person-product-preference} semantic tags. $action.getHelper().renderConfluenceMacro("$codeS$body$codeE")

Output Flags

Lexical Item Flags:

  • SEMANTIC_TAG - Identifies all lexical items which are semantic tags.

Vertex Flags:

No vertices are created in this stage.

Patterns

The Advanced Pattern Stage accepts patterns that span any number of tokens/items. This is not a character matching/regex engine. The way the patterns works is to replace and expand tokens, not characters, using precedence rules and simple symbols to define repetitions and optional tokens.

An advanced pattern always requires a tag to work. (e.g "token {tag}")


These are the symbols supported for patterns:


SymbolUsage
Zero or more of preceding*{tag}* token*
One or more of preceding+{tag}+ token+
Zero or one of preceding? or [ ]{tag}? [{tag}] token? [token]
Grouping and escaping( ) " "({tag} token) "some tokens" "what?"
Alternations |token1|{tag}|token2

There cannot be whitespaces between the tokens and the symbols, for example “token1 | token2” (notice there are whitespaces between token1 and the pipe symbol as well as after) is not valid, it should be written as “token1|token2”. Or “{tag}?”, adding a space between the closing curly bracket and the question mark will not yield the expected result.

Precedence

This is the precedence order used by the pattern expansion process (from highest to lowest):

""escapingAll other symbols are turned off inside of quotes
( ) [ ]groupings and optionalAll other chars will act on groups
? + *wildcardsApply to tokens, tags, or groups
|alternationsApply to tokens, tags, or groups

Wildcards such as + and * have a limit of repetitions set by the "Maximum Repeats" value on the configuration.


Example patterns:

this that

Matches the literal token "this" followed by the literal token "that"

this|thatMatches "this" or "that".
{animal} Matches any token that has been tagged as an {animal}.
this {tag}? thatMatches "this" followed by "that", optionally with a {tag} token in between
this {tag}* thatAs above, but with any number (zero or more) of {tag} tokens in between.
this {tag}+ thatAs above, but with at least one {tag} token in between
(this that)+Matches "this that this that..." etc.
(left handed)|(right handed) toolMatches "left handed tool" or "right handed tool".
Phrase “here is optional”?Matches “Phrase” and “here is optional” might be at the end of the phrase.
Phrase [here is optional]Matches same as above.
Is the last phrase “here optional?”Matches the whole phrase as the ? mark is within the escaping.
One|two|three|more alternativesMatches multiple optional values. 
In a galaxy far+ awayMatches up to 5 (depending on Maximum Repeats values) repetitions of “far” e.g. In a galaxy far far far far far away.


Resource Data

The resource data is a database of advanced patterns and the resulting semantic tags that they produce.

Resource Format

The pattern database is a series of JSON records, typically indexed by "pattern block ID".  Each JSON record represents a block of patterns (one or more) that all produce the same semantic tag.  The format is as follows: $action.getHelper().renderConfluenceMacro("$codeS$body$codeE")

  • Multiple entries can have the same pattern. If the pattern is matched, then it will be tagged with multiple (ambiguous) entry IDs.
  • Additional fielded data can be added to the record; as needed by downstream processes.

Fields

  • tag ( type=string | required ) - Tag which will identify any match in the graph, as an interpretation

    • These will all be added to the interpretation graph with the SEMANTIC_TAG flag.

      Tags are hierarchical representations of the same intent. For example, {city} → {administrative-area} → {geographical-area}

  • pattern ( type=string | required ) - Pattern to match in the content

  • _id ( type=string | required ) - Identifies the entity by unique ID. This identifier must be unique across all entries (across all dictionaries).

  • confAdjust ( type=boolean | required ) - Adjustment factor to apply to the confidence value of 0.0 to 2.0

    • This is the confidence of the entry, in comparison to all of the other entries. (Essentially, the likelihood that this entity will be randomly encountered.)
    • 0.0 to < 1.0  decreases confidence value
    • 1.0 confidence value remains the same
    • > 1.0 to  2.0 increases confidence value
  • updatedAt ( type=date epoch | required ) - Date in milliseconds of the last time the entry was updated
  • createdAt ( type=date epoch | required ) - Date in milliseconds of the creation time of the entry