Page History

Versions Compared

Key

This line was added.
This line was removed.
Formatting was changed.

...

If the token being processed is "token.123", the parameter "punctuation" is set, then one of tokens generated should be "." and it will have the ALL_PUNCTUATION flag.
If the token being processed is "token.123", the parameter "punctuation" is NOT set, then one of tokens generated should be "token." and it will have the HAS_PUNCTUATION flag.

Lex-Item Flags:

TOKEN - All tokens produced are tagged as TOKEN.
ALL_PUNCTUATION - Tokens processed or produced composed only of punctuation characters are tagged as ALL_PUNCTUATION.
HAS_DIGIT - Tokens produced with at least one digit character are tagged as HAS_DIGIT.
HAS_PUNCTUATION - Tokens produced with at least one punctuation character are tagged as HAS_PUNCTUATION. (ALL_PUNCTUATION will not be tagged as HAS_PUNCTUATION).

Vertex Flags:

If no flag is set on the "splitFlag" parameter:

...