Page History

Versions Compared

Key

This line was added.
This line was removed.
Formatting was changed.

...

TOKEN - All tokens produced are tagged as TOKEN.
ALL_PUNCTUATION - Tokens processed or produced composed only of punctuation characters are tagged as ALL_PUNCTUATION.
HAS_DIGIT - Tokens produced with at least one digit character are tagged as HAS_DIGIT.
HAS_PUNCTUATION - Tokens produced with at least one punctuation character are tagged as HAS_PUNCTUATION. (ALL_PUNCTUATION will not be tagged as HAS_PUNCTUATION).

Info

In version 1.2.2 these flags were added:

ALL_DIGITS - All characters in the token are digits
HAS_LETTER - At least one character is a letter
ALL_LETTERS - All characters in the token are letters

Vertex Flags:

If no flag is set on the "splitFlag" parameter:

...