Page History

...

• TOKEN - All tokens produced are tagged as TOKEN.
• ALL_PUNCTUATION - Tokens processed or produced composed only of punctuation characters are tagged as ALL_PUNCTUATION.
• HAS_DIGIT - Tokens produced with at least one digit character are tagged as HAS_DIGIT. 
• HAS_PUNCTUATION - Tokens produced with at least one punctuation character are tagged as HAS_PUNCTUATION. (ALL_PUNCTUATION will not be tagged as HAS_PUNCTUATION).
• ALL_DIGITS -  All characters in the token are digits.
• HAS_LETTER - At least one character is a letter.
• ALL_LETTERS - All characters in the token are letters.
• SPLIT - Tokens splat are tagged with SPLITsplit are tagged with SPLIT.
• SPLIT_ON_CASING - Split on lower to upper case.
• SPLIT_ON_NUM_TO_ALPHA - Split on number to alphabetic.
• SPLIT_ON_ALPHA_TO_NUM - Split on alphabetic to number.
• SPLIT_BEFORE_PUNCT - Split before punctuation.
• SPLIT_AFTER_PUNCT - Split after punctuation.
• SPLIT_DOUBLE_PUNCT - Split on two punctuation characters.

Vertex Flags:

If no flag is set on the "splitFlag" parameter:

...