...
- TOKEN - All tokens produced are tagged as TOKEN.
- ALL_PUNCTUATION - Tokens processed or produced composed only of punctuation characters are tagged as ALL_PUNCTUATION.
- HAS_DIGIT - Tokens produced with at least one digit character are tagged as HAS_DIGIT.
- HAS_PUNCTUATION - Tokens produced with at least one punctuation character are tagged as HAS_PUNCTUATION. (ALL_PUNCTUATION will not be tagged as HAS_PUNCTUATION).
- ALL_DIGITS - All characters in the token are digits.
- HAS_LETTER - At least one character is a letter.
- ALL_LETTERS - All characters in the token are letters.
- SPLIT - Tokens splat are tagged with SPLITsplit are tagged with SPLIT.
- SPLIT_ON_CASING - Split on lower to upper case.
- SPLIT_ON_NUM_TO_ALPHA - Split on number to alphabetic.
- SPLIT_ON_ALPHA_TO_NUM - Split on alphabetic to number.
- SPLIT_BEFORE_PUNCT - Split before punctuation.
- SPLIT_AFTER_PUNCT - Split after punctuation.
- SPLIT_DOUBLE_PUNCT - Split on two punctuation characters.
Vertex Flags:
If no flag is set on the "splitFlag" parameter:
...