...
- TOKEN - All tokens produced are tagged as TOKEN.
- ALL_PUNCTUATION - Tokens processed or produced composed only of punctuation characters are tagged as ALL_PUNCTUATION.
- HAS_DIGIT - Tokens produced with at least one digit character are tagged as HAS_DIGIT.
- HAS_PUNCTUATION - Tokens produced with at least one punctuation character are tagged as HAS_PUNCTUATION. (ALL_PUNCTUATION will not be tagged as HAS_PUNCTUATION).
- ALL_DIGITS - All characters in the token are digits
- HAS_LETTER - At least one character is a letter
- ALL_LETTERS - All characters in the token are letters
Vertex Flags:
If no flag is set on the "splitFlag" parameter:
- ALLCHAR_PUNCTUATION CHANGE - Identifies the vertex all punctuation charactersas change type between 2 character (between letter, number or punctuation).