Page History

...

Output Flags

Lex-Item Flags:

In version 1.2.2 these flags were added:

• TOKEN - All tokens produced are tagged as TOKEN

...

• ALL_PUNCTUATION - Tokens processed or produced composed only of punctuation characters are tagged as ALL_PUNCTUATION.
• HAS_DIGIT - Tokens produced with at least one digit character are tagged as HAS_DIGIT. 
• HAS_PUNCTUATION - Tokens produced with at least one punctuation character are tagged as HAS_PUNCTUATION. (ALL_PUNCTUATION will not be tagged as HAS_PUNCTUATION).
• ALL_DIGITS -  All characters in the token are digits.
• HAS_LETTER - At least one character is a letter.
• ALL_LETTERS - All characters in the token are

...

• letters.

Vertex Flags:

• if no flag is set on the "splitFlag" parameter:

  • ALL_PUNCTUATION - Tokens processed or produced composed only of punctuation characters are tagged as ALL_PUNCTUATION.

...

• :

  • ALL_PUNCTUATION - Tokens processed or produced composed only of punctuation characters are tagged as ALL_PUNCTUATION.
  • HAS_DIGIT - Tokens produced with at least one digit character are tagged as HAS_DIGIT. 
  • HAS_PUNCTUATION - Tokens produced with at least one punctuation character are tagged as HAS_PUNCTUATION. (ALL_PUNCTUATION will not be tagged as HAS_PUNCTUATION).
  • ALL_DIGITS -  All characters in the token are digits.
  • HAS_LETTER - At least one character is a letter.
  • ALL_LETTERS - All characters in the token are

...

• • letters ALL_PUNCTUATION - Tokens processed or produced composed only of punctuation characters are tagged as ALL_PUNCTUATION.
• And if not if no flag is set on the "splitFlag" parameter, not extra tag is added.