Versions Compared

Old Version 15

changes.mady.by.user Potter (Esteban Alvarado)

Saved on

compared with

New Version 16

changes.mady.by.user Mauricio Rizo

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Note that flags are placed on every token as appropriate to the current token:

  • If the token being process is "President" then that token will have the TITLE_CASE flag."token.123", the parameter "punctuation" is set, then one of tokens generated should be ".", and it will have the ALL_PUNCTUATION flag.
  • If the token being process is "token.123", the parameter "punctuation" is NOT set, then one of tokens generated should be "token.", and it will have the HAS_PUNCTUATION If a new lower-case normalized interpretation, "president" is added as an alternative, this alternative will have the "ALL_LOWER_CASE" flag.

Lex-Item Flags:

  • TOKEN - All tokens produced are tagged as TOKEN
  • ALL_PUNCTUATION - Tokens processed or produced composed only of punctuation characters are tagged as ALL_PUNCTUATION 
  • HAS_DIGIT - Tokens produced with at least one digit character are tagged as HAS_DIGIT 
  • HAS_PUNCTUATION - Tokens produced with at least one punctuation character are tagged as HAS_PUNCTUATION. (ALL_PUNCTUATION will not be tagged as HAS_PUNCTUATION)

...