Versions Compared

Old Version 7

changes.mady.by.user Potter (Esteban Alvarado)

Saved on

compared with

New Version 8

changes.mady.by.user Potter (Esteban Alvarado)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • TOKEN - Identifies that the Lex-Items produced by this stage are tokens and not text blocks.
  • ORIGINAL - Identifies that the Lex-Items produced by this stage are the original, as written, representation of every token (e.g. before normalization)
  • HAS_DIGIT - Tokens produced with at least one digit character are tagged as HAS_DIGIT 
  • HAS_PUNCTUATION - Tokens produced with at least one punctuation character are tagged as HAS_PUNCTUATION.

Vertex Flags:

  • ALL_WHITESPACE - Identifies that the characters spanned by the vertex are all whitespace characters (spaces, tabs, new-lines, carriage returns, etc.)

...