Versions Compared

Old Version 16

changes.mady.by.user user-1b188

Saved on

compared with

New Version 17

changes.mady.by.user Potter (Esteban Alvarado)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Include Page
Generic Configuration Parameters
Generic Configuration Parameters

{ "type":"WhitespaceTokenizer" }
Code Block
languagejs
themeEclipse
titleExample Configuration
Saga_config_stage
Output Flags
Lex-Item Flags
  • TOKEN - Identifies that the Lex-Items produced by this stage are tokens and not text blocks.
  • ORIGINAL - Identifies that the Lex-Items produced by this stage are the original, as written, representation of every token (e.g. before normalization).
  • HAS_DIGIT - Tokens produced with at least one digit character are tagged as HAS_DIGIT.
  • HAS_PUNCTUATION - Tokens produced with at least one punctuation character are tagged as HAS_PUNCTUATION.
...