Page History

...

Readers read text streams and create text blocks to process.

• SimpleReader - 

  Excerpt Include
  SimpleReader Simple Reader StageSimpleReader Simple Reader Stage nopanel true

Text Block Breakers

...

Tokenizers read text blocks and divide them up into individual tokens to be processed.

• WhitespaceTokenizer - 

  Excerpt Include
  WhitespaceTokenizer Whitespace Tokenizer StageWhitespaceTokenizer Whitespace Tokenizer Stage nopanel true

Splitters

Splitters split up tokens into multiple smaller tokens as an alternative interpretation.

• CharacterSplitter - 

  Excerpt Include
  CharacterSplitter Character Splitter StageCharacterSplitter Character Splitter Stage nopanel true

• CharChangeSplitter - 

  Excerpt Include
  CharChangeSplitter Character Change Splitter StageCharChangeSplitter Character Change Splitter Stage nopanel true

Normalizers

Normalizers create alternative normalized interpretations of tokens from original tokens.

• CaseAnalysis - 

  Excerpt Include
  CaseAnalysis Case Analysis StageCaseAnalysis Case Analysis Stage nopanel true

Recognizers

Recognizers identify and flag tokens based on their character patterns.

• NumberRecognizer - 

  Excerpt Include
  NumberRecognizer Number Recognizer StageNumberRecognizer Number Recognizer Stage nopanel true

• StopWords -

  Excerpt Include
  StopWords Stop Words StageStopWords Stop Words Stage nopanel true

• Lemmatize -

  Excerpt Include
  Lemmatize Stage Lemmatize Stage nopanel true

...

• RegexPattern - 

  Excerpt Include
  Regex Pattern Stage Regex Pattern Stage nopanel true

• DictionaryTagger - 

  Excerpt Include
  DictionaryTagger Dictionary Tagger StageDictionaryTagger Dictionary Tagger Stage nopanel true

• AdvancedPattern - 

  Excerpt Include
  Advanced Pattern Stage Advanced Pattern Stage nopanel true

• Fragmentation - 

  Excerpt Include
  Fragmentation AdvancedPattern StageAdvancedPattern Fragmentation Stage nopanel true