...
Readers read text streams and create text blocks to process.
- SimpleReader -
Excerpt Include |
---|
| SimpleReader Simple Reader StageSimpleReader |
---|
| Simple Reader Stage |
---|
nopanel | true |
---|
|
Text Block Breakers
...
Tokenizers read text blocks and divide them up into individual tokens to be processed.
- WhitespaceTokenizer -
Excerpt Include |
---|
| WhitespaceTokenizer Whitespace Tokenizer StageWhitespaceTokenizer |
---|
| Whitespace Tokenizer Stage |
---|
nopanel | true |
---|
|
Splitters
Splitters split up tokens into multiple smaller tokens as an alternative interpretation.
CharacterSplitter -
Excerpt Include |
---|
| CharacterSplitter Character Splitter StageCharacterSplitter |
---|
| Character Splitter Stage |
---|
nopanel | true |
---|
|
- CharChangeSplitter -
Excerpt Include |
---|
| CharChangeSplitter Character Change Splitter StageCharChangeSplitter |
---|
| Character Change Splitter Stage |
---|
nopanel | true |
---|
|
Normalizers
Normalizers create alternative normalized interpretations of tokens from original tokens.
- CaseAnalysis -
Excerpt Include |
---|
| CaseAnalysis Case Analysis StageCaseAnalysis |
---|
| Case Analysis Stage |
---|
nopanel | true |
---|
|
Recognizers
Recognizers identify and flag tokens based on their character patterns.
- NumberRecognizer -
Excerpt Include |
---|
| NumberRecognizer Number Recognizer StageNumberRecognizer |
---|
| Number Recognizer Stage |
---|
nopanel | true |
---|
|
- StopWords -
Excerpt Include |
---|
| StopWords Stop Words StageStopWords |
---|
| Stop Words Stage |
---|
nopanel | true |
---|
|
- Lemmatize -
Excerpt Include |
---|
| Lemmatize Stage |
---|
| Lemmatize Stage |
---|
nopanel | true |
---|
|
...
- RegexPattern -
Excerpt Include |
---|
| Regex Pattern Stage |
---|
| Regex Pattern Stage |
---|
nopanel | true |
---|
|
- DictionaryTagger -
Excerpt Include |
---|
| DictionaryTagger Dictionary Tagger StageDictionaryTagger |
---|
| Dictionary Tagger Stage |
---|
nopanel | true |
---|
|
- AdvancedPattern -
Excerpt Include |
---|
| Advanced Pattern Stage |
---|
| Advanced Pattern Stage |
---|
nopanel | true |
---|
|
- Fragmentation -
Excerpt Include |
---|
| Fragmentation AdvancedPattern StageAdvancedPattern |
---|
| Fragmentation Stage |
---|
nopanel | true |
---|
|