This stages are contained inside the Saga Core library and available in all time
Text Block Readers
Readers read text streams and create text blocks to process.
Tokenizers
Tokenizers read text blocks and divide them up into individual tokens to be processed.
- WhitespaceTokenizer - Divides text blocks into tokens based on white space.
Splitters
Splitters split up tokens into multiple smaller tokens as an alternative interpretation.
CharacterSplitter - Tokens are split when any in a specified set of characters (typically punctuation) is encountered.
- CharChangeSplitter - Tokens are split when any difference between caharaters is encountered.
Normalizers
Normalizers create alternative normalized interpretations of tokens from original tokens.
- CaseAnalysis - Analyzes and flags the case of tokens and then (optionally) normalizes the token to lower case.
Recognizers
Recognizers identify and flag tokens based on their character patterns.
- NumberRecognizer - Identifies tokens which look like numbers and flags them with the "NUMBER" flag.
Taggers
Taggers create semantic tags which are added to the interpretation graph as alternative interpretations.
- DictionaryTagger - Looks up all combinations of tokens in a dictionary and tags any that are found.
- AdvancedPattern -
Overview
Content Tools