Page History

Versions Compared

Key

This line was added.
This line was removed.
Formatting was changed.

...

CharacterSplitter - Tokens are split when any in a specified set of characters (typically punctuation) is encountered.
CharChangeSplitter - Tokens are split when any difference between caharaters is encountered.

Normalizers

Normalizers create alternative normalized interpretations of tokens from original tokens.

...