You are viewing an old version of this page. View the current version.
Compare with Current
View Page History
Version 1
Next »
Breaks a text block into sentences, using OpenNLP Sentence Detector
Operates On: Lexical Items with TEXT_BLOCK and possibly other flags as specified below.
Configuration Parameters
- language (string, required) - Indicates the language of the text to process. default "en" (i.e. English)
- Languages available are
- English - "en"
- German - "de"
- Portuguese - "pt"
- Dutch - "nl"
- skipFlags (string array, optional) - Flags to be skipped by this stage
- Tokens marked with this flags will be ignore by this stage, and no process will be performed.
- requiredFlags (string array, optional)
- Tokens need to have all the specified flags, in order to be processed
- debug (boolean, optional)
- Enable all debug log functionality of the stage, if any.
{
"type":"SentenceBreakerStage",
"language":"en",
"skipFlags": ["SKIP"],
"debug": true,
}
Example Output
The sentence detector will remove any space and dot, at the beginning and end of a sentence
V-----------[Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed eget luctus lorem. Nam cursus eleifend efficitur. Cras nec ultricies nulla. Maecenas porta cursus massa non consectetur.]------------V
^--[Lorem ipsum dolor sit amet, consectetur adipiscing elit]--V--[Sed eget luctus lorem]--V--[Nam cursus eleifend efficitur]--V--[Cras nec ultricies nulla]--V--[Maecenas porta cursus massa non consectetur]--^
Output Flags
Lex-Item Flags:
- TEXT_BLOCK - Flags all text blocks produced by the SimpleReader
- SENTENCE - Every text block processed by this breaker will be marked as SENTENCE
Vertex Flags:
- SENTENCE_SPLIT - Indicates the split (start/end) between sentences