Page History

Breaks a text block into sentences, using the BreakIterator of java

Operates On:  Lexical Items with TEXT_BLOCK and possibly other flags as specified below.

Configuration Parameters

• language (string, optional) - Select the language to use for processing the text.
  • Use the java Locales, a summarized list can be found in  Supported Languages
• breakers (string, optional) - String containing the characters where the sentences will be split 
  • The character split is done after the sentence split is perform to the text
• boundaryFlags (string, optional) 
  • The tokens to process must be inside two vertex mark with this flags (e.g ["TEXT_BLOCK_SPLIT"])
• skipFlags (string array, optional) - Flags to be skipped by this stage
  • Tokens marked with this flags will be ignore by this stage, and no process will be performed.
• requiredFlags (string array, optional)
  • Tokens need to have all the specified flags, in order to be processed
• debug (boolean, optional)
  • Enable all debug log functionality of the stage, if any.

{
 "type":"TextBreakerStage",
 "language":"en",
 "breakers":";",
 "boundaryFlags":["TEXT_BLOCK_SPLIT"],
 "debug": true,
}

...

V------------[Lorem ipsum sit amet, consectetur adipisci. Sed luctus lorem. Cras nec ultricies nulla. Maecenas porta cursus; massa non consectetur.]-------------V  
^--[Lorem ipsum sit amet, consectetur adipiscing]--V--[Sed luctus lorem]--V--[Cras nec ultricies nulla]--V--[Maecenas porta cursus]--V--[massa non consectetur]--^  

Output Flags

Lex-Item Flags:

• TEXT_BLOCK - Flags all text blocks produced by the TextBreakerStage
• SENTENCE - Every text block processed by this breaker will be marked as SENTENCE

Vertex Flags:

• SENTENCE_SPLIT - Indicates the split (start/end) between sentences

Supported Languages