Reads a text stream and outputs a list of text blocks.
Configuration Parameters
- splitRegex - A Java Pattern which identifies where a block should be split.
Example Configuration
{
"splitRegex":"\s*\n\n\s*"
}
(split blocks on two new-lines)
Flags
Lex-Item Flags:
- TEXT_BLOCK - Flags all text blocks produced by the SimpleReader
Vertex Flags:
- TEXT_BLOCK_SPLIT - Identifies the vertex as a split between text blocks.
- OVERFLOW_SPLIT - Identifies that an entire buffer was read without finding a split between text blocks.
- The current maximum size of a text block is 64K characters.
- Text blocks larger than this will be arbitrarily split, and the vertex will be marked with "OVERFLOW_SPLIT"\
Overview
Content Tools