CharChangeSplitter Stage

Separates tokens based on character changes from lowercase-uppercase, letter-number, alphanumeric-punctuation. Without taking any character in the vertex, and respecting the capital letter

Operates On: Lexical Items with TOKEN flag

Configuration

case(boolean, optional) - If true, all tokens are separated on a case change (default = true)
number(boolean, optional) - If true, all tokens are separated on a letter-number. (default = true)
punctuation(boolean, optional) - If true, all tokens are separated on an alphanumeric-punctuation. (default = true)

Example Configuration

{
  	"type":"CharChangeSplitter",
  	"case": true
	"number": true
	"punctuation": true
}

Output Flags

Note that flags are placed on every token as appropriate to the current token:

If the token is "President" then that token will have the TITLE_CASE flag.
If a new lower-case normalized interpretation, "president" is added as an alternative, this alternative will have the "ALL_LOWER_CASE" flag.

Lex-Item Flags:

TOKEN - All tokens produced are tagged as TOKEN

Vertex Flags:

CHAR_CHANGE - Identifies the vertex as a change between character formats

Example

With Defaults

V--[Issues]--V--[with]--V------------[SagaToken123@#$]-------------V--[send]--V--[email]--V--[to]--V-------------------[[email protected]]--------------------V--[for]--V--[HELP]--V
                        ^--[Saga]--V--[Token]--V--[123]--V--[@#$]--^                               ^--[it]--V--[@]--V--[this]--V--[Company]--V--[.]--V--[com]--^

Without Punctuation Separation

{
	"type":"CharChangeSplitter",
	"punctuation" : false
}

V--[Issues]--V--[with]--V---------[SagaToken123@#$]---------V--[send]--V--[email]--V--[to]--V-----[[email protected]]------V--[for]--V--[HELP]--V
                        ^--[Saga]--V--[Token]--V--[123@#$]--^                               ^--[it@this]--V--[Company.com]--^

Without Number Separation

{
	"type":"CharChangeSplitter",
	"number" : false
}

V--[Issues]--V--[with]--V---------[SagaToken123@#$]---------V--[send]--V--[email]--V--[to]--V-------------------[[email protected]]--------------------V--[for]--V--[HELP]--V
                        ^--[Saga]--V--[Token123]--V--[@#$]--^                               ^--[it]--V--[@]--V--[this]--V--[Company]--V--[.]--V--[com]--^

Without Case Separation

{
	"type":"CharChangeSplitter",
	"case" : false
}

V--[Issues]--V--[with]--V---------[SagaToken123@#$]---------V--[send]--V--[email]--V--[to]--V----------------[[email protected]]----------------V--[for]--V--[HELP]--V
                        ^--[SagaToken]--V--[123]--V--[@#$]--^                               ^--[it]--V--[@]--V--[thisCompany]--V--[.]--V--[com]--^

Page tree