Page History

...

Configuration Parameters

• splitChars (string, optional) -

  Parameter
  summary

  List of characters which should be used to split tokens

  .

  name splitChars

  • If not present, then tokens are split on any sequence of punctuation. 
• dontSplitChars (string, optional) -

  Parameter
  summary

  List of characters which will NOT be used to split tokens. name dontSplitChars

  
  
  • This is typically used to identify exceptions (characters which are not used to split tokens) when splitChars is missing.
  • These characters are included in the produced tokens.

• Parameter
  summary if any character in this list occurs inside a token, * that token will be split just before that character name splitBeforeChars

• Parameter
  summary if any character in this list occurs inside a token, * that token will be split just after that character name splitAfterChars

• Parameter
  summary true/false whether to split on all punctuation (default: true) name splitPrefixChars

• Parameter
  summary true/false whether to split on all punctuation (default: true) name splitSuffixChars

• splitFlag (string, optional) -

  Parameter
  summary

  The flag to be put on the vertex between the two tokens. name splitFlag

  
  
  • If missing, defaults to ALL_PUNCTUATION.

...