Versions Compared

Old Version 7

changes.mady.by.user Potter (Esteban Alvarado)

Saved on

compared with

New Version 8

changes.mady.by.user Potter (Esteban Alvarado)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Excerpt

Flag tokens matched to stop words, to be skipped for following stages


Operates On:  Lexical Items with TOKEN

Include Page
Generic Configuration Parameters
Generic Configuration Parameters

Configuration Parameters

  • caseInsensitive (string, optional) - If true, all stop words and tokens will be process as case insensitive (default = true)

  • stopWords (string, optional) - The resource containing the list of stop words

    • See below for the format. if no resource is provided the stage will use the default list of stop words


Info
titleDefault list of stop words

a, an, and, are, as, at, be, but, by, for, if, in, into, is, it, no, not, of, on, or, such, that, the, their, then, there, these, they, this, to, was, will, with

Code Block
languagejs
themeEclipse
titleExample Configuration
{
  "type": "StopWordsStage",
  "caseInsensitive" : true,
  "stopWords" : "words-provider:stop_words"
}


Example Output

Code Block
languagetext
themeFadeToGrey
V--------------[A test to be skipped]--------------V  
  ^--[A]--V--[test]--V--[to]--V--[be]--V--[skipped]--^  
  ^--[a]--^  

Item [A] - [TOKEN, SKIP]
Item [to] - [TOKEN, SKIP]
Item [be] - [TOKEN, SKIP]
Item [a] - [TOKEN, SKIP]

Output Flags

Lex-Item Flags:

  • SKIP - All matched stop words will be marked as SKIP

Resource Data

The resource data will be a json file with an array of words in a field named stopWrods

Code Block
languagejs
themeEclipse
{
  "stopWords": ["a", "about", "above", "after", "again", "all", "am", "an", "and", "the", "i", "who", ...]
}