This stage flags vertices with “Skip-Sentence”. The vertex flag is the start of the sentence. This can be used to ignore a complete sentence by a later stage.The conditions evaluated by the processor are:
Deny listing a tag always has precedence over the other values, so any sentence with a deny listed flag will always be flagged as “SKIP_SENTENCE”. Allow listed tags will always have precedence over the token limit restriction. And finally token limit restriction is on effect.
Operates On: Lexical Items with VERTEX and possibly other flags as specified below.
At this moment only the Python Model Recognizer Stage is capable of using this flag.
"removeSimpleSentence": true, "minTokensOnSentence": 3, "keepSemanticTags": true, "tagsList": ["works"], "markTagsList": ["filtered"]
V----------------------[This is short. This is a longer sentence. This {works}. This is a {filtered}]-----------------------V ^-[This]-V-[is]-V-[short]-V-[This]-V-[is]-V-[a]-V-[longer]-V-[sentence]-V-[This]-V-{works}-V-[This]-V-[is]-V-[a]-V-{filtered}-^ 1 2 3 4 Vertex 1: SKIP_SENTENCE (3 or lest tokens) Vertex 2: (larger than 4 tokens) Vertex 3: (tag {works} found, not flagged) Vertex 3: SKIP_SENTENCE (tag {filtered} found, flagged)
No vertices are created in this stage