Page History
...
...
Operates On: Lexical Items with TOKEN and possibly other flags as specified below.
| Note |
|---|
This lemmatization does not use rules. |
| Include Page | ||||
|---|---|---|---|---|
|
Configuration Parameters
...
Parameter summary The resource containing the list of words and relationships name
...
dictionary - If no dictionary is provided, a
...
- built-in dictionary will be
...
- used, based on the languageISO3.
Parameter summary A list of the relationships to include name include
...
type string array Parameter summary A list of the relationships to exclude
...
- Tokens marked with this flags will be ignore by this stage, and no process will be performed.
...
- Tokens need to have all the specified flags, in order to be processed
...
- Enable all debug log functionality of the stage, if any.
...
name exclude type string array Parameter summary The language the lemmatize stage should use. The value needs to be one of the ISO 3 letter language codes
...
name languageISO3 - By default, English is always used unless configured otherwise. At the moment, only English (ENG) and Spanish (SPA) are available.
- By default, English is always used unless configured otherwise. At the moment, only English (ENG) and Spanish (SPA) are available.
| Note |
|---|
...
A default dictionary is available in English |
...
. Spanish is supported when parameter languageISO3 is configured properly |
...
. |
| Code Block | ||||||
|---|---|---|---|---|---|---|
|
...
|
...
|
...
|
...
| |
"include" : ["pl", "vf"],
|
...
"exclude" : ["ob"],
|
...
"dictionary" : "lemmatize-provider:lemmatize_words",
|
...
"languageISO3":"SPA" |
...
Example Output
| Code Block | ||
|---|---|---|
|
...
...
V--------------------[I am liking this projects very much]--------------------V
|
...
^--[I]--V--[am]--V--[liking]--V--[this]--V--[projects]--V--[very]--V--[much]--^
|
...
^--[be]--^---[like]---^ ^--[project]---^ |
...
|
...
|
...
Output Flags
Lex-Item Flags:
- LEMMATIZE- All words
...
- retrieved will be marked as LEMMATIZE.
- ALL_LOWER_CASE - All of the characters in the token are lower-case characters.
- TOKEN - This stage creates a new token.
Vertex Flags:
| Info |
|---|
| No vertices are created in this stage |
Resource Data
| Anchor | ||||
|---|---|---|---|---|
|
The resource data will be a json file with an array of words in a field named words. This is when the 'dictionary' parameter is used.
| Code Block | ||
|---|---|---|
|
...
...
"words": [
|
...
{
|
...
"confidence": 0.0049,
|
...
"rel": [
|
...
"wnm",
|
...
"sp"
|
...
],
|
...
"from": "encyclopaedia",
|
...
"to": "encyclopedia"
|
...
},
|
...
{
|
...
"confidence": 0.0752,
|
...
|
...
"rel": [
|
...
"wnm",
|
...
"sp"
|
...
],
|
...
|
...
"from": "word",
|
...
"to": "worth"
|
...
}
|
...
] |
...
When the 'dictionary' parameter is not
...
used, an embedded
...
Wiktionary file will be used. This file is formatted as a 1 entry json per line:
| Code Block | |||
|---|---|---|---|
|
...
| |||
{"confidence":0,"rel":["syn"],"from":"japonés","to":"nipón"}
{"confidence":0,"rel":["syn"],"from":"alemán","to":"germano"}
{"confidence":0,"rel":["syn"],"from":"alemán","to":"tedesco"}
{"confidence":0,"rel":["syn"],"from":"alemán","to":"teutón"}
{"confidence":0,"rel":["syn"],"from":"alemán","to":"gringo"}
{"confidence":0,"rel":["syn"],"from":"mayo","to":"guainica"}
{"confidence":0,"rel":["syn"],"from":"mayo","to":"maisito"}
{"confidence":0,"rel":["syn"],"from":"mayo","to":"mayito"}
{"confidence":0,"rel":["syn"],"from":"mayo","to":"turpial de sureste"}
{"confidence":0,"rel":["syn"],"from":"domingo","to":"paga"} |
Relationships
| Anchor | ||||
|---|---|---|---|---|
|
The required fields for each entry are:
- from - Original word to search for.
...
- This field will be eliminated
...
- after it is added to the entities of the LexItem.
- to - Resulting
...
- word.
...
- It will be a new LexItem on its own.
- rel - List of relationships between the original word and the resulting word.
- List of relationships in the default dictionary:
- pl - pluralization
- vf - verb form
- ob - obsolete
- syn - synonym
- alt - alternative
- wwm - word with meaning (more than one)
- wnm - word no meaning (no additional meaning)
- List of relationships in the default dictionary:
...
Overview
Content Tools