Page History
...
| Tip |
|---|
It can detect 103 languages outputting ISO 639-3 language codes. (https://opennlp.apache.org/news/model-langdetect-183.html) |
| Note |
|---|
It is important to note that the The model works better with longer texts that have containing at least 2 two sentences. So it It is important to configure this stage earlier in the pipeline and before tokenizing the text. |
...
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
{
"type":"LangDetectorStage",
} |
Example Output
As you can see, the first sentence is tagged with "LANG_ENG" and the second sentence with "LANG_SPA". For
In this case, a sentence breaker stage was configured before the language detector stage. This way As a result, language identification could happen can occur at the sentence level.
Output Flags
Lex-Item Flags
...
- TEXT_BLOCK - Flags all text blocks produced by the SimpleReader.
- LANG_??? - Flags all text blocks where a language was identified.
Notice '???' at the end of the Flag. This is replaced by a an ISO 3 three letter language code. For
For example, if Spanish is detected, 3 the three letter code is SPA, then and the Flag will be "LANG_SPA"
Vertex Flags
...
- none
Overview
Content Tools