...

It's recommended that you read this guide before starting writing code, so you get an better understanding of the big picture.

Step-by-step guide

Step 1: Download Wiktionary dump file

Dump files are periodically generated as a back up in the Wikimedia site. You can see a list of all Wiki backups including Wiktionary for each language at this URL: https://dumps.wikimedia.org/backup-index.html

...

Once downloaded just unzip it and you are ready to go.

Step 2: Get familiar with Wiktionary format and specific templates for you language

As you may already know, Wiktionary is an multilingual project to create free content dictionary of all words in all languages.

...

This means that before implementing a new language for SAGA Lemmatize Stage, it is a good idea to get familiar with the specifics of the language.

2.1 Inspecting an Entry

First step is to load wiktionary in your browser: https://www.wiktionary.org/

...

• Portions that appear enclosed in curly brackets are called templates. Templates have a name and usually none to N parameters which can be named-parameters or numbered-parameters (have no name but need to appear in order)
• When there is a sequence of  2 to 4 equal (=) characters, it usually means it is the start of a new section. In the image above the section WordSense is started with === {{sustantivo femenino|es}} ===. This section defines the word form (a noun in this case) and all different senses the word has.

2.2 Learning more about Wiktionary

For more information use the Help in the Wiktionary site. Also look for your language specific help page.  Some examples:

...

Step 3: Get familiar with JWKTL and add support for your language

Wiktionary was created to be understood by humans which means data is not available in a structured way that is easily consumable by computers. 

...

Following image shows the structure of the JWKTL project. Notice there is a folder for each language. You'll need to add a new folder for your desired language. Each language folder has a set of handlers which are the ones actually parsing the wiktionary:

3.1 How JWKTL works?

1. It will detect the language of the dump file and use the correct parser for the language detected.
2. For each entry in the dump file:
1. It will iterate the entry line by line:
1. Send the line to eachone of the Handlers until finding which one can handle the current line. So for example, if the current line has the pattern of a new section and the section is Etymology, then the EtymologyHandler will process and extract the information from the line. A section is usually conformed by several lines,  There is code in place to know if the next line belongs to the same section and needs to be handled by the same handler or if a new section was found and a new handler needs to handle the new section.

...

3.2 Additional changes

1. Add your new language as a new static field in this class: src/main/java/de/tudarmstadt/ukp/jwktl/api/util/Language.java  

...

2. Add your new language parser instantiation in the 'onSiteInfoComplete' method in the class: 'src/main/java/de/tudarmstadt/ukp/jwktl/parser/WiktionaryArticleParser.java'

3.3 Testing

The project has a lot of unit tests. Just grab some for an existing ones and adjust them for your new language.

Step 4:  Update Wiktionary Dump Parser tool for your new language

Even though JWKTL does a lot of parsing and give us information in a structured way, there is still some parsing we need to do in the dump parser tool.

...

Therefore, the dump parser tool will create a plural type relationship between the word 'casas' and its root 'casa'. This relationship entry is something SAGA understands and will use to reduce the word 'casas' to 'casa' in the Lemmatize stage.

4.1 Steps to add your new language

1. Get the code from Git: https://source.digital.accenture.com/projects/ST/repos/saga-wiktionary-dump-parser/browse

...

3. Add page parser instantiation for your language in the method 'GetPageParser' in the class: '\src\main\java\com\searchtechnologies\wiktionary\WiktionaryParser.java':

...

3. Add normalizer instantiation for your language in the method 'GetNormalizer' in the class: '\src\main\java\com\searchtechnologies\wiktionary\RelationNormalization.java':

4.2 Using the Dump Parser Tool

The tool is a command line tool. If you run it without any parameter you'll get help information.

...

 -dict lang=spa indexDir=c:/temp/index outputDir=c:/temp/saga host=localhost port=27017 db=dictionary collection=wiktionary

Step 5: Add Wiktionary file to Saga Library

Once you have created a new dictionary file for you language, you'll need to add it to Saga-Library so it can be used by the Lemmatize Stage.

...

Then add the file to saga-library GIT: https://source.digital.accenture.com/projects/ST/repos/saga-library/browse at the following path:  \src\main\resources

5.1 Testing

In order to test SAGA Lemmatize stage using your new language dictionary, add new unit tests to the file: "TestLemmatizeStage":

...

Related articles

...