Endpoints Configuration

Endpoints are JavaScript scripts. To add a new endpoint, just need a script to run when that endpoint is invoked (and some configuration to control it). Using the vast array of JavaScript modules available for NodeJS, it's possible to make the endpoints for any propose. 

Adding New Endpoints 

In the folder server/endpoints can be found all the endpoints available which can be use as reference for the new endpoint. Make sure you have the following requirements:

• Create a folder with the endpoint name of your choice. The folder must be inside the folder server/endpoints 
• Inside the folder two files are require:
  •  endpoint.json : The endpoint configuration 
  •  And a javascript file. 
  In the following section is explained how to configure each of this files.
• The Enterprise Search UI, also needs to be updated:
  • The service file src/app/shared/services/api.service.ts needs to be updated with a new function which will point to the new endpoint.  
  • Later this function can be use where it is need in the user interface code.

Configuration and Script Location

All endpoint configuration and scripts are held on disk in the endpoints directory under the installation directory. There is a directory for each endpoint so, for instance, the search endpoint (as shipped) can be found in the endpoints/search directory.

In this directory, you'll find an endpoint.json file. This is the main configuration file for the endpoint and holds its properties.

Configuring an Endpoint

The endpoint.json file for the search endpoint is shown below:

module.exports = {
    endpoint: 'search',
    enabled: true,
    description: 'Default search endpoint',
    processing: [
        [
            {
                label: 'search',
                engine: 'elastic',
                scripts: [
                    {
                        script: 'scripts/search',
                        config: {
                            default_index: {
                                compositeSearchFields: {
                                    content: 1.0,
                                    title: 1.0,
                                    url: 1.0
                                },
                                suggest: {
                                    field: 'title',
                                    candidates: 3,
                                    gram_size: 3,
                                    mode: 'always'
                                },
                                security: {
                                    aclField: 'acls'
                                },
                                dateField: 'lastModified',
                                filters: {},
                                fetch: {
                                    excludes: ['cleandescriptionsearch']
                                },
                                parsed_options: {
                                    extendedOperators: true,
                                    customOperators: true,
                                    wildcards: true
                                }
                            }
                        }
                    }
                ]
            }
        ]
    ]
}

Reading down the file, you can see the following:

• endpoint - The endpoint that will be published on the server. If you have search (or in fact /search), then the server will listen for put requests on http://server/search (assuming it's enabled)
• enabled - true to indicate this endpoint can be used (defaults to true if not present)
• description - A human readable description of the endpoint. Displayed if you browse to the server
• processing
  • Fields:
    • label -This value will be used to name the object containing the results of the endpoint.
    • engine - The search engine to use. Defined in the folder server/engines
    • scripts - Array of scripts that will be executed by the endpoint.

      • script - The path of the script 

      • config - This is the config of each script. You can access it from the javascript script. You can define the your own structure. This is the suggested config settings. 

        • default_index - This will be the configuration use as default in case a configuration with the name of the index doesn't exist

          • compositeSearchFields:- This is the list of fields where qpl will generate the query within the boosting value.

          • suggest - This is the configuration of elastic search for the suggestion feature. 

          • security - Security properties applicable for the search

            • aclField - Field of the acls applicable for each document

          • dateField - Date field used for the advanced search form

          • filters - Filter object to be applied to the search
          • fetch - Object indicating the fields to fetch
            • excludes - Fields to be excluded from the response
            • includes - Fields to be included from the response
          • parsed_options - Properties applicable for the QPL Parser

Endpoint Processing

When a request is received on an endpoint, it triggers the executions of scripts associated with the endpoint (and configured in the endpoint.json file described above). Multiple scripts can be run, sequentially or in parallel, with the results of one script available to the next. Once all scripts have been run, the result is returned to the caller. This flexible processing allows for a number of scenarios, including making multiple searches with merged results to be handled.

See here for detailed information on endpoint processing.

Engine Configuration

Configuring engines allows the Enterprise Search server to talk to search engines. It is designed to be used with QPL for JavaScript. 

When connecting to search engines, it's likely that whilst you might have a certain number of endpoints, there will not typically be a one to one correspondence to the search engines used. For example, in the shipped configuration, both the search and typeahead endpoints use the same search engine configuration.

Configuration Location

Engine configurations are held in the engines directory under the installation directory. Each engine configuration is held in its own json file, for example elastic.json.

Configuration Example

An example configuration (for Elasticsearch) is shown below:

{
  "id": "elastic",
  "enabled": true,
  "implementation": "qpl-elastic", /* QPL JS Engine specific version */
  "config": {
    "server": "http://localhost:9200",
    "version": 6,
    "serverAuth": "http://user:password@localhost:9200"
  }
}

Reading through the file you'll see:

• id - The id of the engine. This must match the value given in engine in the processing scripts described above
• enabled - true if the engine can be used (and true if not present)
• implementation - The engine implementation. Most typically this this will be a QPL engine-specific module (deployed to Artifactory or the central NodeJS package repository). It can also be a reference to a JavaScript file on disk.
• config - The engine configuration that will be passed to the processing script
  • If the Elasticsearch server uses authentication ( user/password) you can use the property serverAuth to set up the credentials. Enterprise Search will use this method to connect to the server if the field is set. 

If you need to connect more than one engine of the same type, you'll simply have two engine configurations with different ids using the same implementation but different configurations. You'll then use the appropriate id in the appropriate script configuration.

Use

When an endpoint that has an engine configured is called, the SEIA server first loads the script referenced in the implementation and calls new on it, passing the configuration, to get a reference. This reference will then be passed to the processing scripts, where its methods may be called to access engine functionality.