GAIA API has a versatile search engine solution designed to seamlessly connect with any NonSQL engine. It offers a unified interface for interacting with different search engines, allowing you to utilize the same code for every request. The GAIA API achieves this through its implementation-defined integration with specific search engines while maintaining consistency in its usage.


Key Features

  1. Search Engine Agnosticism: The GAIA API is designed to be independent of any particular search engine. It provides a flexible and adaptable solution that can seamlessly integrate with various NonSQL engines.

  2. Universal Code Usage: With the GAIA API, you can utilize the same code for all search engine requests. Regardless of the underlying search engine implementation, the API provides a consistent interface to interact with.

  3. Normalization of Parameters and Responses: The GAIA API incorporates a normalization process for parameters and responses. This normalization ensures that requests and results are interpreted uniformly across different search engine implementations.

  4. Implementation-Specific Integration: Each specific search engine implementation is responsible for interpreting and executing the normalized parameters and responses. The GAIA API facilitates this integration by providing a standardized interface, allowing for efficient communication between the API and the search engine.

Connection Manager

The connection manager is in charge of the loading of the connection configurations and the initialization with every engine specific implementation. It is also in charge to open and close the connection, pool size, throttling, node balancing,...

Accessing the Connection Manager in Code

You can get the connection Manager in your code using:

from app.rest import connection_manager

We recommend using this import as a local import, calling it exactly when is required and not before, this is to prevent issues with dependencies

Getting Connection by Default

The default connection is the engine connection with the explicit default flag in true, or if none with the flag, the first in the list of connections. This connection is used as well for all internal transactions of GAIA API  such as logging, feedback, analytics,...

To get the default connection, just use the code line below: 

engine_conn = connection_manager.get_default()

Getting Connection by name

You can have multiple connections to the same or different types of NonSQL engines, all with a name you can refer to, in order to get said connection. 

To get a connection by name, just use the code line below:

engine = connection_manager.get_engine(name=engine_name)

Engine Connection Config

All engine connections have the generic connection focus on managing the specific aspects of the REST connection.


PropertyDescriptionDefaultTypeRequired
1nameSpecify the name of this engine
string (pattern: "[a-zA-Z\d_-]+")Yes
2typeType of the engine to use for searches
string (enum: "elasticsearch", "opensearch")Yes
3defaultIf more than one, default = True indicates which engines will be used for GAIA API storagefalsebooleanNo
4engine_urlURLs to the engine nodes
string or array of strings (minLength: 1, maxLength: 65536, format: uri)Yes
5index_prefixPrefix for all GAIA API indexes"sa"string (pattern: "[a-zA-Z\d_]+")No
6pool_connectionsThe number of connection pools to cache10integerNo
7pool_maxsizeThe maximum number of connections to save in the pool.100integerNo
8pool_blockWhether the connection pool should block for connections.falsebooleanNo
9headersDictionary with the predefined headers, express has header name (key), and its value{}objectNo
10proxies
{}objectNo
11paramsDictionary with the predefined parameters, express has parameter name (key), and its value{}objectNo
12authSection for the authentication of the connectionAuthobjectNo
13verifyDefaults to True, requiring requests to verify the TLS certificate at the remote end.truebooleanNo
14cookiesDictionary with the predefined cookies, express has cookie name (key), and its value{}objectNo
15certIf String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair or ('cert', 'key', 'key_password') triplet
string or array or string or file-pathNo
16max_redirectsMaximum times the request is allowed to be redirected.30integer (minimum: 1)No
17max_retriesNumber of times the server will retry a failed request before throwing an error.10integer (minimum: 0)No
18retry_backoff_factorA backoff factor to apply between attempts after the second try. The connection will sleep for {backoff factor} * (2 ** ({number of total retries} - 1)) seconds. If the backoff_factor is 0.1, then will sleep for [0.0s, 0.2s, 0.4s, ...] between retries2.5number (minimum: 0)No
19retry_on_timeoutShould timeout trigger a retry on a different node?falsebooleanNo
20timeoutMaximum time the server will wait for a request to be answered.30integer (minimum: 1)No
21allow_redirectsAllow the request to be redirectedtruebooleanNo
22trust_envTrust environment settings for proxy configuration, default authentication, and similar.truebooleanNo
23use_throttlingLimit the number of API requests the server can make in a certain period.falsebooleanNo
24throttling_rateNumber of simultaneous calls allowed per minute5000integer (minimum: 100)No
25throttling_connection_rate
50integer (minimum: 1)No
26randomize_nodes_in_poolSet to false to not randomize nodes within the pool.truebooleanNo
27node_selector_classClass to be used to select nodes within the NodePool"round_robin"string (enum: "round_robin", "random")No
28dead_node_backoff_factorExponential backoff factor to calculate the amount of time to timeout a node after an unsuccessful API call
numberNo
29max_dead_node_backoffMaximum amount of time to timeout a node after an unsuccessful API call
numberNo
30log_requestsIf True, every request done by the engine will be loggedfalsebooleanNo

Engine Connection Authentication 

Engine authentication section, if none, there will be no authentication performed. If basic, Basic Authentication is used to authenticate against the engine.  There are two possibilities for Azure AI Search: API_KEY and Managed Identity. And finally aws uses AWS credentials with or without credential provider

PropertyDescriptionDefaultTypeRequired
typeIndicates the type of authentication.

none

string - Options["none", "basic", "aws"]No
usernameApplicable for basic authentication. Username for auth.nullstring/nullNo
passwordApplicable for basic authentication. Password for auth.nullstring/nullNo
api_keyApplicable for Azure AI Search API Key authenticationnullstring/nullNo
credentials_providerApplicable for AWS authentication. Use AWS credentials.falsebooleanNo
aws_serviceApplicable for AWS authentication. AWS service to access. (With credential provider)nullstring/nullNo
aws_regionApplicable for AWS authentication. AWS region. (With credential provider)nullstring/nullNo
aws_access_keyApplicable for AWS authentication. AWS access key. (Without credential provider)nullstring/nullNo
aws_secret_keyApplicable for AWS authentication. AWS secret key. (Without credential provider)nullstring/nullNo

Configuration Examples

In this section we show some engine configuration examples for better understanding.

Azure AI Search


{
            'name': 'Azure Ai Search',
            'type': EngineTypes.AZURE_AI_SEARCH,
            'default': True,
            'headers': {
                'Content-Type': 'application/json'
            },
            'engine_url': 'http://engineurl.com:8085',
            'index_prefix': 'gaia',
            'pool_connections': 10,
            'pool_maxsize': 100,
            'params': {
                'api_version': '2024-05-01-preview'
            },
            'pool_block': True,
            'verify': False,
            'max_redirects': 30,
            'max_retries': 10,
            'retry_wait_time': 10,
            'timeout': 60,
            'allow_redirects':True,
            'trust_env': True,
            'use_throttling': True,
            'throttling_rate': 5000,
            'throttling_connection_rate': 50,
            'auth': {
                'type': Authentications.API_KEY,
                # NOTE that the name of the property has an underscore
                'api_key': 123456789012345678901234567890

                # 'type': Authentications.AZURE_MID,
                # 'azure_scope': 'https://search.azure.com/.default'
            },
            'log_requests': False
}




  • No labels