Page History

Base Configuration (Without Authentication)

Parameters

• Parameter
  summary The name of the provider, used in the pipeline configuration when required by stages. default saga-provider name name required true

• Parameter
  summary The provider type. default Elastic name type required true

   See Resources for more information.

• Parameter
  summary Name for the prefix for each index used in the Elasticsearch server. default saga name indexName

• Parameter
  summary List of Elasticsearch hosts and ports (including schema) default http://localhost:9200 name nodeUrls

• Parameter
  summary Authentication to use, it could be basic, aws or azure. default none name authentication

   

  • Tip
    If none, it can be omitted

• Parameter
  summary Field in which SAGA search the path where the elastic certificate is. FOR ELASTIC version 8. name caFilePath

• Parameter
  summary Field in indices used as timestamp, automatically included when fetching data. name timestamp

• Parameter
  summary Time in seconds, of how much to wait between retries default 5 name delay

• Parameter
  summary In case of a connection issue, how many time it will try before throwing an error default 3 name retries

• Parameter
  summary Fields in indices to include when fetching data. name include

• Parameter
  summary Fields in indices to exclude when fetching data. name exclude

• Parameter
  summary Maximum amount of results to return per request. By default is -1, which means the engine's default will be not overwritten default -1 name maxResults type integer

• Parameter
  summary Indicates, if the engine must track always the real total of results available, otherwise it will return an estimated beyond certain point (e.g. gt 10000)) default true name track_total_hits type boolean

"providers": [
	{
		"name": "saga-provider",
  		"type": "Elastic",
		"nodeUrls": ["http://localhost:9200"],
		"timestamp": "updatedAt",
		"indexName": "saga",
 		"caFilePath": "", 
	    "timeout": 90,
		"delay": 5,
		"retries": 3,
		"include": [],
		"exclude": [],
		"track_total_hits": true,
		"maxResults": 10000
	}
]

Step-by-step guide (Basic Authentication)

To connect to an Elasticsearch with basic authentication you will need to encrypt your password and add the right values to the configuration.

Follow the next steps:

1. Encrypt the password.
   1. Locate on <saga>/bin the file saga-secure-<version>.jar

   2. In a terminal run, in Saga's root folder.

      Code Block
      language text title Console / Terminal / Command
      java -jar bin/saga-secure-<version>.jar -ep=<password> -config="config/config.json"

   3. Keep the generated pwd.txt file at hand, you'll need to reference it in the configuration.
2. Update the configuration in <saga>/config/config.json.

   1. Update "providers" using authentication factor as "basic".

      Code Block
      language text theme FadeToGreyjs
      "providers": [ { "name": "saga-provider", "type": "Elastic", "nodeUrls": ["http://localhost:9200"], "timestamp": "updatedAt", "indexName": "saga", "encryptionKeyFile" : "./bin/saga.ek", "caFilePath": "",   "authentication": "basic", "user": "<username>", "password": "<path_to_pwd_file>", "timeout": 90, "delay": 5, "retries": 3, "exclude": [] } ]

      Note
      Notice the values of "user", "password" and "encryptionKeyFile"

Step-by-step guide (Azure AD)

To connect to an Elasticsearch with Azure AD authentication you will need to set your Azure AD tenant and then you register your app in the Azure portal so the Microsoft identity platform can provide authentication and authorization services.

Follow the next steps:

1. Set your Azure AD tenant.
   1. You can see how to do that here.
2. Register your app on Azure portal.
   1. You can see how to do that here.
3. Update the configuration in <saga>/config/config.json.

   1. Update "providers" using authentication factor as "azure".

      Code Block
      theme language text js FadeToGrey
      "providers": [ { "name": "filesystem-provider", "type": "FileSystem", "baseDir": "./config" }, { "name": "saga-provider", "type": "Elastic", "indexName": "saga", "nodeUrls": ["http://localhost:9200"], "caFilePath": "",       "authentication":"azure", "resourceId": "<resourceID>", "tenantId": "<tenantID>" "clientId": "<clientID>", "clientSecret": "<clientSecretKey>", timeout": 90, "delay": 5, "retries": 3, "maxResults": 2000000, "exclude": [] } ]

      Note
      Notice the values of "resourceId", "tenantId", "clientId" and "clientSecret" using the values provided by the Azure Portal

Step-by-step guide (AWS)

To connect to an Elasticsearch with AWS authentication you will need to set your Amazon Web Services credentials locally as environmental variables or get them from the ECS or EC2 credentials.

Follow the next steps:

1. Set your AWS credentials.
   1. Set your credentials using the AWS CLI. You can see how to do that here.
   2. Or, you can load credentials from you ECS or EC2 instance. With IAM roles for Amazon ECS Tasks, you can specify an IAM role that can be used by the containers in a task to access AWS resources.

      
      

      Info
      title Note
      SagaElasticIndexer gets the credentials automatically by getting the credentials file. That is why you only need to specify region and service in the config file which is below this note.

2. Update the configuration in <saga>/config/config.json.

   1. Update "providers" using authentication factor as "aws".

   2. Then we have 2 options with Credential Chain Provider, or with Access & Secret Keys

      1. With Credential Chain Provider
         
         

         Code Block
         language text theme FadeToGreyjs
         "providers": [ { "name": "saga-provider", "type": "Elastic", "nodeUrls": ["http://localhost:9200"], "timestamp": "updatedAt", "indexName": "saga", "encryptionKeyFile" : "./bin/saga.ek", "caFilePath": "",   "authentication": "aws", "awsRegion": "<region_where_the_aws_service_is_located>", "awsService": "<aws_service>", "useCredentialsProviderChain": true, "timeout": 90, "delay": 5, "retries": 3, "exclude": [] } ]

         Note
         Notice the values of "awsService", "awsRegion" and "useCredentialsProviderChain"

         
         

         
         

      2. With Access & Secret Keys
         
         

         Code Block
         language textjs theme FadeToGrey
         "providers": [ { "name": "saga-provider", "type": "Elastic", "nodeUrls": ["http://localhost:9200"], "timestamp": "updatedAt", "indexName": "saga", "encryptionKeyFile" : "./bin/saga.ek", "caFilePath": "",   "authentication": "aws", "awsRegion": "<region_where_the_aws_service_is_located>", "awsService": "<aws_service>", "awsAccessKey": "<service_access_key>", "awsSecretKey": "<service_secret_key>", "useCredentialsProviderChain": false, "timeout": 90, "delay": 5, "retries": 3, "exclude": [] } ]

         Note
         Notice that now we also have the values of "awsAccessKey" and "awsSecretKey". You can ommit "useCredentialsProviderChain" if you want to

In case of Aspire Saga Parse (or any component using Saga)

If you are using Saga within Aspire, the configurations are the same for providers, but you will need to:

1. Copy Saga's encryption key file to <aspire>/bin/

2. Copy pwd.txt to <aspire>/bin/ as well.

3. Update the Saga's config file (the one within the Aspire configuration folder) to reflect the relative path of those files:

   Code Block
   language text theme FadeToGreyjs
   { "config": { "security": { "encryptionKeyFile": "./bin/saga.ek" }, "libraryJars": [ "./lib" ], ... "providers": [ { "name": "filesystem-provider", "type": "FileSystem", "baseDir": "./config" }, { "name": "saga-provider", "type": "Elastic", "nodeUrls": ["http://localhost:9200"], "timestamp": "updatedAt", "indexName": "saga", "encryptionKeyFile" : "./bin/saga.ek", "caFilePath": "",     "authentication": "basic", "user": "<username>", "password": "<path_to_pwd_file>", "timeout": 90, "delay": 5, "retries": 3, "maxResults": 2000000 "exclude": [] } ] }

Connect to Elasticsearch 8 with security enabled (HTTPS)

To connect to an Elasticsearch running with security enabled (as it is by default in version 8.X) you need to add a certificate to SAGA, so SAGA can use it to make the connection to Elastic, in order to do that the steps are these:

1. Create a certificate authority for Elastic. (If you already have one, skip to step 3)
   1. Go to the folder where elasticsearch binaries are, the place where you execute elasticsearch. 
      1. If you are using docker images, the same applies, you can always create the certificates and extract them from the container.
   2. Execute the next command (in any terminal you like) to create our new certificate:

      1. NOTE: Save the zip file into the "certs" folder of Elasticsearch binaries root (for convenience).

         1. Code Block
            language powershell
            elasticsearch-certutil ca --pem -out <PATH_WHERE_YOU_WANT_THE_CERTS>\<NAME_OF_CERTIFICATE_AUTHORITY>.zip Example:elasticsearch-certutil ca --pem -out C:\dev\Elastic\elasticsearch-8.8.1\config\certs\ca.zip

   3. Unzip the zip with our certificate authority.
2. Create the certificate for Elastic.
   1. In the same folder where the elasticsearch binaries are, even on the same termnial where you just run the earlier command, run this other command to create the certificate:

   2. Code Block
      language powershell
      elasticsearch-certutil cert -out <PATH_WHERE_YOU_WANT_THE_CERTS>\<NAME_OF_CERTIFICATE>.zip --name <NAME_OF_CERT> --ca-cert <PLACE_OF_CA_CREATED_EARLIER>.crt --ca-key <PLACE_OF_CA_CREATED_EARLIER>.key --pem Example:elasticsearch-certutil ca --pem -out C:\dev\Elastic\elasticsearch-8.8.1\config\certs\elastic.zip --name elastic --ca-cert C:\dev\Elastic\elasticsearch-8.8.1\config\certs\ca.crt --ca-key C:\dev\Elastic\elasticsearch-8.8.1\config\certs\ca.key

3. Update Elastic configuration (on the elasticsearch.yml file inside the config folder where elastic is stored):

   1.  The paths are on base the config folder inside Elastic.

      Code Block
      language yml theme DJangotext title Extract of elasticsearch.yml
      # Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents xpack.security.http.ssl: enabled: true certificate: certs/<REST_OF_THE_PATH_OF_CERTIFICATE>.crt key: certs/<REST_OF_THE_PATH_OF_CERTIFICATE>.key certificate_authorities: certs/<REST_OF_THE_PATH_OF_CERTIFICATE_AUTHORITY>.crt

4. Add the certificate path to the SAGA config file.
   1. On the "caFilePath" key you enter the path where the new certificate is.

      1. Code Block
         language textjs title Extract of SAGA config file
         { "name": "saga-provider", "type": "Elastic", "nodeUrls": ["https://localhost:9200"], "timestamp": "updatedAt", "indexName": "saga", "encryptionKeyFile" : "./bin/saga.ek", "authentication": "basic", "username": "<ELASTIC_USERNAME>", "password": "<ELASTIC_PASSWORD>", "caFilePath": "<PATH_OF_NEW_CERT>", "timeout": 90, "delay": 5, "retries": 3, "exclude": [ ] }

      2. Please note the authentication, which is "basic" to add the elastic username and password (that are generated by Elastic automatically the first time starts, you can overwrite the password).
      3. Also, the recommended path for the certificate is on the bin folder inside the SAGA_HOME.
5. Start SAGA as usual.

Related articles