There are 3 different ways to connect with authentication: with Basic Authentication, Azure AD and Amazon Web Services (AWS).

Or, if you don't want any authentication, you can set that as well.

Base Configuration (Without Authentication)

Parameters

  • name ( type=string | default=saga-provider | required ) - The name of the provider, used in the pipeline configuration when required by stages.
  • type ( type=string | default=Elastic | required ) - The provider type. See Resources for more information.
  • indexName ( type=string | default=saga | optional ) - Name for the prefix for each index used in the Elasticsearch server.
  • nodeUrls ( type=string | default=http://localhost:9200 | optional ) - List of Elasticsearch hosts and ports (including schema)
  • authentication ( type=string | default=none | optional ) - Authentication to use, it could be basic, aws or azure.  

    • If none, it can be omitted 

  • caFilePath ( type=string | optional ) - Field in which SAGA search the path where the elastic certificate is. FOR ELASTIC version 8.
  • trustAllSSL ( type=string | default=false | optional ) - Disables SSL certificate validation when connecting to ElasticSearch. (ONLY USE IT FOR DEVELOPMENT PURPOSES)
  • timestamp ( type=string | optional ) - Field in indices used as timestamp, automatically included when fetching data.
  • delay ( type=string | default=5 | optional ) - Time in seconds, of how much to wait between retries
  • retries ( type=string | default=3 | optional ) - In case of a connection issue, how many time it will try before throwing an error
  • include ( type=string | optional ) - Fields in indices to include when fetching data.
  • exclude ( type=string | optional ) - Fields in indices to exclude when fetching data.
  • maxResults ( type=integer | default=-1 | optional ) - Maximum amount of results to return per request. By default is -1, which means the engine's default will be not overwritten
  • track_total_hits ( type=boolean | default=true | optional ) - Indicates, if the engine must track always the real total of results available, otherwise it will return an estimated beyond certain point (e.g. gt 10000))
"providers": [
	{
		"name": "saga-provider",
  		"type": "Elastic",
		"nodeUrls": ["http://localhost:9200"],
		"timestamp": "updatedAt",
		"indexName": "saga",
 		"caFilePath": "", 
	 	"trustAllSSL": false,
	    "timeout": 90,
		"delay": 5,
		"retries": 3,
		"include": [],
		"exclude": [],
		"track_total_hits": true,
		"maxResults": 10000
	}
]




Step-by-step guide (Basic Authentication)

To connect to an Elasticsearch with basic authentication you will need to encrypt your password and add the right values to the configuration.

Follow the next steps:

  1. Encrypt the password.
    1. Locate on <saga>/bin the file saga-secure-<version>.jar

    2. In a terminal run, in Saga's root folder.

      Console / Terminal / Command
      java -jar bin/saga-secure-<version>.jar -ep=<password> -config="config/config.json"
    3. Keep the generated pwd.txt file at hand, you'll need to reference it in the configuration.
  2. Update the configuration in <saga>/config/config.json.

    1. Update "providers" using authentication factor as "basic".

      "providers": [
	{
		"name": "saga-provider",
  		"type": "Elastic",
		"nodeUrls": ["http://localhost:9200"],
		"timestamp": "updatedAt",
		"indexName": "saga",
		"encryptionKeyFile" : "./bin/saga.ek",
 		"caFilePath": "",
	 	"trustAllSSL": false, 
	    "authentication": "basic",
		"user": "<username>",
		"password": "<path_to_pwd_file>",
		"timeout": 90,
		"delay": 5,
		"retries": 3,
		"exclude": []
	}
]

      Notice the values of "user", "password" and "encryptionKeyFile"

Step-by-step guide (Azure AD)

To connect to an Elasticsearch with Azure AD authentication you will need to set your Azure AD tenant and then you register your app in the Azure portal so the Microsoft identity platform can provide authentication and authorization services.

Follow the next steps:

  1. Set your Azure AD tenant.
    1. You can see how to do that here.
  2. Register your app on Azure portal.
    1. You can see how to do that here.
  3. Update the configuration in <saga>/config/config.json.

    1. Update "providers" using authentication factor as "azure".

      "providers": [
      {
        "name": "filesystem-provider",
        "type": "FileSystem",
        "baseDir": "./config"
      },
      {
        "name": "saga-provider",
        "type": "Elastic",
        "indexName": "saga",
        "nodeUrls": ["http://localhost:9200"],
 		"caFilePath": "",
	 	"trustAllSSL": false,
        "authentication":"azure",
        "resourceId": "<resourceID>",
        "tenantId": "<tenantID>"
        "clientId": "<clientID>",
        "clientSecret": "<clientSecretKey>",
		timeout": 90,
		"delay": 5,
		"retries": 3,
		"maxResults": 2000000,
		"exclude": []
	}
]

      Notice the values of "resourceId", "tenantId", "clientId" and "clientSecret" using the values provided by the Azure Portal

Step-by-step guide (AWS)

To connect to an Elasticsearch with AWS authentication you will need to set your Amazon Web Services credentials locally as environmental variables or get them from the ECS or EC2 credentials.

Follow the next steps:

  1. Set your AWS credentials.
    1. Set your credentials using the AWS CLI. You can see how to do that here.
    2. Or, you can load credentials from you ECS or EC2 instance. With IAM roles for Amazon ECS Tasks, you can specify an IAM role that can be used by the containers in a task to access AWS resources.


      Note

      SagaElasticIndexer gets the credentials automatically by getting the credentials file. That is why you only need to specify region and service in the config file which is below this note.

  2. Update the configuration in <saga>/config/config.json.

    1. Update "providers" using authentication factor as "aws".

    2. Then we have 2 options with Credential Chain Provider, or with Access & Secret Keys

      1. With Credential Chain Provider

        "providers": [
	{
		"name": "saga-provider",
		"type": "Elastic",
		"nodeUrls": ["http://localhost:9200"],
		"timestamp": "updatedAt",
		"indexName": "saga",
		"encryptionKeyFile" : "./bin/saga.ek",
 		"caFilePath": "",
		"trustAllSSL": false,
	    "authentication": "aws",
		"awsRegion": "<region_where_the_aws_service_is_located>",
		"awsService": "<aws_service>",
		"useCredentialsProviderChain": true,
		"timeout": 90,
		"delay": 5,
		"retries": 3,
		"exclude": []
	}
]

        Notice the values of "awsService", "awsRegion" and "useCredentialsProviderChain"



      2. With Access & Secret Keys

        "providers": [
	{
		"name": "saga-provider",
		"type": "Elastic",
		"nodeUrls": ["http://localhost:9200"],
		"timestamp": "updatedAt",
		"indexName": "saga",
		"encryptionKeyFile" : "./bin/saga.ek",
		"caFilePath": "",
		"trustAllSSL": false,
	    "authentication": "aws",
		"awsRegion": "<region_where_the_aws_service_is_located>",
		"awsService": "<aws_service>",
		"awsAccessKey": "<service_access_key>",
  		"awsSecretKey": "<service_secret_key>",
		"useCredentialsProviderChain": false,
		"timeout": 90,
		"delay": 5,
		"retries": 3,
		"exclude": []
	}
]

        Notice that now we also have the values of "awsAccessKey" and "awsSecretKey". You can ommit "useCredentialsProviderChain" if you want to

In case of Aspire Saga Parse (or any component using Saga)

If you are using Saga within Aspire, the configurations are the same for providers, but you will need to:

  1. Copy Saga's encryption key file to <aspire>/bin/

  2. Copy pwd.txt to <aspire>/bin/ as well.

  3. Update the Saga's config file (the one within the Aspire configuration folder) to reflect the relative path of those files:

    {
	"config": {
    	"security": {
			"encryptionKeyFile": "./bin/saga.ek"
        },
    	"libraryJars": [
            "./lib"
        ],
		...
	"providers": [
     	{
			"name": "filesystem-provider",
			"type": "FileSystem",
			"baseDir": "./config"
		},
 		{
			"name": "saga-provider",
  			"type": "Elastic",
			"nodeUrls": ["http://localhost:9200"],
			"timestamp": "updatedAt",
			"indexName": "saga",
			"encryptionKeyFile" : "./bin/saga.ek",
			"caFilePath": "",
			"trustAllSSL": false,
    		"authentication": "basic",
			"user": "<username>",
			"password": "<path_to_pwd_file>",
			"timeout": 90,
			"delay": 5,
			"retries": 3,
 			"maxResults": 2000000 
			"exclude": []
		} 

	]
}

Connect to Elasticsearch 8 with security enabled (HTTPS)


To connect to an Elasticsearch running with security enabled (as it is by default in version 8.X) you need to add a certificate to SAGA, so SAGA can use it to make the connection to Elastic, in order to do that the steps are these:


  1. Create a certificate authority for Elastic. (If you already have one, skip to step 3)
    1. Go to the folder where elasticsearch binaries are, the place where you execute elasticsearch. 
      1. If you are using docker images, the same applies, you can always create the certificates and extract them from the container.
    2. Execute the next command (in any terminal you like) to create our new certificate:

      1. NOTE: Save the zip file into the "certs" folder of Elasticsearch binaries root (for convenience).

        1. elasticsearch-certutil ca --pem -out <PATH_WHERE_YOU_WANT_THE_CERTS>\<NAME_OF_CERTIFICATE_AUTHORITY>.zip

Example:elasticsearch-certutil ca --pem -out C:\dev\Elastic\elasticsearch-8.8.1\config\certs\ca.zip
    3. Unzip the zip with our certificate authority.
  2. Create the certificate for Elastic.
    1. In the same folder where the elasticsearch binaries are, even on the same termnial where you just run the earlier command, run this other command to create the certificate:
    2. elasticsearch-certutil cert -out <PATH_WHERE_YOU_WANT_THE_CERTS>\<NAME_OF_CERTIFICATE>.zip --name <NAME_OF_CERT> --ca-cert <PLACE_OF_CA_CREATED_EARLIER>.crt --ca-key <PLACE_OF_CA_CREATED_EARLIER>.key --pem


Example:elasticsearch-certutil ca --pem -out C:\dev\Elastic\elasticsearch-8.8.1\config\certs\elastic.zip --name elastic --ca-cert C:\dev\Elastic\elasticsearch-8.8.1\config\certs\ca.crt --ca-key C:\dev\Elastic\elasticsearch-8.8.1\config\certs\ca.key
  3. Update Elastic configuration (on the elasticsearch.yml file inside the config folder where elastic is stored):

    1.  The paths are on base the config folder inside Elastic.

      Extract of elasticsearch.yml
      # Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  certificate: certs/<REST_OF_THE_PATH_OF_CERTIFICATE>.crt
  key: certs/<REST_OF_THE_PATH_OF_CERTIFICATE>.key
  certificate_authorities: certs/<REST_OF_THE_PATH_OF_CERTIFICATE_AUTHORITY>.crt
  4. Add the certificate path to the SAGA config file.
    1. On the "caFilePath" key you enter the path where the new certificate is.
      1. Extract of SAGA config file
        {
        "name": "saga-provider",
        "type": "Elastic",
        "nodeUrls": ["https://localhost:9200"],
        "timestamp": "updatedAt",
        "indexName": "saga",
        "encryptionKeyFile" : "./bin/saga.ek",
        "authentication": "basic",
        "username": "<ELASTIC_USERNAME>",
        "password": "<ELASTIC_PASSWORD>",
        "caFilePath": "<PATH_OF_NEW_CERT>",
		"trustAllSSL": false,
        "timeout": 90,
        "delay": 5,
        "retries": 3,
        "exclude": [ ]
      }
      2. Please note the authentication, which is "basic" to add the elastic username and password (that are generated by Elastic automatically the first time starts, you can overwrite the password).
      3. Also, the recommended path for the certificate is on the bin folder inside the SAGA_HOME.
  5. Start SAGA as usual.