...
...
Base Configuration (Without Authentication)
Parameters
Parameter |
---|
summary | The name of the provider, used in the pipeline configuration when required by stages. |
---|
default | saga-provider |
---|
name | name |
---|
required | true |
---|
|
Parameter |
---|
summary | The provider type. |
---|
default | Elastic |
---|
name | type |
---|
required | true |
---|
|
See Resources for more information. Parameter |
---|
summary | Name for the prefix for each index used in the Elasticsearch server. |
---|
default | saga |
---|
name | indexName |
---|
|
Parameter |
---|
summary | List of Elasticsearch hosts and ports (including schema) |
---|
default | http://localhost:9200 |
---|
name | nodeUrls |
---|
|
Parameter |
---|
summary | Authentication to use, it could be basic, aws or azure. |
---|
default | none |
---|
name | authentication |
---|
|
Tip |
---|
If none, it can be omitted |
Parameter |
---|
summary | Field in which SAGA search the path where the elastic certificate is. FOR ELASTIC version 8. |
---|
name | caFilePath |
---|
|
Parameter |
---|
summary | Disables SSL certificate validation when connecting to ElasticSearch. (ONLY USE IT FOR DEVELOPMENT PURPOSES) |
---|
default | false |
---|
name | trustAllSSL |
---|
|
Parameter |
---|
summary | Field in indices used as timestamp, automatically included when fetching data. |
---|
name | timestamp |
---|
|
Parameter |
---|
summary | Time in seconds, of how much to wait between retries |
---|
default | 5 |
---|
name | delay |
---|
|
Parameter |
---|
summary | In case of a connection issue, how many time it will try before throwing an error |
---|
default | 3 |
---|
name | retries |
---|
|
Parameter |
---|
summary | Fields in indices to include when fetching data. |
---|
name | include |
---|
|
Parameter |
---|
summary | Fields in indices to exclude when fetching data. |
---|
name | exclude |
---|
|
Parameter |
---|
summary | Maximum amount of results to return per request. By default is -1, which means the engine's default will be not overwritten |
---|
default | -1 |
---|
name | maxResults |
---|
type | integer |
---|
|
Parameter |
---|
summary | Indicates, if the engine must track always the real total of results available, otherwise it will return an estimated beyond certain point (e.g. gt 10000)) |
---|
default | true |
---|
name | track_total_hits |
---|
type | boolean |
---|
|
Code Block |
---|
|
"providers": [
{
"name": "saga-provider",
"type": "Elastic",
"nodeUrls": ["http://localhost:9200"],
"timestamp": "updatedAt",
"indexName": "saga",
"caFilePath": "",
"trustAllSSL": false,
"timeout": 90,
"delay": 5,
"retries": 3,
"include": [],
"exclude": [],
"track_total_hits": true,
"maxResults": 10000
}
] |
Step-by-step guide (
...
Basic Authentication)
To connect to an Elasticsearch with basic authentication you will need to encrypt your password and add the right values to the configuration.
Follow the next steps:
- Encrypt the password.
- Locate on <saga>/bin the file saga-secure-<version>.jar
In a terminal run, in Saga's root folder.
Code Block |
---|
language | text |
---|
title | Console / Terminal / Command |
---|
|
java -jar bin/saga-secure-<version>.jar -ep=<password> -config="config/config.json" |
- Keep the generated pwd.txt file at hand, you'll need to reference it in the configuration.
- Update the configuration in <saga>/config/config.json.
Update "providers" using authentication factor as "basic".
Code Block |
---|
|
"providers": [
{
"name": "saga-provider",
"type": "Elastic",
"nodeUrls": ["http://localhost:9200"],
"timestamp": "updatedAt",
"indexName": "saga",
"encryptionKeyFile" : "./bin/saga.ek",
"caFilePath": "",
"trustAllSSL": false,
"authentication": "basic",
"user": "<username>",
"password": "<path_to_pwd_file>",
"timeout": 90,
"delay": 5,
"retries": 3,
"exclude": []
}
] |
Note |
---|
Notice the values of "user", "password" and "encryptionKeyFile" |
Step-by-step guide (Azure AD)
To connect to an Elasticsearch with Azure AD authentication you will need to set your Azure AD tenant and then you register your app in the Azure portal so the Microsoft identity platform can provide authentication and authorization services.
Follow the next steps:
- Set your Azure AD tenant.
- You can see how to do that here.
- Register your app on Azure portal.
- You can see how to do that here.
- Update the configuration in <saga>/config/config.json.
Update "providers" using authentication factor as "azure".
Code Block |
---|
|
"providers": [
{
"name": "filesystem-provider",
"type": "FileSystem",
"baseDir": "./config"
},
{
"name": "saga-provider",
"type": "Elastic",
"indexName": "saga",
"nodeUrls": ["http://localhost:9200"],
|
...
...
...
...
...
...
...
...
...
...
...
...
...
...
<clientSecretKey>",
timeout": 90,
"delay": 5,
"retries": 3,
"maxResults": |
...
2000000,
"exclude": []
}
] |
Note |
---|
Notice the values of " |
...
resourceId", "tenantId", "clientId" and " |
...
Update "solutions"
...
clientSecret" using the values provided by the Azure Portal |
Step-by-step guide (AWS)
To connect to an Elasticsearch with AWS authentication you will need to set your Amazon Web Services credentials locally as environmental variables or get them from the ECS or EC2 credentials.
Follow the next steps:
- Set your AWS credentials.
- Set your credentials using the AWS CLI. You can see how to do that here.
- Or, you can load credentials from you ECS or EC2 instance. With IAM roles for Amazon ECS Tasks, you can specify an IAM role that can be used by the containers in a task to access AWS resources.
Info |
---|
|
SagaElasticIndexer gets the credentials automatically by getting the credentials file. That is why you only need to specify region and service in the config file which is below this note. |
- Update the configuration in <saga>/config/config.json.
Update "providers" using authentication factor as "aws".
- Then we have 2 options with Credential Chain Provider, or with Access & Secret Keys
With Credential Chain Provider
Code Block |
---|
|
"providers": [
{
"name": "saga-provider",
"type": "Elastic",
"nodeUrls": ["http://localhost:9200"],
"timestamp": "updatedAt",
"indexName": "saga",
"encryptionKeyFile" : "./bin/saga.ek",
"caFilePath": "",
"trustAllSSL": false,
"authentication": "aws",
"awsRegion": "<region_where_the_aws_service_is_located>",
"awsService": "<aws_service>",
"useCredentialsProviderChain": true,
"timeout": 90,
"delay": 5,
"retries": 3,
"exclude": []
}
] |
Note |
---|
Notice the values of "awsService", "awsRegion" and "useCredentialsProviderChain" |
With Access & Secret Keys
Code Block |
---|
|
"providers": [
{
"name": "saga-provider",
"type": "Elastic",
"nodeUrls": ["http://localhost:9200"],
"timestamp": "updatedAt",
"indexName": "saga",
"encryptionKeyFile" : "./bin/saga.ek",
"caFilePath": "",
"trustAllSSL": false,
"authentication": |
...
...
<region_where_the_aws_service_is_located>",
|
...
...
...
...
...
...
...
...
...
"<service_secret_key>",
"useCredentialsProviderChain": false,
"timeout": |
...
...
...
...
Note |
---|
Notice that now we also have |
...
...
...
awsSecretKey". You can ommit "useCredentialsProviderChain" if you want to |
...
Parse (or any component using Saga)
If you are using Saga within Aspire, the configurations are the same for providers, but you will need to:
Copy Saga's encryption key file to <aspire>/bin/
Copy pwd.txt to <aspire>/bin/ as well.
Update the Saga's config file (the one within the Aspire configuration folder)
...
to reflect the relative path of those files:
...
...
|
{
"config": {
"security": {
"encryptionKeyFile": "./bin/saga.ek"
},
"libraryJars": [
"./lib"
],
...
"providers": [
{
"name": "filesystem-provider",
"type": "FileSystem",
"baseDir": "./config"
},
{
"name": "saga-provider",
"type": "Elastic",
" |
...
...
["http://localhost:9200"],
" |
...
...
"updatedAt",
"indexName": "saga",
"encryptionKeyFile" |
...
...
bin/saga.ek",
"caFilePath": "",
|
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
- Notice the values of "encryptionKeyFile" and "password".
Connect to Elasticsearch 8 with security enabled (HTTPS)
To connect to an Elasticsearch
...
Follow the next steps:
...
Info |
---|
|
SagaElasticIndexer gets the credentials automatically by getting the credentials file. That is why you only need to specify region and service in the config file which is below this note. |
...
Update "providers" using authentication factor as "aws".
...
running with security enabled (as it is by default in version 8.X) you need to add a certificate to SAGA, so SAGA can use it to make the connection to Elastic, in order to do that the steps are these:
- Create a certificate authority for Elastic. (If you already have one, skip to step 3)
- Go to the folder where elasticsearch binaries are, the place where you execute elasticsearch.
- If you are using docker images, the same applies, you can always create the certificates and extract them from the container.
- Execute the next command (in any terminal you like) to create our new certificate:
NOTE: Save the zip file into the "certs" folder of Elasticsearch binaries root (for convenience).
Code Block |
---|
|
elasticsearch-certutil ca --pem -out <PATH_WHERE_YOU_WANT_THE_CERTS>\<NAME_OF_CERTIFICATE_AUTHORITY>.zip
Example:elasticsearch-certutil ca --pem -out C:\dev\Elastic\elasticsearch-8.8.1\config\certs\ca.zip |
- Unzip the zip with our certificate authority.
- Create the certificate for Elastic.
- In the same folder where the elasticsearch binaries are, even on the same termnial where you just run the earlier command, run this other command to create the certificate:
Code Block |
---|
|
elasticsearch-certutil cert -out <PATH_WHERE_YOU_WANT_THE_CERTS>\<NAME_OF_CERTIFICATE>.zip --name <NAME_OF_CERT> --ca-cert <PLACE_OF_CA_CREATED_EARLIER>.crt --ca-key <PLACE_OF_CA_CREATED_EARLIER>.key --pem
Example:elasticsearch-certutil ca --pem -out C:\dev\Elastic\elasticsearch-8.8.1\config\certs\elastic.zip --name elastic --ca-cert C:\dev\Elastic\elasticsearch-8.8.1\config\certs\ca.crt --ca-key C:\dev\Elastic\elasticsearch-8.8.1\config\certs\ca.key |
- Update Elastic configuration (on the elasticsearch.yml file inside the config folder where elastic is stored):
The paths are on base the config folder inside Elastic.
Code Block |
---|
language | yml |
---|
theme | DJango |
---|
title | Extract of elasticsearch.yml |
---|
|
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: true
certificate: certs/<REST_OF_THE_PATH_OF_CERTIFICATE>.crt
key: certs/<REST_OF_THE_PATH_OF_CERTIFICATE>.key
certificate_authorities: certs/<REST_OF_THE_PATH_OF_CERTIFICATE_AUTHORITY>.crt |
- Add the certificate path to the SAGA config file.
- On the "caFilePath" key you enter the path where the new certificate is.
Code Block |
---|
language | js |
---|
title | Extract of SAGA config file |
---|
|
{
"name": "saga-provider",
"type": "Elastic |
...
...
...
...
...
...
timestamp": "updatedAt",
" |
...
...
...
...
...
...
...
- Notice the values of "service" and "region".
Update "solutions" using authentication factor as "aws".
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
"retries": 3,
"exclude": [ |
...
...
- Please note the authentication, which is "basic" to add the elastic username and password (that are generated by Elastic automatically the first time starts, you can overwrite the password).
- Also, the recommended path for the certificate is on the bin folder inside the SAGA_HOME.
- Start SAGA as usual
...
Related articles
Content by Label |
---|
showLabels | false |
---|
max | 5 |
---|
spaces | saga131 |
---|
showSpace | false |
---|
sort | modified |
---|
reverse | true |
---|
type | page |
---|
cql | label in ("elasticsearch","configuration","authentication","aspire","providers") and type = "page" and space = "saga131" |
---|
labels | Elasticsearch |
---|
|