All properties can be passed either as a environment variable or as a JVM parameter. The "." and "_" characters can be replaced with each other.
Code Block | ||||
---|---|---|---|---|
| ||||
SET aspire_.noSql_.elastic_.server=http://localhost:9200 //Setting a property as a environment variable java -Daspire.noSql.elastic.server=http://localhost:9200 //Passing a property like JVM parameter |
Warning |
---|
In the case of environment variables that , property name should always use "_" instead of "." |
Bellow are the list of properties to configure the Elasticsearch NoSQL Provider
In Linux based systems, the memory usage can be changed with the following environment variables
Parameter | Required | ParameterDefault | Description | |
---|---|---|---|---|
aspire. noSqlmax.elastic heap.server | http://localhost:9200 | (Required) The elasticsearch server url to use. It can be multiple urls separated by "," | ||
aspire.noSql.elastic.authentication.basic | false | (Optional) Enables elasticsearch basic authentication | ||
memory | no | 2g | Maximum Heap the JVM will request and use from the host system | |
aspire.max.metaspace.size | no | 256m | Maximum metaspace the JVM will request and use from the host system. |
In windows-based systems, if using the aspire.bat file, the memory usage should be changed in the aspire.bat itself. Otherwise, if it is running as a Windows Service:
HKEY_LOCAL_MACHINE / SOFTWARE / WOW6432Node / Apache Software / Procrun
2.0
/ AspireService / Parameters / Java
Below are the list of properties to configure the Elasticsearch NoSQL Provider
Parameter | Required | Default | New & Available from | Description | |||
---|---|---|---|---|---|---|---|
aspire.noSql.elastic.user | null | (Optional) The elasticsearch user to use for basic authentication | |||||
aspire.noSql.elastic.password | null | (Optional) The elasticsearch password to use for basic authentication||||||
aspire.noSql.elastic. | authentication.awsserver | false | (Optional) Enables elasticsearch AWS authenticationyes | http://localhost:9200 | The Elasticsearch server URL to use. It can be multiple URLs separated by "," | ||
aspire.noSql.elastic.authentication. | useCredentialsProviderChainbasic | no | false | (Optional) Option to use the AWS credentials provider chain to get the credentialsEnables Elasticsearch basic authentication | |||
aspire.noSql.elastic. | aws.access.keyuser | no | null | (Optional) AWS access key forThe Elasticsearch user to use for basic authentication | |||
aspire.noSql.elastic. | aws.secret.keypassword | no | null | (Optional) AWS secret key forThe Elasticsearch password to use for basic authentication | |||
aspire.noSql.elastic.authentication.aws | .regionno | null | (Optional) The AWS regionfalse | Enables Elasticsearch AWS authentication | |||
aspire.noSql.elastic. | keepSearchContextAlive5m | (Optional) The amount of time to keep elasticsearch scrolls requests search context alive using "m" as a unitauthentication.useCredentialsProviderChain | no | false | Option to use the AWS credentials provider chain to get the credentials | ||
aspire.noSql.elastic. | maxRequestSize10485760B | (Optional) The maximum size for a bulk request. The value can be specified in B, K, M and G unitsaws.assumeRole | no | false | 5.0.3 | If a role must be assumed to access Elasticsearch. Must be true/false | |
aspire.noSql.elastic. | maxConnections100 | (Optional) The maximum amount of connections to mantain openaws.roleArn | no | null | 5.0.3 | The Role ARN to be assumed. | |
aspire.noSql.elastic. | maxConnectionsPerRoute10 | (Optional) The maximum amount of connections per serveraws.access.key | no | null | AWS access key for authentication | ||
aspire.noSql.elastic | .maxRetries3 | (Optional) The amount of times to retry each request.aws.secret.key | no | null | AWS secret key for authentication | ||
aspire.noSql.elastic.aws. | retriesWaitTimeregion | 5000 | (Optional) The time to wait in ms between retriesno | null | The AWS region | ||
aspire.noSql.elastic. | useThrottlingkeepSearchContextAlive | no | false | (Optional) Enables requests throttling to elasticsearch5m | The amount of time to keep Elasticsearch scrolls requests search context alive using "m" as a unit | ||
aspire.noSql.elastic. | throttlingRatemaxRequestSize | no | 500010485760B | (Optional) The throttling rate in msThe maximum size for a bulk request. The value can be specified in B, K, M and G units | |||
aspire.noSql.elastic. | throttlingConnectionRatemaxConnections | no | 500100 | (Optional)The maximum | amount of requests allowed in the period specified by the throttlingRatenumber of connections to keep open. | ||
aspire.noSql.elastic. | waitTime429maxConnectionsPerRoute | no | 300010 | (Optional) The time to wait in ms before retrying a 429 errorThe maximum number of connections per server | |||
aspire.noSql.elastic. | bulkreadTimeout | no | true | (Optional) Enables using bulk for requests30000 | The socket timeout. | ||
aspire.noSql.elastic. | bulkSizeconnectionTimeout | no | 50015000 | (Optional) The maximum number of documents to include in a bulk requestThe connection timeout. | |||
aspire.noSql.elastic. | bulkInactivityTimeoutmaxRetries | 5 | (Optional) The inactivity in s before flushing a bulkno | 3 | The number of times to retry each request | ||
aspire.noSql.elastic. | bulkRegularTimeoutretriesWaitTime | no | 305000 | (Optional)The | maximum amount oftime | in s for a bulk request to be kept in memory before flushingto wait in ms between retries | |
aspire.noSql.elastic. | debugFileuseThrottling | no | null | (Optional) The path to the debug file, request to ES are logged in this filefalse | Enables requests throttling to Elasticsearch | ||
aspire.noSql.elastic. | mappingFilethrottlingRate | no | null5000 | (Optional) The file path (including file name) that includes the mapping for the indexes used by Aspire. By default mapping included in the provider is usedThe throttling rate in ms | |||
aspire.noSql.elastic. | index.prefixaspire | (Optional) The prefix to use for the indexes created by the provider | |||||
aspire.noSql.elastic.debug | false | (Optional) Enables debug logging information | |||||
aspire.noSql.elastic.usePooling | true | (Optional) Enables HTTP connection pooling |
Bellow are the list of properties to configure the SSL Certificates
Bellow are the list of properties related to the Aspire security and encryption
throttlingConnectionRate | no | 500 | The maximum number of requests allowed in the period specified by the throttlingRate | |
aspire.noSql.elastic.waitTime429 | no | 3000 | The time to wait in ms before retrying a 429 error | |
aspire.noSql.elastic.bulk | no | true | Enables using bulk for requests | |
aspire.noSql.elastic.bulkSize | no | 500 | The maximum number of documents to include in a bulk request | |
aspire.noSql.elastic.bulkInactivityTimeout | no | 5 | The inactivity in s before flushing a bulk request | |
aspire.noSql.elastic.bulkRegularTimeout | no | 30 | The maximum amount of time in s for a bulk request to be kept in memory before flushing | |
aspire.noSql.elastic.debugFile | no | null | The path to the debug file, request to ES are logged in this file | |
aspire.noSql.elastic.mappingFile | no | null | The file path (including file name) that includes the mapping for the indexes used by Aspire. By default, the mapping included in the provider is used | |
aspire.noSql.elastic.index.prefix | no | aspire | The prefix to use for the indexes created by the provider | |
aspire.noSql.elastic.debug | no | false | Enables debug logging information | |
aspire.noSql.elastic.usePooling | no | true | Enables HTTP connection pooling | |
aspire.nosql.elastic.timeSeriesType | no | index | 5.0.3, 5.3 (Opensearch) | Enables rollover possibility for audit, error, log time series index. Value "index" - default without rollover, Value "dataStream" - rollover based on data stream. Value "dataStreamOpensearch" - rollover based on Opensearch data stream. |
aspire.nosql.elastic.ilmPolicyFile | no | default provider policy file | 5.0.3 | The file path (including file name) that includes the ilm policy file for the indexes used by Aspire. Used only for the time series option "dataStream*". |
aspire.nosql.elastic.indexTemplateFile | no | default provider template file | 5.0.3 | The file path (including file name) that includes the template file for the indexes used by Aspire. Used only for the time series option "dataStream*". |
Each index sharding and replicas settings can be changed by using a custom mappingFile and configuring the aspire.noSql.elastic.mappingFile property.
Since 5.0.1, when using the default mapping settings, you can use the following prefix parameter:
If none of these properties are specified, the default replicas and shard number of the cluster will be used (usually 1 shard and 1 replica).
The different index names available are:
audit
base
errors
hierarchy
log
map
queue
retryLog
set
settings
snapshot
updateQueue
identityCache
Bellow are the list of properties to configure the SSL Certificates
Parameter | Required | Default | Description |
---|---|---|---|
aspire.ssl.trustAll | no | false | Configure if all certificates should be trusted |
aspire.ssl.overwriteFactory | no | false | Configure if the created key managers should overwrite the Java connection factory |
aspire.ssl.truststore.file | no | null | The path of the trust store file |
aspire.ssl.truststore.password | no | null | The trust store file password |
aspire.ssl.truststore.type | no | jks | The file format of the trust store file |
aspire.ssl.keystore.file | no | null | The path of the key store file |
aspire.ssl.keystore.password | no | null | The key store file password |
aspire.ssl.keystore.type | no | jks | The file format of the key store file |
Below are the list of properties related to the Aspire security and encryption
Parameter | Required | Default | Description |
---|---|---|---|
aspire.ldap.bind.dn.password | no | null | The password of the User DN. Not required if the authentication is anonymous |
aspire.security.api.auditing | no | true | It audits the Aspire API calls. It is available from Aspire 5.1 and on. |
When a password/secret/token in a configuration must be persisted, Aspire encrypts it and stores it encrypted. The default encryption mechanism is AES 256 given a local key. AWS KMS encryption can be used instead as of version 5.0.3
Parameter | Required | Default | Description |
---|---|---|---|
aspire.encryption.key.file | no | null | (Optional) Path (including file name) where the encryption key is located, if not provided a default in-memory key will be used, for production installations it must be always provided. This can also be passed as a JVM parameter or as an environment variable, aspire_encryption_key_file. This should be a 32 byte file, if longer, the first 32 bytes will be used as the encryption key. Grant read access to the Aspire user only (chmod 400 <file>) This file could be generated randomly
|
Available since Aspire 5.0.3, Uses AWS Key Management Service (KMS) to encrypt the sensitive data. It uses a key in KMS to encrypt and decrypt data. See more details about this encryption provider at Aspire KMS encryption
Parameter | Required | Default | Description |
---|---|---|---|
aspire.encryption.kms.roleARN | no | null | (Optional) If the KMS service must be accessed through the assumption of an IAM role, specify the role ARN. |
aspire.encryption.kms.keyARN | yes | N/A | The KMS key ARN. See Aspire KMS encryption for more information about creating a KMS key for Aspire. |
aspire.encryption.kms.region | yes | N/A | The AWS region on which the KMS service will be used |
aspire.encryption.kms.accessKey | no | null | (Optional) Specify the access key if static credentials must be used. If this is not specified, the Default Credential Provider Chain will be used. |
aspire.encryption.kms.secretKey | no | null | (Optional) Specify the secret key if static credentials must be used. If this is not specified, the Default Credential Provider Chain will be used. |
These properties will be used by all worker nodes in the cluster.
Parameter | Required | Default | Description |
---|---|---|---|
aspire.node.worker.maxMemQueueSize | yes | 4000 | The maximum number of items to keep in the in-memory queue |
aspire.node.worker.queueSizeThreshold | yes | 0.5 | The capacity threshold of the in memory queue before requesting more items to the managers |
aspire.node.worker.cleanUpWaitTime | yes | 300000 | The wait time in ms for the thread that checks the connectors clean up threshold |
aspire.node.worker.cleanUpThreshold | yes | 3600000 | The time in ms for a connector to be idle before being removed from memory |
aspire.node.worker.maxEnqueueRetries | yes | 5 | The number of retries to enqueue an item into the framework pipeline |
aspire.node.worker.workflow.appCleanUpWaitTime | yes | 60000 | The wait time in ms for the thread that checks the workflow application's clean-up threshold |
aspire.node.worker.workflow.appCleanUpThreshold | yes | 3600000 | The time in ms for a workflow application to be idle before being removed from memory |
aspire.node.worker.tags | no | The tags of the worker node. These tags will determine which items this node can process. It should be a comma separated list of tags. | |
aspire.node.worker.entryProcessorBaseSleep | yes | 10 | The base sleep time in ms for the thread in charge of queuing received items into the connector framework pipelines |
aspire.node.worker.entryProcessorMaxSleep | yes | 2000 | The maximum sleep in ms for the thread in charge of queuing received items into the connector framework pipelines |
aspire.node.worker.entryProcessorMaxIterations | yes | 5 | The number of iterations without queuing items before in increasing the sleep time |
aspire.node.worker.entryProcessorMultiplier | yes | 1.25 | The multiplier used to increase the sleep time after the specified iterations without queuing items |
aspire.node.worker.batchLoaderBaseSleep | yes | 10 | The base sleep time in ms for the thread in charge of requesting batches to the manager nodes |
aspire.node.worker.batchLoaderMaxSleep | yes | 2000 | The maximum sleep in ms for the thread in charge of requesting batches to the manager nodes |
aspire.node.worker.batchLoaderMaxIterations | yes | 5 | The number of iterations without receiving batches from the managers nodes before in increasing the sleep time |
aspire.node.worker.batchLoaderMultiplier | yes | 1.25 | The multiplier used to increase the sleep time after the specified iterations without receiving batches from the managers nodes |
aspire.node.worker.connectionTimeout | yes | 20000 | The connection timeout for requests to other aspire nodes |
aspire.node.worker.socketTimeout | yes | 20000 | The socket timeout for requests to other aspire nodes |
aspire.node.worker.maxRetries | yes | 3 | The number of retries for requests to other aspire nodes |
aspire.node.worker.proxyHost | no | null | The proxy host to use for requests to other aspire nodes |
aspire.node.worker.proxyPort | no | 0 | The proxy port to use for requests to other aspire nodes. Must be provided if the proxyHost is configured |
aspire.node.worker.proxyUser | no | null | The proxy user to use for requests to other aspire nodes |
aspire.node.worker.proxyPassword | no | null | The proxy password to use for requests to other aspire nodes. |
aspire.node.worker.pingFrequency | yes | 15000 | The frequency for the node to ping to Elasticsearch. The pings are used to determine if a node is alive and working properly |
aspire.node.worker.nodeFailureTimeout | yes | 30000 | The ping timeout used to determine if a node is not working. The node will be marked as failed in this case and the node eventually will shut down itself |
These properties will be used by all manager nodes in the cluster.
Parameter | Required | Default | Description |
---|---|---|---|
aspire.node.manager.scanBatchCreatorBaseSleep | yes | 30 | The base sleep time in ms for the thread in charge of creating batches from the scan queue |
aspire.node.manager.scanBatchCreatorMaxSleep | yes | 2000 | The maximum sleep in ms for the thread in charge of creating batches from the scan queue |
aspire.node.manager.scanBatchCreatorMaxIterations | yes | 10 | The number of iterations without creating new scan batches before in increasing the sleep time |
aspire.node.manager.scanBatchCreatorMultiplier | yes | 1.25 | The multiplier used to increase the sleep time after the specified iterations without creating new scan batches |
aspire.node.manager.processBatchCreatorBaseSleep | yes | 30 | The base sleep time in ms for the thread in charge of creating batches from the process queue |
aspire.node.manager.processBatchCreatorMaxSleep | yes | 2000 | The maximum sleep in ms for the thread in charge of creating batches from the process queue |
aspire.node.manager.processBatchCreatorMaxIterations | yes | 10 | The number of iterations without creating new process batches before in increasing the sleep time |
aspire.node.manager.processBatchCreatorMultiplier | yes | 1.25 | The multiplier used to increase the sleep time after the specified iterations without creating new process batches |
aspire.node.manager.crawlProgressManagerBaseSleep | yes | 100 | The base sleep time in ms for the thread in charge of monitoring active crawls |
aspire.node.manager.schedulerBaseSleep | yes | 10000 |
These properties will be used by all worker nodes in the cluster.
These properties will be used by all manager nodes in the cluster.
Parameter | Default | Description | |||
---|---|---|---|---|---|
com.accenture.aspire.node.manager.scanBatchCreatorBaseSleep | 200 | (Required) The base sleep time in ms for the thread in charge of creating batches from the scan queueexecuting seeds based on the configured schedules | |||
com.accenture.aspire.node.manager.scanBatchCreatorMaxSleepmaxBatches | yes | 100001000 | (Required) The maximum sleep in ms for the thread in charge of creating batches from the scan queuenumber of batches the manager will keep in memory | ||
com.accenture.aspire.node.manager.scanBatchCreatorMaxIterationsmaxBatchItems | yes | 10100 | (Required) The maximum number of iterations without creating new scan batches before in increasing the sleep timedocuments per batch | ||
com.accenture.aspire.node.manager.scanBatchCreatorMultiplierconnectionTimeout | yes | 220000 | (Required) The multiplier used to increase the sleep time after the specified iterations without creating new scan batches | The connection timeout for requests to other aspire nodes | |
com.accenture.aspire.node.manager.processBatchCreatorBaseSleepsocketTimeout | yes | 20020000 | (Required) The base sleep time in ms for the thread in charge of creating batches from the process queue | The socket timeout for requests to other aspire nodes | |
aspire.node.manager.maxRetries | yes | 3 | The number of retries for requests to other aspire nodes | ||
aspirecom.accenture.aspire.node.manager.processBatchCreatorMaxSleepproxyHost | no | 10000null | (Required) The maximum sleep in ms for the thread in charge of creating batches from the process queue | The proxy host to use for requests to other aspire nodes | |
com.accenture.aspire.node.manager.processBatchCreatorMaxIterationsproxyPort | no | 100 | (Required) The number of iterations without creating new process batches before in increasing the sleep time | The proxy port to use for requests to other aspire nodes. Must be provided if the proxyHost is configured | |
com.accenture.aspire.node.manager.processBatchCreatorMultiplierproxyUser | no | 2null | (Required) The multiplier used to increase the sleep time after the specified iterations without creating new process batches | The proxy user to use for requests to other aspire nodes | |
com.accenture.aspire.node.manager.crawlProgressManagerBaseSleepproxyPassword | no | 200null | (Required) The base sleep time in ms for the thread in charge of monitoring active crawls | The proxy password to use for requests to other aspire nodes. | |
com.accenture.aspire.node.manager.schedulerBaseSleeppingFrequency | yes | 1000015000 | (Required) The base sleep time in ms for the thread in charge of executing seeds based on the configured schedules | The frequency for the node to ping to Elasticsearch. The pings are used to determine if a node is alive and working properly | |
com.accenture.aspire.node.manager.maxBatchesnodeFailureTimeout | yes | 1000 | (Required) The maximum number of batches the manager will keep in memory | ||
com.accenture.aspire.node.manager.maxBatchItems | 100 | (Required) The maximum number of documents per batch | |||
30000 | The ping timeout used to determine if a node is not working. The node will be marked as failed in this case and the node eventually will shutdown itself | ||||
com.accenture.aspire.node.manager.connectionTimeoutinProgressJobTimeout | yes | 200003600000 | (Required) The connection timeout for requests to other aspire nodes | ||
com.accenture.aspire.node.manager.socketTimeout | 20000 | (Required) The socket timeout for requests to other aspire nodes | |||
com.accenture.aspire.node.manager.maxRetries | 3 | (Required) The number of retries for requests to other aspire nodes | |||
The maximum time in ms, a job can be in "in-progress" status before being released. Default is 1 hour | |||||
aspire.node.manager.inProgressJobTimeoutCheckFrequency | yes | 1800000 | How frequently to verify for timed-out "in-progress" jobs | ||
com.accenture.aspire.node.manager.proxyHostackCleanBaseSleep | no | null | (Optional) The proxy host to use for requests to other aspire nodes | 5000 | How frequently the manager should check for Acknowledged Batches that need to be removed from memory. |
com.accenture.aspire.node.manager.proxyPortproxyCall | no | 0 | (Optional) The proxy port to use for requests to other aspire nodes. Must be provided if the proxyHost is configured | false | Configure if you want to perform a proxy call to the main manager from a non-main manager and avoid redirect calls. |
com.accenture.aspire.node.manager.proxyUsertags | nullno | (Optional) The proxy user to use for requests to other aspire nodes | The tags of the manager node. These tags will determine which seeds this node can process. It should be a comma separated list of tags. | ||
com.accenture.aspire.node.manager.proxyPasswordworkerRoundRobin | null | (Optional) The proxy password to use for requests to other aspire nodes. | no | false | If round-robin should be applied when serving workers with batches |
com.accenture.aspire.node.manager.pingFrequencyworkerRoundRobinTimeout | no | 15000600000 | (Required) The ping timeout used to determine if a node is not working. The node will be marked as failed in this case and the node eventually will shutdown itself | ||
com.accenture.aspire.node.manager.nodeFailureTimeout | 30000 | (Required) The frequency for the node to ping to Elastisearch. The pings are used to determine if a node is alive and working properly | |||
com.accenture.aspire.node.manager.inProgressJobTimeout | 3600000 | (Optional) The maximum time in ms a job can be in "in-progress" status before being released. Default is 1 hour | |||
com.accenture.aspire.node.manager.inProgressJobTimeoutCheckFrequency | 1800000 | (Optional) How frequently to verify for timed-out "in-progress" jobs | |||
The time in ms after which the worker is considered timed out when round-robin is used. |
Since 5.0.3, the following properties are also available for managing the behavior of the user interface.
Parameter | Required | Default | Description |
---|---|---|---|
aspire.ui.refreshRate | no | 5s | How often trigger the auto-refresh for the listing pages. It can be defined with a time unit, i.e. 15s, 1 m. |
These properties will be used to generate the dashboard links in the UI
Parameter | Required | Default | Description |
---|---|---|---|
aspire.dashboards.enabled | no | false |
Enables the dashboard links on the UI |
aspire.dashboards.base |
no | Base URL to Kibana | |
aspire.dashboards.main | no |
Main |
Dashboard relative URL | |
aspire.dashboards.metrics | no |
Metrics Dashboard relative URL |