All properties can be passed either as a environment variable or as a JVM parameter. The "." and "_" characters can be replaced with each other.
SET aspire.noSql.elastic.server=http://localhost:9200 //Setting a property as a environment variable java -Daspire.noSql.elastic.server=http://localhost:9200 //Passing a property like JVM parameter
In the case of environment variables, property name should always use "_" instead of "."
In Linux based systems, the memory usage can be changed with the following environment variables
Parameter | Required | Default | Description |
---|---|---|---|
aspire.max.heap.memory | no | 2g | Maximum Heap the JVM will request and use from the host system |
aspire.max.metaspace.size | no | 256m | Maximum metaspace the JVM will request and use from the host system. |
In windows-based systems, if using the aspire.bat file, the memory usage should be changed in the aspire.bat itself. Otherwise, if it is running as a Windows Service:
HKEY_LOCAL_MACHINE / SOFTWARE / WOW6432Node / Apache Software / Procrun
2.0
/ AspireService / Parameters / Java
Below are the list of properties to configure the Elasticsearch NoSQL Provider
Parameter | Required | Default | New & Available from | Description |
---|---|---|---|---|
aspire.noSql.elastic.server | yes | http://localhost:9200 | The Elasticsearch server URL to use. It can be multiple URLs separated by "," | |
aspire.noSql.elastic.authentication.basic | no | false | Enables Elasticsearch basic authentication | |
aspire.noSql.elastic.user | no | null | The Elasticsearch user to use for basic authentication | |
aspire.noSql.elastic.password | no | null | The Elasticsearch password to use for basic authentication | |
aspire.noSql.elastic.authentication.aws | no | false | Enables Elasticsearch AWS authentication | |
aspire.noSql.elastic.authentication.useCredentialsProviderChain | no | false | Option to use the AWS credentials provider chain to get the credentials | |
aspire.noSql.elastic.aws.assumeRole | no | false | 5.0.3 | If a role must be assumed to access Elasticsearch. Must be true/false |
aspire.noSql.elastic.aws.roleArn | no | null | 5.0.3 | The Role ARN to be assumed. |
aspire.noSql.elastic.aws.access.key | no | null | AWS access key for authentication | |
aspire.noSql.elastic.aws.secret.key | no | null | AWS secret key for authentication | |
aspire.noSql.elastic.aws.region | no | null | The AWS region | |
aspire.noSql.elastic.keepSearchContextAlive | no | 5m | The amount of time to keep Elasticsearch scrolls requests search context alive using "m" as a unit | |
aspire.noSql.elastic.maxRequestSize | no | 10485760B | The maximum size for a bulk request. The value can be specified in B, K, M and G units | |
aspire.noSql.elastic.maxConnections | no | 100 | The maximum number of connections to keep open. | |
aspire.noSql.elastic.maxConnectionsPerRoute | no | 10 | The maximum number of connections per server | |
aspire.noSql.elastic.readTimeout | no | 30000 | The socket timeout. | |
aspire.noSql.elastic.connectionTimeout | no | 15000 | The connection timeout. | |
aspire.noSql.elastic.maxRetries | no | 3 | The number of times to retry each request | |
aspire.noSql.elastic.retriesWaitTime | no | 5000 | The time to wait in ms between retries | |
aspire.noSql.elastic.useThrottling | no | false | Enables requests throttling to Elasticsearch | |
aspire.noSql.elastic.throttlingRate | no | 5000 | The throttling rate in ms | |
aspire.noSql.elastic.throttlingConnectionRate | no | 500 | The maximum number of requests allowed in the period specified by the throttlingRate | |
aspire.noSql.elastic.waitTime429 | no | 3000 | The time to wait in ms before retrying a 429 error | |
aspire.noSql.elastic.bulk | no | true | Enables using bulk for requests | |
aspire.noSql.elastic.bulkSize | no | 500 | The maximum number of documents to include in a bulk request | |
aspire.noSql.elastic.bulkInactivityTimeout | no | 5 | The inactivity in s before flushing a bulk request | |
aspire.noSql.elastic.bulkRegularTimeout | no | 30 | The maximum amount of time in s for a bulk request to be kept in memory before flushing | |
aspire.noSql.elastic.debugFile | no | null | The path to the debug file, request to ES are logged in this file | |
aspire.noSql.elastic.mappingFile | no | null | The file path (including file name) that includes the mapping for the indexes used by Aspire. By default, the mapping included in the provider is used | |
aspire.noSql.elastic.index.prefix | no | aspire | The prefix to use for the indexes created by the provider | |
aspire.noSql.elastic.debug | no | false | Enables debug logging information | |
aspire.noSql.elastic.usePooling | no | true | Enables HTTP connection pooling | |
aspire.noSql.elastic.timeSeriesType | no | index | 5.0.3, 5.3 (Opensearch) | Enables rollover possibility for audit, error, log time series index. Value "index" - default without rollover, Value "dataStream" - rollover based on data stream. Value "dataStreamOpensearch" - rollover based on Opensearch data stream. |
aspire.noSql.elastic.ilmPolicyFile | no | default provider policy file | 5.0.3 | The file path (including file name) that includes the ilm policy file for the indexes used by Aspire. Used only for the time series option "dataStream*". |
aspire.noSql.elastic.indexTemplateFile | no | default provider template file | 5.0.3 | The file path (including file name) that includes the template file for the indexes used by Aspire. Used only for the time series option "dataStream*". |
Each index sharding and replicas settings can be changed by using a custom mappingFile and configuring the aspire.noSql.elastic.mappingFile property.
Since 5.0.1, when using the default mapping settings, you can use the following prefix parameter:
If none of these properties are specified, the default replicas and shard number of the cluster will be used (usually 1 shard and 1 replica).
The different index names available are:
audit
base
errors
hierarchy
log
map
queue
retryLog
set
settings
snapshot
updateQueue
identityCache
Bellow are the list of properties to configure the SSL Certificates
Parameter | Required | Default | Description |
---|---|---|---|
aspire.ssl.trustAll | no | false | Configure if all certificates should be trusted |
aspire.ssl.overwriteFactory | no | false | Configure if the created key managers should overwrite the Java connection factory |
aspire.ssl.truststore.file | no | null | The path of the trust store file |
aspire.ssl.truststore.password | no | null | The trust store file password |
aspire.ssl.truststore.type | no | jks | The file format of the trust store file |
aspire.ssl.keystore.file | no | null | The path of the key store file |
aspire.ssl.keystore.password | no | null | The key store file password |
aspire.ssl.keystore.type | no | jks | The file format of the key store file |
Below are the list of properties related to the Aspire security and encryption
Parameter | Required | Default | Description |
---|---|---|---|
aspire.ldap.bind.dn.password | no | null | The password of the User DN. Not required if the authentication is anonymous |
aspire.security.api.auditing | no | true | It audits the Aspire API calls. It is available from Aspire 5.1 and on. |
When a password/secret/token in a configuration must be persisted, Aspire encrypts it and stores it encrypted. The default encryption mechanism is AES 256 given a local key. AWS KMS encryption can be used instead as of version 5.0.3
Parameter | Required | Default | Description |
---|---|---|---|
aspire.encryption.key.file | no | null | (Optional) Path (including file name) where the encryption key is located, if not provided a default in-memory key will be used, for production installations it must be always provided. This can also be passed as a JVM parameter or as an environment variable, aspire_encryption_key_file. This should be a 32 byte file, if longer, the first 32 bytes will be used as the encryption key. Grant read access to the Aspire user only (chmod 400 <file>) This file could be generated randomly
|
Available since Aspire 5.0.3, Uses AWS Key Management Service (KMS) to encrypt the sensitive data. It uses a key in KMS to encrypt and decrypt data. See more details about this encryption provider at Aspire KMS encryption
Parameter | Required | Default | Description |
---|---|---|---|
aspire.encryption.kms.roleARN | no | null | (Optional) If the KMS service must be accessed through the assumption of an IAM role, specify the role ARN. |
aspire.encryption.kms.keyARN | yes | N/A | The KMS key ARN. See Aspire KMS encryption for more information about creating a KMS key for Aspire. |
aspire.encryption.kms.region | yes | N/A | The AWS region on which the KMS service will be used |
aspire.encryption.kms.accessKey | no | null | (Optional) Specify the access key if static credentials must be used. If this is not specified, the Default Credential Provider Chain will be used. |
aspire.encryption.kms.secretKey | no | null | (Optional) Specify the secret key if static credentials must be used. If this is not specified, the Default Credential Provider Chain will be used. |
These properties will be used by all worker nodes in the cluster.
Parameter | Required | Default | Description |
---|---|---|---|
aspire.node.worker.maxMemQueueSize | yes | 4000 | The maximum number of items to keep in the in-memory queue |
aspire.node.worker.queueSizeThreshold | yes | 0.5 | The capacity threshold of the in memory queue before requesting more items to the managers |
aspire.node.worker.cleanUpWaitTime | yes | 300000 | The wait time in ms for the thread that checks the connectors clean up threshold |
aspire.node.worker.cleanUpThreshold | yes | 3600000 | The time in ms for a connector to be idle before being removed from memory |
aspire.node.worker.maxEnqueueRetries | yes | 5 | The number of retries to enqueue an item into the framework pipeline |
aspire.node.worker.workflow.appCleanUpWaitTime | yes | 60000 | The wait time in ms for the thread that checks the workflow application's clean-up threshold |
aspire.node.worker.workflow.appCleanUpThreshold | yes | 3600000 | The time in ms for a workflow application to be idle before being removed from memory |
aspire.node.worker.tags | no | The tags of the worker node. These tags will determine which items this node can process. It should be a comma separated list of tags. | |
aspire.node.worker.entryProcessorBaseSleep | yes | 10 | The base sleep time in ms for the thread in charge of queuing received items into the connector framework pipelines |
aspire.node.worker.entryProcessorMaxSleep | yes | 2000 | The maximum sleep in ms for the thread in charge of queuing received items into the connector framework pipelines |
aspire.node.worker.entryProcessorMaxIterations | yes | 5 | The number of iterations without queuing items before in increasing the sleep time |
aspire.node.worker.entryProcessorMultiplier | yes | 1.25 | The multiplier used to increase the sleep time after the specified iterations without queuing items |
aspire.node.worker.batchLoaderBaseSleep | yes | 10 | The base sleep time in ms for the thread in charge of requesting batches to the manager nodes |
aspire.node.worker.batchLoaderMaxSleep | yes | 2000 | The maximum sleep in ms for the thread in charge of requesting batches to the manager nodes |
aspire.node.worker.batchLoaderMaxIterations | yes | 5 | The number of iterations without receiving batches from the managers nodes before in increasing the sleep time |
aspire.node.worker.batchLoaderMultiplier | yes | 1.25 | The multiplier used to increase the sleep time after the specified iterations without receiving batches from the managers nodes |
aspire.node.worker.connectionTimeout | yes | 20000 | The connection timeout for requests to other aspire nodes |
aspire.node.worker.socketTimeout | yes | 20000 | The socket timeout for requests to other aspire nodes |
aspire.node.worker.maxRetries | yes | 3 | The number of retries for requests to other aspire nodes |
aspire.node.worker.proxyHost | no | null | The proxy host to use for requests to other aspire nodes |
aspire.node.worker.proxyPort | no | 0 | The proxy port to use for requests to other aspire nodes. Must be provided if the proxyHost is configured |
aspire.node.worker.proxyUser | no | null | The proxy user to use for requests to other aspire nodes |
aspire.node.worker.proxyPassword | no | null | The proxy password to use for requests to other aspire nodes. |
aspire.node.worker.pingFrequency | yes | 15000 | The frequency for the node to ping to Elasticsearch. The pings are used to determine if a node is alive and working properly |
aspire.node.worker.nodeFailureTimeout | yes | 30000 | The ping timeout used to determine if a node is not working. The node will be marked as failed in this case and the node eventually will shut down itself |
These properties will be used by all manager nodes in the cluster.
Parameter | Required | Default | Description |
---|---|---|---|
aspire.node.manager.scanBatchCreatorBaseSleep | yes | 30 | The base sleep time in ms for the thread in charge of creating batches from the scan queue |
aspire.node.manager.scanBatchCreatorMaxSleep | yes | 2000 | The maximum sleep in ms for the thread in charge of creating batches from the scan queue |
aspire.node.manager.scanBatchCreatorMaxIterations | yes | 10 | The number of iterations without creating new scan batches before in increasing the sleep time |
aspire.node.manager.scanBatchCreatorMultiplier | yes | 1.25 | The multiplier used to increase the sleep time after the specified iterations without creating new scan batches |
aspire.node.manager.processBatchCreatorBaseSleep | yes | 30 | The base sleep time in ms for the thread in charge of creating batches from the process queue |
aspire.node.manager.processBatchCreatorMaxSleep | yes | 2000 | The maximum sleep in ms for the thread in charge of creating batches from the process queue |
aspire.node.manager.processBatchCreatorMaxIterations | yes | 10 | The number of iterations without creating new process batches before in increasing the sleep time |
aspire.node.manager.processBatchCreatorMultiplier | yes | 1.25 | The multiplier used to increase the sleep time after the specified iterations without creating new process batches |
aspire.node.manager.crawlProgressManagerBaseSleep | yes | 100 | The base sleep time in ms for the thread in charge of monitoring active crawls |
aspire.node.manager.schedulerBaseSleep | yes | 10000 | The base sleep time in ms for the thread in charge of executing seeds based on the configured schedules |
aspire.node.manager.maxBatches | yes | 1000 | The maximum number of batches the manager will keep in memory |
aspire.node.manager.maxBatchItems | yes | 100 | The maximum number of documents per batch |
aspire.node.manager.connectionTimeout | yes | 20000 | The connection timeout for requests to other aspire nodes |
aspire.node.manager.socketTimeout | yes | 20000 | The socket timeout for requests to other aspire nodes |
aspire.node.manager.maxRetries | yes | 3 | The number of retries for requests to other aspire nodes |
aspire.node.manager.proxyHost | no | null | The proxy host to use for requests to other aspire nodes |
aspire.node.manager.proxyPort | no | 0 | The proxy port to use for requests to other aspire nodes. Must be provided if the proxyHost is configured |
aspire.node.manager.proxyUser | no | null | The proxy user to use for requests to other aspire nodes |
aspire.node.manager.proxyPassword | no | null | The proxy password to use for requests to other aspire nodes. |
aspire.node.manager.pingFrequency | yes | 15000 | The frequency for the node to ping to Elasticsearch. The pings are used to determine if a node is alive and working properly |
aspire.node.manager.nodeFailureTimeout | yes | 30000 | The ping timeout used to determine if a node is not working. The node will be marked as failed in this case and the node eventually will shutdown itself |
aspire.node.manager.inProgressJobTimeout | yes | 3600000 | The maximum time in ms, a job can be in "in-progress" status before being released. Default is 1 hour |
aspire.node.manager.inProgressJobTimeoutCheckFrequency | yes | 1800000 | How frequently to verify for timed-out "in-progress" jobs |
aspire.node.manager.ackCleanBaseSleep | no | 5000 | How frequently the manager should check for Acknowledged Batches that need to be removed from memory. |
aspire.node.manager.proxyCall | no | false | Configure if you want to perform a proxy call to the main manager from a non-main manager and avoid redirect calls. |
aspire.node.manager.tags | no | The tags of the manager node. These tags will determine which seeds this node can process. It should be a comma separated list of tags. | |
aspire.node.manager.workerRoundRobin | no | false | If round-robin should be applied when serving workers with batches |
aspire.node.manager.workerRoundRobinTimeout | no | 600000 | The time in ms after which the worker is considered timed out when round-robin is used. |
Since 5.0.3, the following properties are also available for managing the behavior of the user interface.
Parameter | Required | Default | Description |
---|---|---|---|
aspire.ui.refreshRate | no | 5s | How often trigger the auto-refresh for the listing pages. It can be defined with a time unit, i.e. 15s, 1 m. |
These properties will be used to generate the dashboard links in the UI
Parameter | Required | Default | Description |
---|---|---|---|
aspire.dashboards.enabled | no | false | Enables the dashboard links on the UI |
aspire.dashboards.base | no | Base URL to Kibana | |
aspire.dashboards.main | no | Main Dashboard relative URL | |
aspire.dashboards.metrics | no | Metrics Dashboard relative URL |