All properties can be passed either as a environment variable or as a JVM parameter. The "." and "_" characters can be replaced with each other.
Code Block | ||||
---|---|---|---|---|
| ||||
SET aspire_.noSql_.elastic_.server=http://localhost:9200 //Setting a property as a environment variable java -Daspire.noSql.elastic.server=http://localhost:9200 //Passing a property like JVM parameter |
Warning |
---|
In the case of environment variables that , property name should always use "_" instead of "." |
In Linux based systems, the memory usage can be changed with the following environment variables
Parameter | Required | Default | Description |
---|---|---|---|
aspire_.max_.heap_.memory | no | 2g | Maximum Heap the JVM will request and use from the host system |
aspire_.max_.metaspace_.size | no | 256m | Maximum metaspace the JVM will request and use from the host system. |
In windows-based systems, if using the aspire.bat file, the memory usage should be changed in the aspire.bat itself. Otherwise, if it is running as a Windows Service:
HKEY_LOCAL_MACHINE / SOFTWARE / WOW6432Node / Apache Software / Procrun
2.0
/ AspireService / Parameters / Java
Bellow Below are the list of properties to configure the Elasticsearch NoSQL Provider
Parameter | Required | Default | New & Available from | Description | |||
---|---|---|---|---|---|---|---|
aspire.noSql.elastic.server | yes | http://localhost:9200 | The | elasticsearchElasticsearch server | urlURL to use. It can be multiple | urlsURLs separated by "," | |
aspire.noSql.elastic.authentication.basic | no | false | Enables | elasticsearchElasticsearch basic authentication | |||
aspire.noSql.elastic.user | no | null | The | elasticsearchElasticsearch user to use for basic authentication | |||
aspire.noSql.elastic.password | no | null | The | elasticsearchElasticsearch password to use for basic authentication | |||
aspire.noSql.elastic.authentication.aws | no | false | Enables | elasticsearchElasticsearch AWS authentication | |||
aspire.noSql.elastic.authentication.useCredentialsProviderChain | no | false | Option to use the AWS credentials provider chain to get the credentials | ||||
aspire.noSql.elastic.aws. | access.keyassumeRole | no | null | AWS access key for authenticationfalse | 5.0.3 | If a role must be assumed to access Elasticsearch. Must be true/false | |
aspire.noSql.elastic.aws. | secret.keyroleArn | no | null | AWS secret key for authentication | 5.0.3 | The Role ARN to be assumed. | |
aspire.noSql.elastic.aws.access.key | no | null | AWS access key for authentication | ||||
aspire.noSql.elastic.aws.secret.key | no | null | AWS secret key for authentication | ||||
aspire.noSql.elastic. | aspire.noSql.elastic.aws.region | no | null | The AWS region | |||
aspire.noSql.elastic.keepSearchContextAlive | no | 5m | The amount of time to keep | elasticsearchElasticsearch scrolls requests search context alive using "m" as a unit | |||
aspire.noSql.elastic.maxRequestSize | no | 10485760B | The maximum size for a bulk request. The value can be specified in B, K, M and G units | ||||
aspire.noSql.elastic.maxConnections | no | 100 | The maximum | amountnumber of connections to | mantainkeep open. | ||
aspire.noSql.elastic.maxConnectionsPerRoute | no | 10 | The maximum | amountnumber of connections per server | |||
aspire.noSql.elastic.readTimeout | no | 30000 | The socket timeout. | ||||
aspire.noSql.elastic.connectionTimeout | no | 15000 | The connection timeout. | ||||
aspire.noSql.elastic.maxRetries | no | 3 | The | amountnumber of times to retry each request | |||
aspire.noSql.elastic.retriesWaitTime | no | 5000 | The time to wait in ms between retries | ||||
aspire.noSql.elastic.useThrottling | no | false | Enables requests throttling to | elasticsearchElasticsearch | |||
aspire.noSql.elastic.throttlingRate | no | 5000 | The throttling rate in ms | ||||
aspire.noSql.elastic.throttlingConnectionRate | no | 500 | The maximum | amountnumber of requests allowed in the period specified by the throttlingRate | |||
aspire.noSql.elastic.waitTime429 | no | 3000 | The time to wait in ms before retrying a 429 error | ||||
aspire.noSql.elastic.bulk | no | true | Enables using bulk for requests | ||||
aspire.noSql.elastic.bulkSize | no | 500 | The maximum number of documents to include in a bulk request | ||||
aspire.noSql.elastic.bulkInactivityTimeout | no | 5 | The inactivity in s before flushing a bulk request | ||||
aspire.noSql.elastic.bulkRegularTimeout | no | 30 | The maximum amount of time in s for a bulk request to be kept in memory before flushing | ||||
aspire.noSql.elastic.debugFile | no | null | The path to the debug file, request to ES are logged in this file | ||||
aspire.noSql.elastic.mappingFile | no | null | The file | pathpath (including file name) that includes the mapping for the indexes used by Aspire. By default, the mapping included in the provider is used | |||
aspire.noSql.elastic.index.prefix | no | aspire | The prefix to use for the indexes created by the provider | ||||
aspire.noSql.elastic.debug | no | false | Enables debug logging information | ||||
aspire.noSql.elastic.usePooling | no | true | Enables HTTP connection pooling |
Bellow are the list of properties to configure the SSL Certificates
aspire.nosql.elastic.timeSeriesType | no | index | 5.0.3, 5.3 (Opensearch) | Enables rollover possibility for audit, error, log time series index. Value "index" - default without rollover, Value "dataStream" - rollover based on data stream. Value "dataStreamOpensearch" - rollover based on Opensearch data stream. |
aspire.nosql.elastic.ilmPolicyFile | no | default provider policy file | 5.0.3 | The file path (including file name) that includes the ilm policy file for the indexes used by Aspire. Used only for the time series option "dataStream*". |
aspire.nosql.elastic.indexTemplateFile | no | default provider template file | 5.0.3 | The file path (including file name) that includes the template file for the indexes used by Aspire. Used only for the time series option "dataStream*". |
Each index sharding and replicas settings can be changed by using a custom mappingFile and configuring the aspire.noSql.elastic.mappingFile property.
Since 5.0.1, when using the default mapping settings, you can use the following prefix parameter:
If none of these properties are specified, the default replicas and shard number of the cluster will be used (usually 1 shard and 1 replica).
The different index names available are:
audit
base
errors
hierarchy
log
map
queue
retryLog
set
settings
snapshot
updateQueue
identityCache
Bellow are the list of properties to configure the SSL Certificates
Parameter | Required |
---|
Bellow are the list of properties related to the Aspire security and encryption
Parameter | Required | Default | Description | ||
---|---|---|---|---|---|
aspire.ldap.bind.dn.passwordssl.trustAll | no | null | The password of the User DN. Not required if the authentication is anonymous | false | Configure if all certificates should be trusted |
aspire.ssl.overwriteFactory | no | false | Configure if the created key managers should overwrite the Java connection factory | ||
aspire.ssl.truststoreaspire.encryption.key.file | no | null (Optional) Path (including file name) where the encryption key is located, if not provided a default in-memory key will be used, for production installations it must be always provided. This can also be passed as a JVM parameter or as an environment variable aspire_encryption_key_file This should be a 32 byte file, if longer, the first 32 bytes will be used as the encryption key. Grant read access to the Aspire user only (chmod 400 <file>) This file could be generated randomly $ head -c 32 /dev/urandom > encryption.key | The path of the trust store file | ||
aspire.ssl.truststore.password | no | null | The trust store file password | ||
aspire.ssl.truststore.type | no | jks | The file format of the trust store file | ||
aspire.ssl.keystore.file | no | null | The path of the key store file | ||
aspire.ssl.keystore.password | no | null | The key store file password | ||
aspire.ssl.keystore.type | no | jks | The file format of the key store file |
Below are the list of properties related to the Aspire security and encryption
Parameter | Required | Default | Description |
---|---|---|---|
aspire.ldap.bind.dn.password | no | null | The password of the User DN. Not required if the authentication is anonymous |
aspire.security.api.auditing | no | true | It audits the Aspire API calls. It is available from Aspire 5.1 and on. |
When a password/secret/token in a configuration must be persisted, Aspire encrypts it and stores it encrypted. The default encryption mechanism is AES 256 given a local key. AWS KMS encryption can be used instead as of version 5.0.3
Parameter | Required | Default | Description |
---|---|---|---|
aspire.encryption.key.file | no | null | (Optional) Path (including file name) where the encryption key is located, if not provided a default in-memory key will be used, for production installations it must be always provided. This can also be passed as a JVM parameter or as an environment variable, aspire_encryption_key_file. This should be a 32 byte file, if longer, the first 32 bytes will be used as the encryption key. Grant read access to the Aspire user only (chmod 400 <file>) This file could be generated randomly
|
Available since Aspire 5.0.3, Uses AWS Key Management Service (KMS) to encrypt the sensitive data. It uses a key in KMS to encrypt and decrypt data. See more details about this encryption provider at Aspire KMS encryption
Parameter | Required | Default | Description |
---|---|---|---|
aspire.encryption.kms.roleARN | no | null | (Optional) If the KMS service must be accessed through the assumption of an IAM role, specify the role ARN. |
aspire.encryption.kms.keyARN | yes | N/A | The KMS key ARN. See Aspire KMS encryption for more information about creating a KMS key for Aspire. |
aspire.encryption.kms.region | yes | N/A | The AWS region on which the KMS service will be used |
aspire.encryption.kms.accessKey | no | null | (Optional) Specify the access key if static credentials must be used. If this is not specified, the Default Credential Provider Chain will be used. |
aspire.encryption.kms.secretKey | no | null | (Optional) Specify the secret key if static credentials must be used. If this is not specified, the Default Credential Provider Chain will be used. |
These properties will be used by all worker nodes in the cluster.
Parameter | Required | Default | Description |
---|---|---|---|
com.accenture.aspire.node.worker.maxMemQueueSize | yes | 4000 | The maximum number of items to keep in the in-memory queue |
com.accenture.aspire.node.worker.queueSizeThreshold | yes | 0.5 | The capacity threshold of the in memory queue before requesting more items to the managers |
com.accenture.aspire.node.worker.cleanUpWaitTime | yes | 300000 | The wait time in ms for the thread that checks the connectors clean up threshold |
com.accenture.aspire.node.worker.cleanUpThreshold | yes | 3600000 | The time in ms for a connector to be idle before being removed from memory |
com.accenture.aspire.node.worker.maxEnqueueRetries | yes | 5 | The number of retries to enqueue a an item into the framework pipeline |
com.accenture.aspire.node.worker.workflow.appCleanUpWaitTime | yes | 60000 | The wait time in ms for the thread that checks the workflow applications application's clean-up threshold |
com.accenture.aspire.node.worker.workflow.appCleanUpThreshold | yes | 3600000 | The time in ms for a workflow application to be idle before being removed from memory |
com.accenture.aspire.node.worker.tags | no | The tags of the worker node. These tags will determine which items this node can process. Should It should be a comma separated list of tags. | |
com.accenture.aspire.node.worker.entryProcessorBaseSleep | yes | 10 | The base sleep time in ms for the thread in charge of enqueueing of queuing received items into the connector framework pipelines |
com.accenture.aspire.node.worker.entryProcessorMaxSleep | yes | 2000 | The maximum sleep in ms for the thread in charge of enqueueing of queuing received items into the connector framework pipelines |
com.accenture.aspire.node.worker.entryProcessorMaxIterations | yes | 5 | The number of iterations without enqueueing queuing items before in increasing the sleep time |
com.accenture.aspire.node.worker.entryProcessorMultiplier | yes | 1.25 | The multiplier used to increase the sleep time after the specified iterations without enqueueing queuing items |
com.accenture.aspire.node.worker.batchLoaderBaseSleep | yes | 10 | The base sleep time in ms for the thread in charge of requesting batches to the manager nodes |
com.accenture.aspire.node.worker.batchLoaderMaxSleep | yes | 2000 | The maximum sleep in ms for the thread in charge of requesting batches to the manager nodes |
com.accenture.aspire.node.worker.batchLoaderMaxIterations | yes | 5 | The number of iterations without receiving batches from the managers nodes before in increasing the sleep time |
com.accenture.aspire.node.worker.batchLoaderMultiplier | yes | 1.25 | The multiplier used to increase the sleep time after the specified iterations without receiving batches from the managers nodesnodes |
aspire.node.worker.connectionTimeout | yes | 20000 | The connection timeout for requests to other aspire nodes |
aspire.node.worker.socketTimeout | yes | 20000 | The socket timeout for requests to other aspire nodes |
com.accenture.aspire.node.worker.connectionTimeoutmaxRetries | yes | 200003 | The connection timeout number of retries for requests to other aspire nodes |
com.accenture.aspire.node.worker.socketTimeoutproxyHost | yesno | 20000null | The socket timeout proxy host to use for requests to other aspire nodes |
com.accenture.aspire.node.worker.maxRetriesproxyPort | yesno | 30 | The number of retries proxy port to use for requests to other aspire nodes. Must be provided if the proxyHost is configured |
com.accenture.aspire.node.worker.proxyHostproxyUser | no | null | The proxy host user to use for requests to other aspire nodes |
com.accenture.aspire.node.worker.proxyPortproxyPassword | no | 0null | The proxy port password to use for requests to other aspire nodes. Must be provided if the proxyHost is configured |
com.accenture.aspire.node.worker.proxyUserpingFrequency | noyes | null15000 | The proxy user to use for requests to other aspire nodes |
com.accenture.aspire.node.worker.proxyPassword | no | null | The proxy password to use for requests to other aspire nodes. |
com.accenture.aspire.node.worker.pingFrequency | yes | 15000 | The frequency for the node to ping to Elastisearch. The pings are used to determine if a node is alive and working properly |
com.accenture.aspire.node.worker.nodeFailureTimeout | yes | 30000 | The ping timeout used to determine if a node is not working. The node will be marked as failed in this case and the node eventually will shutdown itself |
These properties will be used by all manager nodes in the cluster.
frequency for the node to ping to Elasticsearch. The pings are used to determine if a node is alive and working properly | |||
aspire.node.worker.nodeFailureTimeout | yes | 30000 | The ping timeout used to determine if a node is not working. The node will be marked as failed in this case and the node eventually will shut down itself |
These properties will be used by all manager nodes in the cluster.
Parameter | Required | Default | Description | ||||
---|---|---|---|---|---|---|---|
aspire.node.manager.scanBatchCreatorBaseSleep | yes | 30 | The base sleep time in ms for the thread in charge of creating batches from the scan queue | ||||
aspire.node.manager.scanBatchCreatorMaxSleep | yes | 2000 | The maximum sleep in ms for the thread in charge of creating batches from the scan queue | ||||
aspire.node.manager.scanBatchCreatorMaxIterations | yes | 10 | The number of iterations without creating new scan batches before in increasing the sleep time | ||||
aspire.node.manager.scanBatchCreatorMultiplier | yes | 1.25 | The multiplier used to increase the sleep time after the specified iterations without creating new scan batches | ||||
aspire.node.manager.processBatchCreatorBaseSleep | |||||||
Parameter | Required | Default | Description | ||||
com.accenture.aspire.node.manager.scanBatchCreatorBaseSleep | yes | 30 | The base sleep time in ms for the thread in charge of creating batches from the scan process queue | ||||
com.accenture.aspire.node.manager.scanBatchCreatorMaxSleepprocessBatchCreatorMaxSleep | yes | 2000 | The maximum sleep in ms for the thread in charge of creating batches from the scan process queue | ||||
com.accenture.aspire.node.manager.scanBatchCreatorMaxIterationsprocessBatchCreatorMaxIterations | yes | 10 | The number of iterations without creating new scan process batches before in increasing the sleep time | ||||
com.accenture.aspire.node.manager.scanBatchCreatorMultiplierprocessBatchCreatorMultiplier | yes | 1.25 | The multiplier used to increase the sleep time after the specified iterations without creating new scan batchesthe sleep time after the specified iterations without creating new process batches | ||||
aspire.node.manager.crawlProgressManagerBaseSleep | yes | 100 | The base sleep time in ms for the thread in charge of monitoring active crawls | ||||
com.accenture.aspire.node.manager.processBatchCreatorBaseSleepschedulerBaseSleep | yes | 3010000 | The base sleep time in ms for the thread in charge of creating batches from the process queueof executing seeds based on the configured schedules | ||||
aspire.node.manager.maxBatches | yes | 1000 | The maximum number of batches the manager will keep in memory | ||||
com.accenture.aspire.node.manager.processBatchCreatorMaxSleepmaxBatchItems | yes | 2000100 | The maximum sleep in ms for the thread in charge of creating batches from the process queuenumber of documents per batch | ||||
com.accenture.aspire.node.manager.processBatchCreatorMaxIterationsconnectionTimeout | yes | 10 | The number of iterations without creating new process batches before in increasing the sleep time | 20000 | The connection timeout for requests to other aspire nodes | ||
com.accenture.aspire.node.manager.processBatchCreatorMultipliersocketTimeout | yes | 1.2520000 | The multiplier used to increase the sleep time after the specified iterations without creating new process batches socket timeout for requests to other aspire nodes | ||||
com.accenture.aspire.node.manager.crawlProgressManagerBaseSleepmaxRetries | yes | 1003 | The base sleep time in ms for the thread in charge of monitoring active crawlsnumber of retries for requests to other aspire nodes | ||||
com.accenture.aspire.node.manager.schedulerBaseSleepproxyHost | yesno | 10000null | The base sleep time in ms for the thread in charge of executing seeds based on the configured schedulesproxy host to use for requests to other aspire nodes | ||||
com.accenture.aspire.node.manager.maxBatchesproxyPort | yesno | 1000 | The maximum number of batches the manager will keep in memory | 0 | The proxy port to use for requests to other aspire nodes. Must be provided if the proxyHost is configured | ||
com.accenture.aspire.node.manager.maxBatchItemsproxyUser | yesno | 100 | The maximum number of documents per batch | null | The proxy user to use for requests to other aspire nodes | ||
com.accenture.aspire.node.manager.connectionTimeoutproxyPassword | yesno | 20000null | The connection timeout proxy password to use for requests to other aspire nodes. | ||||
com.accenture.aspire.node.manager.socketTimeoutpingFrequency | yes | 2000015000 | The socket timeout for requests to other aspire nodesfrequency for the node to ping to Elasticsearch. The pings are used to determine if a node is alive and working properly | ||||
com.accenture.aspire.node.manager.maxRetriesnodeFailureTimeout | yes | 330000 | The number of retries for requests to other aspire nodes | ||||
com.accenture.aspire.node.manager.proxyHost | no | null | The proxy host to use for requests to other aspire nodes | ||||
ping timeout used to determine if a node is not working. The node will be marked as failed in this case and the node eventually will shutdown itself | |||||||
com.accenture.aspire.node.manager.proxyPortinProgressJobTimeout | noyes | 3600000 | The maximum time in ms, a job can be in "in-progress" status before being released. Default is 1 hour | ||||
0 | The proxy port to use for requests to other aspire nodes. Must be provided if the proxyHost is configured | com.accenture.aspire.node.manager.proxyUserinProgressJobTimeoutCheckFrequency | noyes | 1800000 | How frequently to verify for timed-out "in-progress" jobs | ||
null | The proxy user to use for requests to other aspire nodes | com.accenture.aspire.node.manager.proxyPasswordackCleanBaseSleep | no | null | 5000 | How frequently the manager should check for Acknowledged Batches that need to be removed from memory. | |
The proxy password to use for requests to other aspire nodes. | com.accenture.aspire.node.manager.pingFrequencyproxyCall | yesno | false | Configure if you want to perform a proxy call to the main manager from a non-main manager and avoid redirect calls. | |||
15000 | The frequency for the node to ping to Elastisearch. The pings are used to determine if a node is alive and working properly | com.accenture.aspire.node.manager.nodeFailureTimeouttags | yesno | The tags of the manager node. These tags will determine which seeds this node can process. It should be a comma separated list of tags. | |||
30000 | The ping timeout used to determine if a node is not working. The node will be marked as failed in this case and the node eventually will shutdown itself | com.accenture.aspire.node.manager.inProgressJobTimeoutworkerRoundRobin | yesno | false | If round-robin should be applied when serving workers with batches | ||
3600000 | The maximum time in ms a job can be in "in-progress" status before being released. Default is 1 hour | com.accenture.aspire.node.manager.inProgressJobTimeoutCheckFrequencyworkerRoundRobinTimeout | yesno | 1800000 | How frequently to verify for timed-out "in-progress" jobs | 600000 | The time in ms after which the worker is considered timed out when round-robin is used. |
Since 5.0.3, the following properties are also available for managing the behavior of the user interface.
Parameter | Required | Default | Description | ||||
---|---|---|---|---|---|---|---|
aspire.ui.refreshRate | no | 5s | How often trigger the auto-refresh for the listing pages. It can be defined with a time unit, i.e. 15s, 1 m | com.accenture.aspire.node.manager.ackCleanBaseSleep | no | 5000 | How frequently the manager should check for Acknowledged Batches that need to be removed from memory. |
These properties will be used to generate the dashboard links in the UI
Parameter | Required | Default | Description |
---|---|---|---|
aspire.dashboards.enabled | no | false | Enables the dashboard links on the UI |
aspire.dashboards.base | no | Base URL to Kibana | |
aspire.dashboards.main | no | Main Dasboard Dashboard relative URL | |
aspire.dashboards.metrics | no | Metrics Dashboard relative URL |