Format
All properties can be passed either as a environment variable or as a JVM parameter. The "." and "_" characters can be replaced with each other.
SET aspire_noSql_elastic_server=http://localhost:9200 //Setting a property as a environment variable java -Daspire.noSql.elastic.server=http://localhost:9200 //Passing a property like JVM parameter
In the case of environment variables that property name should always use "_" instead of "."
Elasticsearch NoSQL Provider
Bellow are the list of properties to configure the Elasticsearch NoSQL Provider
| Parameter | Default | Description |
|---|---|---|
| aspire.noSql.elastic.server | http://localhost:9200 | (Required) The elasticsearch server url to use. It can be multiple urls separated by "," |
| aspire.noSql.elastic.authentication.basic | false | (Optional) Enables elasticsearch basic authentication |
| aspire.noSql.elastic.user | null | (Optional) The elasticsearch user to use for basic authentication |
| aspire.noSql.elastic.password | null | (Optional) The elasticsearch password to use for basic authentication |
| aspire.noSql.elastic.authentication.aws | false | (Optional) Enables elasticsearch AWS authentication |
| aspire.noSql.elastic.authentication.useCredentialsProviderChain | false | (Optional) Option to use the AWS credentials provider chain to get the credentials |
| aspire.noSql.elastic.aws.access.key | null | (Optional) AWS access key for authentication |
| aspire.noSql.elastic.aws.secret.key | null | (Optional) AWS secret key for authentication |
| aspire.noSql.elastic.aws.region | null | (Optional) The AWS region |
| aspire.noSql.elastic.keepSearchContextAlive | 5m | (Optional) The amount of time to keep elasticsearch scrolls requests search context alive using "m" as a unit |
| aspire.noSql.elastic.maxRequestSize | 10485760B | (Optional) The maximum size for a bulk request. The value can be specified in B, K, M and G units |
| aspire.noSql.elastic.maxConnections | 100 | (Optional) The maximum amount of connections to mantain open. |
| aspire.noSql.elastic.maxConnectionsPerRoute | 10 | (Optional) The maximum amount of connections per server |
| aspire.noSql.elastic.maxRetries | 3 | (Optional) The amount of times to retry each request |
| aspire.noSql.elastic.retriesWaitTime | 5000 | (Optional) The time to wait in ms between retries |
| aspire.noSql.elastic.useThrottling | false | (Optional) Enables requests throttling to elasticsearch |
| aspire.noSql.elastic.throttlingRate | 5000 | (Optional) The throttling rate in ms |
| aspire.noSql.elastic.throttlingConnectionRate | 500 | (Optional) The maximum amount of requests allowed in the period specified by the throttlingRate |
| aspire.noSql.elastic.waitTime429 | 3000 | (Optional) The time to wait in ms before retrying a 429 error |
| aspire.noSql.elastic.bulk | true | (Optional) Enables using bulk for requests |
| aspire.noSql.elastic.bulkSize | 500 | (Optional) The maximum number of documents to include in a bulk request |
| aspire.noSql.elastic.bulkInactivityTimeout | 5 | (Optional) The inactivity in s before flushing a bulk request |
| aspire.noSql.elastic.bulkRegularTimeout | 30 | (Optional) The maximum amount of time in s for a bulk request to be kept in memory before flushing |
| aspire.noSql.elastic.debugFile | null | (Optional) The path to the debug file, request to ES are logged in this file |
| aspire.noSql.elastic.mappingFile | null | (Optional) The file path (including file name) that includes the mapping for the indexes used by Aspire. By default mapping included in the provider is used |
| aspire.noSql.elastic.index.prefix | aspire | (Optional) The prefix to use for the indexes created by the provider |
| aspire.noSql.elastic.debug | false | (Optional) Enables debug logging information |
| aspire.noSql.elastic.usePooling | true | (Optional) Enables HTTP connection pooling |
SSL Certificates
Bellow are the list of properties to configure the SSL Certificates
| Parameter | Default | Description |
|---|---|---|
| com.accenture.aspire.ssl.trustAll | false | (Optional) Configure if all certificates should be trusted |
| com.accenture.aspire.ssl.overwriteFactory | false | (Optional) Configure if the created key managers should overwrite the Java connection factory |
| com.accenture.aspire.ssl.truststore.file | null | (Optional) The path of the trust store file |
| com.accenture.aspire.ssl.truststore.password | null | (Optional) The trust store file password |
| com.accenture.aspire.ssl.truststore.type | jks | (Optional) The file format of the trust store file |
| com.accenture.aspire.ssl.keystore.file | null | (Optional) The path of the key store file |
| com.accenture.aspire.ssl.keystore.password | null | (Optional) The key store file password |
| com.accenture.aspire.ssl.keystore.type | jks | (Optional) The file format of the key store file |
Security and encryption
Bellow are the list of properties related to the Aspire security and encryption
| Parameter | Default | Description |
|---|---|---|
| aspire.ldap.bind.dn.password | false | (Required) The password of the User DN. Not required if the authentication is anonymous |
| aspire.encryption.key.file | false | (Optional) The file path (including file name) where master key is located, if not provided a in memory hardcoded key will be used, for production installations it must be always provided. |
Worker Node
These properties will be used by all worker nodes in the cluster.
| Parameter | Default | Description |
|---|---|---|
| com.accenture.aspire.node.worker.maxMemQueueSize | 1000 | (Required) The maximum number of items to keep in the in memory queue |
| com.accenture.aspire.node.worker.queueSizeThreshold | 0.75 | (Required) The capacity threshold of the in memory queue before requesting more items to the managers |
| com.accenture.aspire.node.worker.cleanUpWaitTime | 300000 | (Required) The wait time in ms for the thread that checks the connectors clean up threshold |
| com.accenture.aspire.node.worker.cleanUpThreshold | 3600000 | (Required) The time in ms for a connector to be idle before being removed from memory |
| com.accenture.aspire.node.worker.maxEnqueueRetries | 5 | (Required) The number of retries to enqueue a item into the framework pipeline |
| com.accenture.aspire.node.worker.workflow.appCleanUpWaitTime | 60000 | (Required) The wait time in ms for the thread that checks the workflow applications clean up threshold |
| com.accenture.aspire.node.worker.workflow.appCleanUpThreshold | 3600000 | (Required) The time in ms for a workflow application to be idle before being removed from memory |
| com.accenture.aspire.node.worker.tags | (Optional) The tags of the worker node. These tags will determine which items this node can process | |
| com.accenture.aspire.node.worker.entryProcessorBaseSleep | 200 | (Required) The base sleep time in ms for the thread in charge of enqueueing received items into the connector framework pipelines |
| com.accenture.aspire.node.worker.entryProcessorMaxSleep | 10000 | (Required) The maximum sleep in ms for the thread in charge of enqueueing received items into the connector framework pipelines |
| com.accenture.aspire.node.worker.entryProcessorMaxIterations | 5 | (Required) The number of iterations without enqueueing items before in increasing the sleep time |
| com.accenture.aspire.node.worker.entryProcessorMultiplier | 2 | (Required) The multiplier used to increase the sleep time after the specified iterations without enqueueing items |
| com.accenture.aspire.node.worker.batchLoaderBaseSleep | 200 | (Required) The base sleep time in ms for the thread in charge of requesting batches to the manager nodes |
| com.accenture.aspire.node.worker.batchLoaderMaxSleep | 10000 | (Required) The maximum sleep in ms for the thread in charge of requesting batches to the manager nodes |
| com.accenture.aspire.node.worker.batchLoaderMaxIterations | 5 | (Required) The number of iterations without receiving batches from the managers nodes before in increasing the sleep time |
| com.accenture.aspire.node.worker.batchLoaderMultiplier | 2 | (Required) The multiplier used to increase the sleep time after the specified iterations without receiving batches from the managers nodes |
| com.accenture.aspire.node.worker.connectionTimeout | 20000 | (Required) The connection timeout for requests to other aspire nodes |
| com.accenture.aspire.node.worker.socketTimeout | 20000 | (Required) The socket timeout for requests to other aspire nodes |
| com.accenture.aspire.node.worker.maxRetries | 3 | (Required) The number of retries for requests to other aspire nodes |
| com.accenture.aspire.node.worker.proxyHost | null | (Optional) The proxy host to use for requests to other aspire nodes |
| com.accenture.aspire.node.worker.proxyPort | 0 | (Optional) The proxy port to use for requests to other aspire nodes. Must be provided if the proxyHost is configured |
| com.accenture.aspire.node.worker.proxyUser | null | (Optional) The proxy user to use for requests to other aspire nodes |
| com.accenture.aspire.node.worker.proxyPassword | null | (Optional) The proxy password to use for requests to other aspire nodes. |
| com.accenture.aspire.node.worker.pingFrequency | 15000 | (Required) The ping timeout used to determine if a node is not working. The node will be marked as failed in this case and the node eventually will shutdown itself |
| com.accenture.aspire.node.worker.nodeFailureTimeout | 30000 | (Required) The frequency for the node to ping to Elastisearch. The pings are used to determine if a node is alive and working properly |
Manager Node
These properties will be used by all manager nodes in the cluster.
| Parameter | Default | Description |
|---|---|---|
| com.accenture.aspire.node.manager.scanBatchCreatorBaseSleep | 200 | (Required) The base sleep time in ms for the thread in charge of creating batches from the scan queue |
| com.accenture.aspire.node.manager.scanBatchCreatorMaxSleep | 10000 | (Required) The maximum sleep in ms for the thread in charge of creating batches from the scan queue |
| com.accenture.aspire.node.manager.scanBatchCreatorMaxIterations | 10 | (Required) The number of iterations without creating new scan batches before in increasing the sleep time |
| com.accenture.aspire.node.manager.scanBatchCreatorMultiplier | 2 | (Required) The multiplier used to increase the sleep time after the specified iterations without creating new scan batches |
| com.accenture.aspire.node.manager.processBatchCreatorBaseSleep | 200 | (Required) The base sleep time in ms for the thread in charge of creating batches from the process queue |
| com.accenture.aspire.node.manager.processBatchCreatorMaxSleep | 10000 | (Required) The maximum sleep in ms for the thread in charge of creating batches from the process queue |
| com.accenture.aspire.node.manager.processBatchCreatorMaxIterations | 10 | (Required) The number of iterations without creating new process batches before in increasing the sleep time |
| com.accenture.aspire.node.manager.processBatchCreatorMultiplier | 2 | (Required) The multiplier used to increase the sleep time after the specified iterations without creating new process batches |
| com.accenture.aspire.node.manager.crawlProgressManagerBaseSleep | 200 | (Required) The base sleep time in ms for the thread in charge of monitoring active crawls |
| com.accenture.aspire.node.manager.schedulerBaseSleep | 10000 | (Required) The base sleep time in ms for the thread in charge of executing seeds based on the configured schedules |
| com.accenture.aspire.node.manager.maxBatches | 1000 | (Required) The maximum number of batches the manager will keep in memory |
| com.accenture.aspire.node.manager.maxBatchItems | 100 | (Required) The maximum number of documents per batch |
| com.accenture.aspire.node.manager.connectionTimeout | 20000 | (Required) The connection timeout for requests to other aspire nodes |
| com.accenture.aspire.node.manager.socketTimeout | 20000 | (Required) The socket timeout for requests to other aspire nodes |
| com.accenture.aspire.node.manager.maxRetries | 3 | (Required) The number of retries for requests to other aspire nodes |
| com.accenture.aspire.node.manager.proxyHost | null | (Optional) The proxy host to use for requests to other aspire nodes |
| com.accenture.aspire.node.manager.proxyPort | 0 | (Optional) The proxy port to use for requests to other aspire nodes. Must be provided if the proxyHost is configured |
| com.accenture.aspire.node.manager.proxyUser | null | (Optional) The proxy user to use for requests to other aspire nodes |
| com.accenture.aspire.node.manager.proxyPassword | null | (Optional) The proxy password to use for requests to other aspire nodes. |
| com.accenture.aspire.node.manager.pingFrequency | 15000 | (Required) The ping timeout used to determine if a node is not working. The node will be marked as failed in this case and the node eventually will shutdown itself |
| com.accenture.aspire.node.manager.nodeFailureTimeout | 30000 | (Required) The frequency for the node to ping to Elastisearch. The pings are used to determine if a node is alive and working properly |
| com.accenture.aspire.node.manager.inProgressJobTimeout | 3600000 | (Optional) The maximum time in ms a job can be in "in-progress" status before being released. Default is 1 hour |
| com.accenture.aspire.node.manager.inProgressJobTimeoutCheckFrequency | 1800000 | (Optional) How frequently to verify for timed-out "in-progress" jobs |
Overview
Content Tools
Copyright © 2024 Accenture
