Format
All properties can be passed either as a environment variable or as a JVM parameter. The "." and "_" characters can be replaced with each other.
SET aspire_noSql_elastic_server=http://localhost:9200 //Setting a property as a environment variable java -Daspire.noSql.elastic.server=http://localhost:9200 //Passing a property like JVM parameter
In the case of environment variables that property name should always use "_" instead of "."
Elasticsearch NoSQL Provider
Bellow are the list of properties to configure the Elasticsearch NoSQL Provider
| Parameter | Required | Default | Description |
|---|---|---|---|
| aspire.noSql.elastic.server | yes | http://localhost:9200 | The elasticsearch server url to use. It can be multiple urls separated by "," |
| aspire.noSql.elastic.authentication.basic | no | false | Enables elasticsearch basic authentication |
| aspire.noSql.elastic.user | no | null | The elasticsearch user to use for basic authentication |
| aspire.noSql.elastic.password | no | null | The elasticsearch password to use for basic authentication |
| aspire.noSql.elastic.authentication.aws | no | false | Enables elasticsearch AWS authentication |
| aspire.noSql.elastic.authentication.useCredentialsProviderChain | no | false | Option to use the AWS credentials provider chain to get the credentials |
| aspire.noSql.elastic.aws.access.key | no | null | AWS access key for authentication |
| aspire.noSql.elastic.aws.secret.key | no | null | AWS secret key for authentication |
| aspire.noSql.elastic.aws.region | no | null | The AWS region |
| aspire.noSql.elastic.keepSearchContextAlive | no | 5m | The amount of time to keep elasticsearch scrolls requests search context alive using "m" as a unit |
| aspire.noSql.elastic.maxRequestSize | no | 10485760B | The maximum size for a bulk request. The value can be specified in B, K, M and G units |
| aspire.noSql.elastic.maxConnections | no | 100 | The maximum amount of connections to mantain open. |
| aspire.noSql.elastic.maxConnectionsPerRoute | no | 10 | The maximum amount of connections per server |
| aspire.noSql.elastic.maxRetries | no | 3 | The amount of times to retry each request |
| aspire.noSql.elastic.retriesWaitTime | no | 5000 | The time to wait in ms between retries |
| aspire.noSql.elastic.useThrottling | no | false | Enables requests throttling to elasticsearch |
| aspire.noSql.elastic.throttlingRate | no | 5000 | The throttling rate in ms |
| aspire.noSql.elastic.throttlingConnectionRate | no | 500 | The maximum amount of requests allowed in the period specified by the throttlingRate |
| aspire.noSql.elastic.waitTime429 | no | 3000 | The time to wait in ms before retrying a 429 error |
| aspire.noSql.elastic.bulk | no | true | Enables using bulk for requests |
| aspire.noSql.elastic.bulkSize | no | 500 | The maximum number of documents to include in a bulk request |
| aspire.noSql.elastic.bulkInactivityTimeout | no | 5 | The inactivity in s before flushing a bulk request |
| aspire.noSql.elastic.bulkRegularTimeout | no | 30 | The maximum amount of time in s for a bulk request to be kept in memory before flushing |
| aspire.noSql.elastic.debugFile | no | null | The path to the debug file, request to ES are logged in this file |
| aspire.noSql.elastic.mappingFile | no | null | The file path (including file name) that includes the mapping for the indexes used by Aspire. By default mapping included in the provider is used |
| aspire.noSql.elastic.index.prefix | no | aspire | The prefix to use for the indexes created by the provider |
| aspire.noSql.elastic.debug | no | false | Enables debug logging information |
| aspire.noSql.elastic.usePooling | no | true | Enables HTTP connection pooling |
SSL Certificates
Bellow are the list of properties to configure the SSL Certificates
| Parameter | Required | Default | Description |
|---|---|---|---|
| com.accenture.aspire.ssl.trustAll | no | false | Configure if all certificates should be trusted |
| com.accenture.aspire.ssl.overwriteFactory | no | false | Configure if the created key managers should overwrite the Java connection factory |
| com.accenture.aspire.ssl.truststore.file | no | null | The path of the trust store file |
| com.accenture.aspire.ssl.truststore.password | no | null | The trust store file password |
| com.accenture.aspire.ssl.truststore.type | no | jks | The file format of the trust store file |
| com.accenture.aspire.ssl.keystore.file | no | null | The path of the key store file |
| com.accenture.aspire.ssl.keystore.password | no | null | The key store file password |
| com.accenture.aspire.ssl.keystore.type | no | jks | The file format of the key store file |
Security and encryption
Bellow are the list of properties related to the Aspire security and encryption
| Parameter | Required | Default | Description |
|---|---|---|---|
| aspire.ldap.bind.dn.password | no | null | The password of the User DN. Not required if the authentication is anonymous |
| aspire.encryption.key.file | no | null | (Optional) Path (including file name) where master key is located, if not provided a default in-memory key will be used, for production installations it must be always provided. This can also be passed as a JVM parameter or as an environment variable aspire.encryption.key.file. This should be a 32 byte file, if longer, the first 32 bytes will be used as the master key. Grant read access to the Aspire user only (chmod 400 <file>) |
Worker Node
These properties will be used by all worker nodes in the cluster.
| Parameter | Required | Default | Description |
|---|---|---|---|
| com.accenture.aspire.node.worker.maxMemQueueSize | yes | 4000 | The maximum number of items to keep in the in memory queue |
| com.accenture.aspire.node.worker.queueSizeThreshold | yes | 0.5 | The capacity threshold of the in memory queue before requesting more items to the managers |
| com.accenture.aspire.node.worker.cleanUpWaitTime | yes | 300000 | The wait time in ms for the thread that checks the connectors clean up threshold |
| com.accenture.aspire.node.worker.cleanUpThreshold | yes | 3600000 | The time in ms for a connector to be idle before being removed from memory |
| com.accenture.aspire.node.worker.maxEnqueueRetries | yes | 5 | The number of retries to enqueue a item into the framework pipeline |
| com.accenture.aspire.node.worker.workflow.appCleanUpWaitTime | yes | 60000 | The wait time in ms for the thread that checks the workflow applications clean up threshold |
| com.accenture.aspire.node.worker.workflow.appCleanUpThreshold | yes | 3600000 | The time in ms for a workflow application to be idle before being removed from memory |
| com.accenture.aspire.node.worker.tags | no | The tags of the worker node. These tags will determine which items this node can process. Should be a comma separated list of tags. | |
| com.accenture.aspire.node.worker.entryProcessorBaseSleep | yes | 10 | The base sleep time in ms for the thread in charge of enqueueing received items into the connector framework pipelines |
| com.accenture.aspire.node.worker.entryProcessorMaxSleep | yes | 2000 | The maximum sleep in ms for the thread in charge of enqueueing received items into the connector framework pipelines |
| com.accenture.aspire.node.worker.entryProcessorMaxIterations | yes | 5 | The number of iterations without enqueueing items before in increasing the sleep time |
| com.accenture.aspire.node.worker.entryProcessorMultiplier | yes | 1.25 | The multiplier used to increase the sleep time after the specified iterations without enqueueing items |
| com.accenture.aspire.node.worker.batchLoaderBaseSleep | yes | 10 | The base sleep time in ms for the thread in charge of requesting batches to the manager nodes |
| com.accenture.aspire.node.worker.batchLoaderMaxSleep | yes | 2000 | The maximum sleep in ms for the thread in charge of requesting batches to the manager nodes |
| com.accenture.aspire.node.worker.batchLoaderMaxIterations | yes | 5 | The number of iterations without receiving batches from the managers nodes before in increasing the sleep time |
| com.accenture.aspire.node.worker.batchLoaderMultiplier | yes | 1.25 | The multiplier used to increase the sleep time after the specified iterations without receiving batches from the managers nodes |
| com.accenture.aspire.node.worker.connectionTimeout | yes | 20000 | The connection timeout for requests to other aspire nodes |
| com.accenture.aspire.node.worker.socketTimeout | yes | 20000 | The socket timeout for requests to other aspire nodes |
| com.accenture.aspire.node.worker.maxRetries | yes | 3 | The number of retries for requests to other aspire nodes |
| com.accenture.aspire.node.worker.proxyHost | no | null | The proxy host to use for requests to other aspire nodes |
| com.accenture.aspire.node.worker.proxyPort | no | 0 | The proxy port to use for requests to other aspire nodes. Must be provided if the proxyHost is configured |
| com.accenture.aspire.node.worker.proxyUser | no | null | The proxy user to use for requests to other aspire nodes |
| com.accenture.aspire.node.worker.proxyPassword | no | null | The proxy password to use for requests to other aspire nodes. |
| com.accenture.aspire.node.worker.pingFrequency | yes | 15000 | The ping timeout used to determine if a node is not working. The node will be marked as failed in this case and the node eventually will shutdown itself |
| com.accenture.aspire.node.worker.nodeFailureTimeout | yes | 30000 | The frequency for the node to ping to Elastisearch. The pings are used to determine if a node is alive and working properly |
Manager Node
These properties will be used by all manager nodes in the cluster.
| Parameter | Required | Default | Description |
|---|---|---|---|
| com.accenture.aspire.node.manager.scanBatchCreatorBaseSleep | yes | 30 | The base sleep time in ms for the thread in charge of creating batches from the scan queue |
| com.accenture.aspire.node.manager.scanBatchCreatorMaxSleep | yes | 2000 | The maximum sleep in ms for the thread in charge of creating batches from the scan queue |
| com.accenture.aspire.node.manager.scanBatchCreatorMaxIterations | yes | 10 | The number of iterations without creating new scan batches before in increasing the sleep time |
| com.accenture.aspire.node.manager.scanBatchCreatorMultiplier | yes | 1.25 | The multiplier used to increase the sleep time after the specified iterations without creating new scan batches |
| com.accenture.aspire.node.manager.processBatchCreatorBaseSleep | yes | 30 | The base sleep time in ms for the thread in charge of creating batches from the process queue |
| com.accenture.aspire.node.manager.processBatchCreatorMaxSleep | yes | 2000 | The maximum sleep in ms for the thread in charge of creating batches from the process queue |
| com.accenture.aspire.node.manager.processBatchCreatorMaxIterations | yes | 10 | The number of iterations without creating new process batches before in increasing the sleep time |
| com.accenture.aspire.node.manager.processBatchCreatorMultiplier | yes | 1.25 | The multiplier used to increase the sleep time after the specified iterations without creating new process batches |
| com.accenture.aspire.node.manager.crawlProgressManagerBaseSleep | yes | 100 | The base sleep time in ms for the thread in charge of monitoring active crawls |
| com.accenture.aspire.node.manager.schedulerBaseSleep | yes | 10000 | The base sleep time in ms for the thread in charge of executing seeds based on the configured schedules |
| com.accenture.aspire.node.manager.maxBatches | yes | 1000 | The maximum number of batches the manager will keep in memory |
| com.accenture.aspire.node.manager.maxBatchItems | yes | 100 | The maximum number of documents per batch |
| com.accenture.aspire.node.manager.connectionTimeout | yes | 20000 | The connection timeout for requests to other aspire nodes |
| com.accenture.aspire.node.manager.socketTimeout | yes | 20000 | The socket timeout for requests to other aspire nodes |
| com.accenture.aspire.node.manager.maxRetries | yes | 3 | The number of retries for requests to other aspire nodes |
| com.accenture.aspire.node.manager.proxyHost | no | null | The proxy host to use for requests to other aspire nodes |
| com.accenture.aspire.node.manager.proxyPort | no | 0 | The proxy port to use for requests to other aspire nodes. Must be provided if the proxyHost is configured |
| com.accenture.aspire.node.manager.proxyUser | no | null | The proxy user to use for requests to other aspire nodes |
| com.accenture.aspire.node.manager.proxyPassword | no | null | The proxy password to use for requests to other aspire nodes. |
| com.accenture.aspire.node.manager.pingFrequency | yes | 15000 | The ping timeout used to determine if a node is not working. The node will be marked as failed in this case and the node eventually will shutdown itself |
| com.accenture.aspire.node.manager.nodeFailureTimeout | yes | 30000 | The frequency for the node to ping to Elastisearch. The pings are used to determine if a node is alive and working properly |
| com.accenture.aspire.node.manager.inProgressJobTimeout | yes | 3600000 | The maximum time in ms a job can be in "in-progress" status before being released. Default is 1 hour |
| com.accenture.aspire.node.manager.inProgressJobTimeoutCheckFrequency | yes | 1800000 | How frequently to verify for timed-out "in-progress" jobs |
Dashboards
These properties will be used to generate the dashboard links in the UI
| Parameter | Required | Default | Description |
|---|---|---|---|
| aspire.dashboards.enabled | no | false | Enables the dashboard links on the UI |
| aspire.dashboards.base | no | Base URL to Kibana | |
| aspire.dashboards.main | no | Main Dasboard relative URL | |
| aspire.dashboards.metrics | no | Metrics Dashboard relative URL |
Overview
Content Tools
Copyright © 2024 Accenture
