Page History
Elesticsearch
properties
Configuration object elasticNoUrl Yes false No Selects how to enter the Elasticsearch URL. True: server and port are specified. False: full URL is specified. true / false elasticUrl
No
Enter the URL for the Elasticsearch bulk index endpoint, it must have this format <protocol>://<host>:<port>/_bulk . Required if elasticNoUrl is false. "http://myServer:9200"
elasticHost No - No Enter the Elasticsearch host. Required if elasticNoUrl is true.
"myServer" elasticPort No 9200 No Enter the Elasticsearch port. Required if elasticNoUrl is true. "9201"
Enter the index to which the jobs are going to be publish
test
index
authType
No
Specify the authentication information. Allowed values are: "none", "basic" and "aws". "none"
user No - No Provide the user for basic authentication. Required if authType is "basic". "john.doe" password No - No Provide the password for basic authentication. Required if authType is "basic". "myP@ssword11" awsRegion No - No Specify the AWS region to use. Required if authType is "aws". "us-east-2" useCredentialsProviderChain No false No Enable to specify a credentials provider chain. true / false awsAccessKey No - No Provide the access key for authentication with AWS. Required if authType is "aws" and useCredentialsProviderChain is false. "AKIAQYO33L5HIS3N55NN" awsSecretKey No - No Provide the secret key for authentication with AWS. Required if authType is "aws" and useCredentialsProviderChain is false. "PPA3Qt+FqUA4Kk07DRS7w4T6Bya1IWebbcqFmqmI" transformType Yes -
transform
JSON
processClear
clearType
if processClear = Yes
-"deleteAllDocuments" uploadMappings No false No Upload Mappings. Relevant for clearType "deleteIndex"
uploadMappingsFrom
No
false
Source for the mappings. Required if uploadMappings is true. If uploadsMappingFrom is true the mappings will be used from the parameter below otherwise from the file true / false mappingsText No - No Mappings specified directly. Required if uploadMappings is true and uploadMappingsFrom is true. "{\"mappings\":{\"properties\":{\"doc._id\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\"}}}}}}" uploadMappingsFile No - No Mappings file path. Required if uploadMappings is true and uploadMappingsFrom is false.
"C:\mappings\myMappings\myMappingsFile.txt"
period
period
Elasticsearch Publisher Properties
| Field | Required | Default | Multiple | Notes | Example |
|---|
| type | Yes | - | No |
The value must be "application". | "application" |
| _type | Yes |
| - | No |
The value must be "publisher". | "publisher" | ||||
| appName | Yes | - | No | The name of the application | "Azure_Publisher" |
| appType |
| Yes | - | No |
| The value must be "azure-search-publisher". | " |
| azure-search- |
| publisher" |
| config |
| Yes | - | No |
| The value must be "com.accenture.aspire:aspire-azure-search-publisher-source". | "com.accenture.aspire:aspire-azure-search-publisher-source" | ||||
| description | Yes | - | No | The description | "Azure Search Publisher" |
| properties | Yes | - | No | Configuration object | |
|---|---|---|---|---|---|
| azureServer | Yes | - | No | Enter the name of the service endpoint to use. | "testdomain.search.windows.net" |
| azureIndex | Yes | - | No | Enter the Azure Search index where the jobs will be stored. | "index1" |
| azureApiVersion | Yes | "2021-04-30-Preview" | No | Enter the Azure Search API version of the REST API. | "2021-04-30-Preview" |
| azureApiKey | Yes | - | No | Enter the Azure Search API Key used to connect to the REST API. | "PPA3Qt+FqUA5ddSSDRS7w4T6Bya1IWebbcqFmqmI" |
| transformType | Yes | "JSON" |
| No | Type of the transformation file used. | "JSON" | |||
| useTransformFile | Yes | true | No | Select if a file path or a resource file upload is used in the resources' manager section. | true / false |
| transformFile | Yes | - | No | Path to the transform file. The default value is set to "${component.home}/config/groovy/ |
| aspireToAzureSearchBulk.groovy" for the default |
| transformation file provided with Aspire. (Required if useTransformFile is true) | "C:\transformfiles\groovy\mytransform.groovy" |
| transformResource | Yes | - | No | Transformation Resource ID. (Required if useTransformFile is false) | "a1f0a708e7965a972cedede5779c656de3574fc88eee983d" |
| cleanOnFullCrawl | Yes | false | No | Set to true to clear the index on full crawls. | true / false |
| useIndexFile | Yes |
| true | No |
Type of the clear. 2 options here :
- "deleteAllDocuments" - deletes documents with the index
- "deleteIndex" - recreates the whole index (with or without requested mapping)
Select if a file path or a resource file upload is used in the resources' manager section. (Required if cleanOnFullCrawl is true) | true / false |
| azureIndexFile |
| Yes |
| - | No |
| Path to the transform file. The default value is set to ".{component.home}/config/json/azureSearchIndex.json" for the default transformation file provided with Aspire. (Required if useIndexFile is true) | C:\transformfiles\json\myindex.json" | ||||
| mappingResource | Yes | - | No | Index Mappings Resource ID. (Required if useIndexFile is false) | "f038945fa320d5f7ee28f835754e9dd699eb36a25ce3439a9" |
| indexDeleteWait | Yes | 10000 | No | The maximum time (in milliseconds) to wait between the deletion/creation of the index and the publishing process. | 100000 |
| idleConnectionTimeout | No | - | No | Maximum time (in milliseconds) to keep an idle connection open. | 3600000 |
| maxConnections | No | - | No | Maximum number of connections to be opened. | 100 |
| maxConnectionsPerRoute | No | - | No | Maximum number of connections opened for the same target. | 10 |
| connectionTimeout | No | - | No | Maximum time (in milliseconds) to wait for the connection. | 15000 |
| socketTimeout | No | - | No | Maximum time (in milliseconds) to wait for a socket response. | 15000 |
| useThrottling | No | - | No | Set to true to specify Throttling Settings. | true / false |
| throttlingRate | No | 5000 | No | Time |
| (in milliseconds) to throttle the connection. | 6000 | ||||
| throttlingConnectionRate | No | 500 | No | Maximum number of connections used during the Throttling Period. | 600 |
| maxRetries | No | 3 | No | Maximum number of retries for a failed document. | 5 |
| retryWaitTime | No | 5 | No | Time |
| (in milliseconds) to wait before a retry. | 6000 | ||||
| batchSize | No | - | No | Maximum size of the batches that will be created. | 50 |
| simultaneousBatches | No | - | No | Number of batches that will be processed simultaneously. | 5 |
| batchTimeout | No | - | No | Period (in ms) after which a batch of documents will be closed and executed. | 5000 |
| debug | No | - | No | Set to true to enable debug mode to show debug messages from the publisher. | true / false |
Create example
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"type": "application",
"_type": "publisher",
"description": "ElasticsearchPublisherTestAzure Search Publisher Test",
"config": "com.accenture.aspire:aspire-azure-elasticsearchsearch-publisher-source",
"appType": "elasticsearchazuresearch-publisher",
"appName": "ElasticsearchAzuresearch Publisher",
"properties": {
"azureServer": "test.search.windows.net",
"elasticNoUrlazureIndex": true"index1",
"elasticHostazureApiVersion": "myEShost2021-04-30-Preview",
"elasticPort "azureApiKey": 9200"encrypted:C4D3B8FE27C26B7D682C8E481D3A53ACEBF7DB23B476F09CB2C0FF866B9E2DCHAFCD08B79B02E51D30E33CB7233CEAD",
"elasticIndextransformType": "test-indexJSON",
"useTransformFile": false,
"authTypetransformResource": "nonea1f0a708e7965a972cede7e4b56ac8e5779c656de3574fc88eee983d",
"transformTypecleanOnFullCrawl": true,
"JSONuseIndexFile": false,
"transformFile "mappingResource": "${component.home}/config/groovy/transform.groovyf038945fa320d5f7ee28f1d7c1719a9be7a8835b36a25ce3439a9",
"processClearindexDeleteWait": false100000,
"idleConnectionTimeout": 3600000,
"maxConnections": 100,
"maxConnectionsPerRoute": 10,
"sharedConnection": false,
"connectionTimeout": 15000,
"socketTimeout": 15000,
"useThrottling": false,
"maxRetries": 3,
"retryWaitTime": 5000,
"batchSize": 50,
"simultaneousBatches": 5,
"batchTimeout": 5000,
"debug": false
}
} |
Overview
Content Tools