Page History
Create Avro Summarizer Executor
Field | Required | Default | Multiple | Notes | Example |
|---|---|---|---|---|---|
| type | Yes | - | No | The value must be "application". | "application" |
| _type | Yes | - | No | The value must be "application". | "application" |
| appName | Yes | - | No | The name of the application | "ParquetAvro-Executor" |
| appType | Yes | - | No | The value must be "parquetavro-summarize-executor". | "parquetavro-summarize-executor" |
| config | Yes | - | No | The value must be "com.accenture.aspire:app-parquetsummarizeavrosummarize-executor". | "com.accenture.aspire:app-parquetsummarizeavrosummarize-executor" |
| description | Yes | - | No | The description | "ParquetAvro-Executor" |
| properties | Yes | - | No | Configuration object | |
| addSchema | Yes | true | No | If enabled, the table schema will be added to the processed columns. | true |
| useTempFiledebug | YesNo | truefalse | No | Enable to download the content stream to a temporary file before processing it.Debug messages will be enabled | true |
| threadPool | Yes | 5 | No | The number of threads to use for parallel processing. | 5 |
| logFrequency | Yes | 1000 | No | The frequency for reporting the processed rows. | 1000 |
| useSampling | Yes | false | No | Enable to process only a random sample of the table rows. This option could increases the memory usage. | true |
| filterRows | Yes | falsetrue | No | Enable to filter the rows to process. | true |
| useFilterFile | Yes | true | No | Enable to use a groovy file to filter the rows. | true |
| groovyPath | No | - | No | The path of the groovy script that contains the filter logic. It must return a boolean value. If true, the row will be filtered. | "C:\\Aspire\\config\\rowsGroovyFilter.txt" |
| groovyScript | No | - | No | Script used to filter the rows. It must return a boolean value. If true, the row will be filtered. | "row.getBoolean(\"sensitive\") == true" |
Example
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"type": "application",
"_type": "application",
"description": "ParquetAvro-Executor",
"config": "com.accenture.aspire:app-parquetsummarizeavrosummarize-executor",
"appType": "parquetavro-summarize-executor",
"appName": "ParquetAvro Summarize Executor",
"properties": {
"addSchema": true,
"useTempFiledebug": true,
"debugthreadPool": false5,
"threadPoollogFrequency": 51000,
"logFrequencyuseSampling": 1000true,
"filterRows": true,
"useFilterFile": false,
"groovyScript": "// This script must return a boolean.\n// The references of the job, doc, component, row and table objects are available.\n// Javadoc references \n// Row (row) - http://{manager}/javadocs/com/accenture/aspire/services/summarization/Row.html\n// Table (table) - http://{manager}/javadocs/com/accenture/aspire/services/summarization/Table.html\nrow.getBoolean(\"sensitive\") == true"
}
} |
Update Avro Summarizer Executor
Field | Required | Default | Multiple | Notes | Example |
|---|---|---|---|---|---|
| id | Yes | - | No | ID of the application to update | "61014782-442a-4587-ab85-ba1439a7f7b5" |
| type | Yes | - | No | The value must be "application". | "application" |
| _type | Yes | - | No | The value must be "application". | "application" |
| appName | Yes | - | No | The name of the application | "ParquetAvro-Executor" |
| appType | Yes | - | No | The value must be "parquetavro-summarize-executor". | "parquetavro-summarize-executor" |
| config | Yes | - | No | The value must be "com.accenture.aspire:app-parquetsummarizeavrosummarize-executor". | "com.accenture.aspire:app-parquetsummarizeavrosummarize-executor" |
| description | Yes | - | No | The description | "ParquetAvro -Executor" |
| properties | Yes | - | No | Configuration object | |
| addSchema | Yes | true | No | If enabled, the table schema will be added to the processed columns. | true |
| useTempFiledebug | YesNo | truefalse | No | Enable to download the content stream to a temporary file before processing it.Debug messages will be enabled | true |
| threadPool | Yes | 5 | No | The number of threads to use for parallel processing. | 5 |
| logFrequency | Yes | 1000 | No | The frequency for reporting the processed rows. | 1000 |
| useSampling | Yes | false | No | Enable to process only a random sample of the table rows. This option could increases the memory usage. | true |
| filterRows | Yes | falsetrue | No | Enable to filter the rows to process. | true |
| useFilterFile | Yes | true | No | Enable to use a groovy file to filter the rows. | true |
| groovyPath | No | - | No | The path of the groovy script that contains the filter logic. It must return a boolean value. If true, the row will be filtered. | "C:\\Aspire\\config\\rowsGroovyFilter.txt" |
| groovyScript | No | - | No | Script used to filter the rows. It must return a boolean value. If true, the row will be filtered. | "row.getBoolean(\"sensitive\") == true" |
Example
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"id": "61014782-442a-4587-ab85-ba1439a7f7b5",
"type": "application",
"_type": "application",
"description": "ParquetAvro-Executor",
"config": "com.accenture.aspire:app-parquetsummarizeavrosummarize-executor",
"appType": "parquetavro-summarize-executor",
"appName": "ParquetAvro Summarize Executor",
"properties": {
"addSchema": true,
"useTempFiledebug": true,
"debugthreadPool": false5,
"threadPoollogFrequency": 51000,
"logFrequencyuseSampling": 1000true,
"filterRows": true,
"useFilterFile": false,
"groovyScript": "// This script must return a boolean.\n// The references of the job, doc, component, row and table objects are available.\n// Javadoc references \n// Row (row) - http://{manager}/javadocs/com/accenture/aspire/services/summarization/Row.html\n// Table (table) - http://{manager}/javadocs/com/accenture/aspire/services/summarization/Table.html\nrow.getBoolean(\"sensitive\") == true"
}
} |
Overview
Content Tools