Page History
Create Avro Summarizer Executor
Field | Required | Default | Multiple | Notes | Example |
|---|---|---|---|---|---|
| type | Yes | - | No | The value must be "application". | "application" |
| _type | Yes | - | No | The value must be "application". | "application" |
| appName | Yes | - | No | The name of the application | "Avro-Executor" |
| appType | Yes | - | No | The value must be "avro-summarize-executor". | "avro-summarize-executor" |
| config | Yes | - | No | The value must be "com.accenture.aspire:app-avrosummarize-executor". | "com.accenture.aspire:app-avrosummarize-executor" |
| description | Yes | - | No | The description | "Avro-Executor" |
| properties | Yes | - | No | Configuration object | |
| addSchema | Yes | true | No | If enabled, the table schema will be added to the processed columns. | true |
| debug | No | false | No | Debug messages will be enabled | true |
| threadPool | Yes | 5 | No | The number of threads to use for parallel processing. | 5 |
| logFrequency | Yes | 1000 | No | The frequency for reporting the processed rows. | 1000 |
| useSampling | Yes | false | No | Enable to process only a random sample of the table rows. This option could increases the memory usage. | true |
| filterRows | Yes | true | No | Enable to filter the rows to process. | true |
| useFilterFile | Yes | true | No | Enable to use a groovy file to filter the rows. | true |
| groovyPath | No | - | No | The path of the groovy script that contains the filter logic. It must return a boolean value. If true, the row will be filtered. | "C:\\Aspire\\config\\rowsGroovyFilter.txt" |
| groovyScript | No | - | No | Script used to filter the rows. It must return a boolean value. If true, the row will be filtered. | "row.getBoolean(\"sensitive\") == true" |
Example
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"type": "application",
"_type": "application",
"description": "Avro-Executor",
"config": "com.accenture.aspire:app-avrosummarize-executor",
"appType": "avro-summarize-executor",
"appName": "Avro Summarize Executor",
"properties": {
"addSchema": true,
"debug": true,
"threadPool": 5,
"logFrequency": 1000,
"useSampling": true,
"filterRows": true,
"useFilterFile": false,
"groovyScript": "// This script must return a boolean.\n// The references of the job, doc, component, row and table objects are available.\n// Javadoc references \n// Row (row) - http://{manager}/javadocs/com/accenture/aspire/services/summarization/Row.html\n// Table (table) - http://{manager}/javadocs/com/accenture/aspire/services/summarization/Table.html\nrow.getBoolean(\"sensitive\") == true"
}
} |
Update Avro Summarizer Executor
Field | Required | Default | Multiple | Notes | Example |
|---|---|---|---|---|---|
| id | Yes | - | No | ID of the application to update | "61014782-442a-4587-ab85-ba1439a7f7b5" |
| type | Yes | - | No | The value must be "application". | "application" |
| _type | Yes | - | No | The value must be "application". | "application" |
| appName | Yes | - | No | The name of the application | "Avro-Executor" |
| appType | Yes | - | No | The value must be "parquetavro-summarize-executor". | "avro-summarize-executor" |
| config | Yes | - | No | The value must be "com.accenture.aspire:app-avrosummarize-executor". | "com.accenture.aspire:app-avrosummarize-executor" |
| description | Yes | - | No | The description | "Avro -Executor" |
| properties | Yes | - | No | Configuration object | |
| addSchema | Yes | true | No | If enabled, the table schema will be added to the processed columns. | true |
| debug | No | false | No | Debug messages will be enabled | true |
| threadPool | Yes | 5 | No | The number of threads to use for parallel processing. | 5 |
| logFrequency | Yes | 1000 | No | The frequency for reporting the processed rows. | 1000 |
| useSampling | Yes | false | No | Enable to process only a random sample of the table rows. This option could increases the memory usage. | true |
| filterRows | Yes | true | No | Enable to filter the rows to process. | true |
| useFilterFile | Yes | true | No | Enable to use a groovy file to filter the rows. | true |
| groovyPath | No | - | No | The path of the groovy script that contains the filter logic. It must return a boolean value. If true, the row will be filtered. | "C:\\Aspire\\config\\rowsGroovyFilter.txt" |
| groovyScript | No | - | No | Script used to filter the rows. It must return a boolean value. If true, the row will be filtered. | "row.getBoolean(\"sensitive\") == true" |
Example
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"type": "application",
"_type": "application",
"description": "Avro-Executor",
"config": "com.accenture.aspire:app-avrosummarize-executor",
"appType": "avro-summarize-executor",
"appName": "Avro Summarize Executor",
"properties": {
"addSchema": true,
"debug": true,
"threadPool": 5,
"logFrequency": 1000,
"useSampling": true,
"filterRows": true,
"useFilterFile": false,
"groovyScript": "// This script must return a boolean.\n// The references of the job, doc, component, row and table objects are available.\n// Javadoc references \n// Row (row) - http://{manager}/javadocs/com/accenture/aspire/services/summarization/Row.html\n// Table (table) - http://{manager}/javadocs/com/accenture/aspire/services/summarization/Table.html\nrow.getBoolean(\"sensitive\") == true"
}
} |
Overview
Content Tools