The Avro Builder provides the following functionality
Avro Builder | |
---|---|
Factory Name | com.searchtechnologies.aspire:app-publish-to-avro |
subType | |
Inputs | AspireObject from a connector's job with metadata and content extracted from a specific file |
Outputs | Avro representation of AspireObject sent to an Avro file. |
Element | Type | Default | Description |
---|---|---|---|
avroSchema | string | Avro schema path | |
outputNonBatchFile | string | Output file path | |
nonBatchFileMaxSize | int | 600 | Max file size (MB) |
outputBatchDirName | string | Output directory path for internal batch files | |
maxInternalBatchSize | int | 1000 | Batch size |
timeRolloverThreshold | long | 86400 | Interval after which rollover should happen in sec |
rolloverFileLocation | string | The directory where Avro files should be moved | |
debug | boolean | false | If true it will log debug information from the component |
<application config="com.searchtechnologies.aspire:app-publish-to-avro" name="/Aspire_Publish_To_Avro_Application"> <properties> <property name="useSizeLimited">false</property> <property name="debug">false</property> <property name="maxInternalBatchSize">${maxInternalBatchSize}</property> <property name="outputBatchDirName">${outputBatchDirName}</property> <property name="useInternalBatch">true</property> <property name="avroSchema">${avroSchema}</property> <timeRolloverThreshold>${timeRolloverThreshold}</timeRolloverThreshold> <rolloverFileLocation>${rolloverFileLocation}</rolloverFileLocation> <property name="OutputType">internalBatches</property> </properties> </application>
<application config="com.searchtechnologies.aspire:app-publish-to-avro" name="/Aspire_Publish_To_Avro_Application"> <properties> <property name="useSizeLimited">true</property> <property name="debug">false</property> <property name="nonBatchFileMaxSize">${nonBatchFileMaxSize}</property> <property name="outputNonBatchFile">${outputNonBatchFile}</property> <property name="useInternalBatch">false</property> <property name="avroSchema">${avroSchema}</property> <timeRolloverThreshold>${timeRolloverThreshold}</timeRolloverThreshold> <rolloverFileLocation>${rolloverFileLocation}</rolloverFileLocation> <property name="OutputType">sizeLimited</property> </properties> </application>