The Branch Handler can be configured to handle batches of jobs. Pipelines typically operate a job at a time, and Aspire is mostly about splitting items up into pieces and processing the smaller pieces (aka, "sub jobs"). However, there are situations where batching is appropriate, mostly when you need communicate with external systems, for example to send a batch of documents to a search engine.
Introduction
When batching is enabled (using configuration parameters on the Branch Handler), the Branch Handler will create a Batch object to represent the batch as a whole. All jobs which belong to the batch will be attached to the batch. You can use the methods Job.isOnBatch() to see if a job is part of a larger batch.
Each component that supports job batching will implement a custom ComponentBatch that contains the required logic for opening/closing events on the batch and processing single jobs that are part of a existing batch. The parent Batch object will contain, essentially, a map of all of the ComponentBatch objects for all components which have implemented batching.
Once all of the jobs in a batch are "complete" (and when the batch is released by the Branch Handler), the batch will be officially "complete". This then causes the batch operations to be performed. Basically this means that all of the ComponentBatch objects will be closed (using the close() method). Each ComponentBatch object will then perform the batch-ed job operation which they need to perform.
Program for Batching In Your Components
Important: Most components operate on documents a job at a time. Batches will flow through these components unmodified. No special programming is required to handle batches unless your component needs to do some special batch operation (such as save up all IDs and do a batch update of a relational database, for example).
Follow these steps to allow your pipeline stage to take advantage of job batching.
- Create a new class that implements ComponentBatch.
Overview
Content Tools