On this page

Related pages

Configuration

The basic structure for configuring a pipeline manager is as follows:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
<component name="PIPELINE-MANAGER-NAME" subType="pipeline" factoryName="aspire-application">
      <queueSize>30</queueSize>
      <maxThreads>10</maxThreads>
      <queueTimeout>300000</queueTimeout>
      <shutdownTimeout>300000</shutdownTimeout>
 
      <components>
        <!-- Identify and configure all components here, any order -->
        <component name="COMPONENT1" subType="SUBTYPE" factoryName="aspire-component1"> ... </component>
        <component name="COMPONENT2" subType="SUBTYPE" factoryName="aspire-component2"> ... </component>
        <component name="COMPONENT3" subType="SUBTYPE" factoryName="aspire-component3"> ... </component>
        .
        .
        .
      </components>
 
      <pipelines>
        <!-- The list of pipelines go here -->
        <pipeline name="PIPELINE1" default="true">
          <stages>
            <!-- List all stages in the pipeline here, in order. -->
            <stage component="COMPONENT1" />
            <stage component="COMPONENT2" />
            <stage component="COMPONENT3" />
            .
            .
            .
          </stages>
        </pipeline>
 
        <!-- A pipeline manager can manage any number of pipelines -->
        <pipeline name="PIPELINE1"> ... </pipeline>
        <pipeline name="PIPELINE2"> ... </pipeline>
        .
        .
        .
      </pipelines>
    </component>

Notes

  • The pipeline contains a list of components, a list of pipelines, and a list of stages for each pipeline.
  • Each stage references a component by name from the list of components configured under the <components> tag.

Execution Parameters

The following execution parameters are configurable for the pipeline manager:

queueSizeintjobs30The size of the queue for processing jobs. If the job queue is full, then feeders, which attempt to put a new job on the queue, will be blocked until the queue has room. It is recommended that the queue size be at least as large as the number of threads, if not two or three times larger.
maxThreadsintthreads10The maximum number of threads to create to handle jobs put on the pipeline manager job queue.
queueTimeoutintms300000
(5 minutes)		The maximum time that a feeder (or, possibly, a sub-job that is enqueued) will be blocked for a full queue. If the queue is still full after the specified time, an exception error is returned to whomever is queuing up the job.
shutdownTimeoutintms300000
(5 minutes)		When shutting down the pipeline manager, the maximum time to wait for all active threads to complete.
gatherStatisticsboolean
falseGather statistics about the stages and pipelines. This can be turned on or off via the UI as well.

Components

The components list specified in the pipeline manager is a simple list of components each with their custom configuration parameters. For more details, see the discussion under the Configuring Components section of the system configuration file documentation.

Note that all components configured under the <components> tag must be pipeline stages if they are to be referenced in a pipeline <stage> element.

  • No labels