The pipeline manager is in charge of the loading of the pipeline configuration and the initialization of its implementation, it will also check for updates to the pipeline configuration and replace the current implementation with a new one.
Pipeline manager is a singleton instance accesible from anywhere in the code, this will give access to all pipeline loaded on it.
You can get the pipeline manager in you code using this code
from app.pipeline import pipeline_manager
Pipeline
A pipeline represents a process divided in one or more steps (Stages), each step is specialised on a single action, being this data manipulation, database query, third party request,... The pipeline will initially get the data from the body request, and send it to the first stage as intermediate data, it is also the pipeline's job to generate and update the final data, with the responses of each stage.
Besides managing the data, the pipeline also manages the flow, and the process executing, being these Process and Post_process (there is a third flow called get_ui_config, but is still on development design). For each data flow the pipeline will execute the stages one at the time (with the exception of the stages inside a parallel stage, but it is actually the parallel stage executing stages in parallel), and in the order they were defined.
The normal execution of a pipeline involves executing the Process flow first and after every stage has executed, Post_process is executed. At the end of the Post_process, the final response is converted into a JSON and sent in the HTTP response.
The pipeline configuration looks like this
Property | Type | Description |
---|---|---|
enable | boolean | Enable pipeline configuration for execution (Default: true) |
stages | array | List of stages to execute (Minimum stages: 1) |
If you need to use a pipeline in code, you can use the pipeline manger to get the desired pipeline and execute the pipeline
pipeline: PipelineImpl = pipeline_manager.get_pipeline('search') result = await pipeline.execute_pipeline(req, props=payload.dict(exclude_none=True))
A Pipeline always needs the request
A Stage is a module specialize on a single action, using the available data provided in either intermediate or final. The range of actions is only limited to your imagination, a stage can either transform data, call a data base and retrieve new data, call a third party service and wait for its response, generate files, ... But we recommend to keep it as focus as posible, the intention of this is to be able to restructure a pipeline if need it, and to reuse as many stages as possible.
The configuration of a stage may vary between one and another but all share a small set of parameters:
Property | Type | Description |
---|---|---|
type | string | Stage class name (Only required when using the JSON format) |
enable | boolean | Enable stage for execution (Default: true) |
name | string | Name for this specific stage. Used on the intermediate and final parameters. |
save_to_intermediate | boolean | If true, the result of the stage will be stored in the intermediate instead of the final section. This will make the response of the stage unavailable for the final result. (Default: false) |
expand_result | boolean | Indicates if the result of this stage should be expanded into the final data dictionary instead of appended as the standard. (Default: false) |
ui_only | object | Section specific for UI configuration. This configuration will be retrieved when necessary and should affect the process of the stage. |
halt_on_exception | boolean | Indicates if, in case of an exception, the pipeline should be interrupted. (Default: false) |
To learn more about stage