Pipeline Manager
The pipeline manager is in charge of the loading of the pipeline configuration and the initialization of its implementation, it will also check for updates to the pipeline configuration and replace the current implementation with a new one.
Pipeline manager is a singleton instance accesible from anywhere in the code, this will give access to all pipeline loaded on it.
Accessing Pipeline Manager in Code
You can get the pipeline manager in you code using this code
from app.pipeline import pipeline_manager
Pipeline
A pipeline represents a process divided in one or more steps (Stages), each step is specialised on a single action, being this data manipulation, database query, third party request,... The pipeline will initially get the data from the body request, and send it to the first stage as intermediate data, it is also the pipeline's job to generate and update the final data, with the responses of each stage.
Besides managing the data, the pipeline also manages the flow, and the process executing, being these Process and Post_process (there is a third flow called get_ui_config, but is still on development design). For each data flow the pipeline will execute the stages one at the time (with the exception of the stages inside a parallel stage, but it is actually the parallel stage executing stages in parallel), and in the order they were defined.
The normal execution of a pipeline involves executing the Process flow first and after every stage has executed, Post_process is executed. At the end of the Post_process, the final response is converted into a JSON and sent in the HTTP response.
The pipeline configuration looks like this
| Property | Type | Description |
|---|---|---|
| enable | boolean | Enable pipeline configuration for execution (Default: true) |
| stages | array | List of stages to execute (Minimum stages: 1) |
Using a Pipeline in Code
If you need to use a pipeline in code, you can use the pipeline manger to get the desired pipeline and execute the pipeline
pipeline: PipelineImpl = pipeline_manager.get_pipeline('search')
result = await pipeline.execute_pipeline(req, props=payload.dict(exclude_none=True))
Pipeline Trigger by a request
A Pipeline always needs the request
Stages
A Stage is a module specialize on a single action, using the available data provided in either intermediate or final. The range of actions is only limited to your imagination, a stage can either transform data, call a data base and retrieve new data, call a third party service and wait for its response, generate files, ... But we recommend to keep it as focus as posible, the intention of this is to be able to restructure a pipeline if need it, and to reuse as many stages as possible.
The configuration of a stage may vary between one and another but all share a small set of parameters:
| Property | Type | Description |
|---|---|---|
| type | string | Stage class name (Only required when using the JSON format) |
| enable | boolean | Enable stage for execution (Default: true) |
| name | string | Name for this specific stage. Used on the intermediate and final parameters. |
| save_to_intermediate | boolean | If true, the result of the stage will be stored in the intermediate instead of the final section. This will make the response of the stage unavailable for the final result. (Default: false) |
| expand_result | boolean | Indicates if the result of this stage should be expanded into the final data dictionary instead of appended as the standard. (Default: false) |
| ui_only | object | Section specific for UI configuration. This configuration will be retrieved when necessary and should affect the process of the stage. |
| halt_on_exception | boolean | Indicates if, in case of an exception, the pipeline should be interrupted. (Default: false) |
To learn more about stages go to How to Create a Stage
Working with Pipelines
Before Working with a pipeline please take into account the following suggestions:
- Stage by Stage
- Apply your bussiness rules one stage at the time, making the whole process modular and easier to customize, and reuse
- When Possible, Make It as Generic as Possible
- Stages modular nature, allows for them to be integrated into the base code, if they are generic enough. The development team, can add them, and increase the built in options for future projects, taking advantage from the project synergy.
- Know Your Pipeline
- Each stage accepts different parameters in the intermediate. Every pipeline is tailored to the clients needs. The combination of parameters and the data flow are unique for each pipeline, so keep the data input documented
- As Many Pipelines As You Want
- A pipeline is just stages in a certain order, since all the logic is in the stages, multiple pipelines doesn’t mean code repetition. Make pipelines for each data flow, with different configurations.
Pipeline Definition
All Pipeline:
- Are .py files, meaning they are programmable
- Declare a PIPELINE variable of type Pipeline, or Dictionary if you are going to use the JSON format (which we do not recommend if possible)
Dictionary
Intended for automatic generation, web UI, and Automation NOT for manual pipeline building
ONLY FOR AUTOMATION
Don't use this format
Pipeline
Intended manual usage in IDE, all settings should have a class for either Dictionary or Pipeline format, so why not use the configuration class directly
Pipeline With Variables
No difference with previuos one, regarding functionality, but much more manageable when defining, and modifying
Pros
- Easier to read
- Generic JSON format
Cons
- Requires conversion to Python classes
- Verbose
- All in one single JSON object
- Susceptible to conversion and format issues
Pros
- No conversion need
- Easier for IDE (Specially Pycharm with Pydantic Plug-in)
- Type errors, in pipeline definition capture via IDE compiler or in runtime easily
- Can go directly to class if need to check types or other parameters
Cons
- You may need to import a lot of configuration classes
- A bit harder to read
Pros
- … Same as previuos one..
- Each stage is an independent variable, allowing to use its parameters, in following stages
- Better comments
- Pipline structure simplified,
and easier to modify
Cons
- You may need to import a lot of configuration classes
- A bit harder to read
- Verbose
Overview
Content Tools