You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

New framework allows for highly custom, complex, configurable and modular process, all accesible from one single endpoint.


Each module can:

  • Access the request
  • Add, Update or Delete variables from the intermediate data
  • Read from the final response so far
  • Return its response to the final response (Or return none at all )



Pipeline Manager

The pipeline manager is in charge of the loading of the pipeline configuration and the initialization of its implementation, it will also check for updates to the pipeline configuration and replace the current implementation with a new one.

Pipeline manager is a singleton instance accesible from anywhere in the code, this will give access to all pipeline loaded on it.

Accessing Pipeline Manager in Code

You can get the pipeline manager in you code using this code

from app.pipeline import pipeline_manager

Pipeline

A pipeline represents a process divided in one or more steps (Stages), each step is specialised on a single action, being this data manipulation, database query, third party request,... The pipeline will initially get the data from the body request, and send it to the first stage as intermediate data, it is also the pipeline's job to generate and update the final data, with the responses of each stage.

Besides managing the data, the pipeline also manages the flow, and the process executing, being these Process and Post_process (there is a third flow called get_ui_config, but is still on development design). For each data flow the pipeline will execute the stages one at the time (with the exception of the stages inside a parallel stage, but it is actually the parallel stage executing stages in parallel), and in the order they were defined.

The normal execution of a pipeline involves executing the Process flow first and after every stage has executed, Post_process is executed. At the end of the Post_process, the final response is converted into a JSON and sent in the HTTP response.

Using a Pipeline in Code

If you need to use a pipeline in code, you can use the pipeline manger to get the desired pipeline and execute the pipeline

pipeline: PipelineImpl = pipeline_manager.get_pipeline('search')
result = await pipeline.execute_pipeline(req, props=payload.dict(exclude_none=True))

A Pipeline always needs the request

Stages

A Stage is a module specialize on a single action, using  the available data provided in either intermediate or final. The range of actions is only limited to your imagination, a stage can either transform data, call a data base and retrieve new data, call a third party service and wait for its response, generate files, ... But we recommend to keep it as focus as posible, the intention of this is to be able to restructure a pipeline if need it, and to reuse as many stages as possible.


The configuration of a stage may vary between one and another but all share a small set of parameters:

PropertyTypeDescription
typestringStage class name (Only required when using the JSON format)
enablebooleanEnable stage for execution (Default: true)
namestringName for this specific stage. Used on the intermediate and final parameters.
save_to_intermediatebooleanIf true, the result of the stage will be stored in the intermediate instead of the final section. This will make the response of the stage unavailable for the final result. (Default: false)
expand_resultbooleanIndicates if the result of this stage should be expanded into the final data dictionary instead of appended as the standard. (Default: false)
ui_onlyobjectSection specific for UI configuration. This configuration will be retrieved when necessary and should affect the process of the stage.
halt_on_exceptionbooleanIndicates if, in case of an exception, the pipeline should be interrupted. (Default: false)



To learn more about stage 



  • No labels