Versions Compared

Old Version 6

changes.mady.by.user Potter (Esteban Alvarado)

Saved on

compared with

New Version Current

changes.mady.by.user Potter (Esteban Alvarado)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

With gapi.py (project manager)

Info

Available since version 0.5

Rename in version 3.0 from sapi to gapi

On the root of the GAIA API project execute on the terminal the following command:

Code Block
languagebash
python gapi.py new stage <name>

This will generate a bare minimum stage with all the steps and changes view on the manual section

Table of Contents

Warning

Documentation below does not apply for version >= 3.0.0

Manual

Use this section if gapi.py is not available in your distribution

Stage Parts

Configuration

As in all other configurations the stage configuration is a Pydantic model, so all advantages of Pydantic applied to the stage configuration (e.g, validations data transformation)

The BaseStage class is a Pydantic model with general configuration setup, were all base parameters are optional except for name

Class

Object

Code Block
languagepy
themeDJango
class DemoStage(BaseStage):
    pass

Image Removed

Code Block
languagepy
themeDJango
DemoStage(
    enable=True,
    name='demo',
    save_to_intermediate=False,
    expand_result=False,
    ui_only=None,
    halt_on_exception=False
)

Table of Contents

To add parameters, just class variables and their type. In the example below, all parameters are required, the only validation besides required, is their type.

Info

In the following example the default values where omitted, leaving only the new and required values

Class

Object

Code Block
languagepy
themeDJango
class DemoStage(BaseStage):
    message: str
    parameter_a: int
    parameter_b: float

Image Removed

Code Block
languagepy
themeDJango
DemoStage(
    name='demo',
    message='Add this to response',
    parameter_a=5,     
    parameter_a=3.6
)

Add Pydantic features

All examples above are Pydantic models, but none is taking advantage of it, so if we add this feature we would get something like this

Class

Object

Code Block
languagepy
themeDJango
from typing import Optional
from pydantic import Field

class DemoStage(BaseStage):
    message: Optional[str] = Field(title='Title Message',
                                   description='Message to put on the response body')
    parameter_a: int = Field(default=1, ge=0, lt=10,
                             description='Parameter to do something')
    parameter_b: float = Field(..., default=0.5, ge=0, lt=10,
                               description='Parameter to do something else')

Image Removed

Code Block
languagepy
themeDJango
DemoStage(
    name='demo',
    message='Add this to response',
    parameter_a=5,     
    parameter_a=3.6
)

Here we are:

  • Adding documentation when putting tittle and description
  • Making message optional, with the Optional typing 
  • Making parameter_a optional by adding a default value

  • Adding restrictions and verification to parameter_a and parameter_b with ge (greater or equal) and lt (lower than)

    Info

    If the value set does not met the gt and lt parameter requirements, an exception is thrown

  • Making parameter_b required even if we set a default, meaning only by setting parameter_b as None, will the default by use

Init

Initialization of the stage (constructor), executed only once when the pipeline is build, or when the pipeline is updated.

The init of the stage must always accept a config parameter, with the type of the Stage class.

Code Block
languagepy
themeDJango
    def __init__(self, config: DemoStage):
        super().__init__(config)

        import requests

        url = "https://api.example.com/data"  # Replace with your desired URL

        response = requests.get(url)

        if response.status_code == 200:
            print(f"Response: {response.text}")
        else:
            print(f"GET request failed with status code: {response.status_code}")

Useful for:

  • Instance variables setup
  • API calls
  • File downloading & loading
  • ML model initialization
  • One time calculations

Methods

Each stage has 3 required methods, process being the main one, post_process the alternative or secondary executed only after process and get_ui_config currently under development and still not a definitive method

This methods share the same parameters:

Column
width25%

self

  • Access to instance variables
  • Access to the configuration via self.config
Column
width25%

intermediate

  • Has the body of the request
  • Has field added by other stages
  • Information stored here will not end in the final response
  • Useful for passing data through stages
Column
width25%

final

  • Has final response, up to that moment
  • Immutable, can’t add or delete from it (but can modify data already stored)
  • Any return from a stage will end in the final using the stage name as key
Column
width25%

request

  • Raw request triggering the pipeline
  • User data, header, query parameter available
Tip

Request allows access to the user information currently sending the request

Code Block
languagepy
themeDJango
user: User = request.state.user
Building a Stage

1) Create the package

  • Start by creating a python file for the pipeline configuration in the folder app/pipeline/stages
  • Create the __init__ file

  • Create the stage file

    Tip

    A descriptive name works the best

Image Modified




on the stage file


1) Add the imports

  • Special attention to BaseStage & BaseStageImpl
Code Block
languagepy
themeDJango
from typing import Any, Dict

from starlette.requests import Request

from app.pipeline import BaseStage, BaseStageImpl
from utils.data import ImmutableDict

2) Declare the Stage class inheriting from BaseStage (Configuration)

  • Essentially a pydantic model, the this is where your configuration will be
  • This is the Stage configuration set on the pipeline
Code Block
languagepy
themeDJango
class DemoStage(BaseStage):
    pass

3) Declare the Stage Implementation ()

  • Inheriting BaseStageImpl, with the Stage class (DemoStage) as a parameter

  • Here is where the logic should be

Code Block
languagepy
themeDJango
class DemoStageImpl(BaseStageImpl[DemoStage]):

4) Add the abstract methods and __init__ if required

  • These are the only required conditions for this class, everything else depends on your implementation
  1. __init__

    1. Stage Initialization, one time code execution

  2. process

    1. Main stage body

    2. Accepts 3 parameters (intermediate, final, request)

    3. If returns a value, value will end in the final response

    4. If no return no value added to the response

  3. post_process

    1. Alternative or secondary method, if need

    2. Execute after all the stages have executed process

    3. Accepts 3 parameters (intermediate, final, request)

    4. If returns a value, value will end in the final response

    5. If no return no value added to the response

  4. get_ui_config

    1. For user interface only

    2. Accepts 1 parameter (request)

    3. Return UI config stored in the ui_only parameter

Code Block
languagepy
themeDJango
class DemoStageImpl(BaseStageImpl[DemoStage]):

    def __init__(self, config: DemoStage):
        super().__init__(config)
    
    async def process(self, intermediate: Dict[str, Any], final: ImmutableDict[str, Any], request: Request) -> Any:
        # your logic in here
        pass

    async def post_process(self, intermediate: Dict[str, Any], final: ImmutableDict[str, Any], request: Request) -> Any:
        # your logic in here
        pass

    async def get_ui_config(self, request: Request) -> Any:
        # your logic in here
        pass



on the __init__ file


5) Add the import for you stage:

  • This is for a better access when importing the stage
Code Block
languagepy
themeDJango
from .demo import DemoStage, DemoStageImpl



on the __init__ file of the stages package


6) Add the import all from the package of your stage

  • This is for a better access when importing the stage,
  • At this point you can import the stage with app.pipeline.stages
Code Block
languagepy
themeDJango
from .dynamic_agg import *
from .dynamic_results import *
from .filter import *
from .highlight import *
from .query import *
from .saga_query import *
from .sample_query import *
from .parallel_wrapper import *
# Your new stage
from .demo import *

Bare Minimum Stage

Code Block
languagepy
themeDJango
from typing import Any, Dict

from starlette.requests import Request

from app.pipeline import BaseStage, BaseStageImpl
from utils.data import ImmutableDict


class DemoStage(BaseStage):
    pass


class DemoStageImpl(BaseStageImpl[DemoStage]):

    def __init__(self, config: DemoStage):
        super().__init__(config)

    async def process(self, intermediate: Dict[str, Any], final: ImmutableDict[str, Any], request: Request) -> Any:
        # your logic in here
        pass

    async def post_process(self, intermediate: Dict[str, Any], final: ImmutableDict[str, Any], request: Request) -> Any:
        # your logic in here
        pass

    async def get_ui_config(self, request: Request) -> Any:
        # your logic in here
        pass