Getting Started

This guide covers Packflow installation and basic usage of the CLI to create a simple Packflow project.

Installing Packflow

Prerequisites

• Python (version 3.10+)

Install from PyPI

Packflow can be installed directly from PyPI:

pip install packflow

Install from Source

For development or to install from source:

# Clone repository and navigate to the root directory
git clone https://github.com/dow-cdao/packflow.git
cd packflow

# Install package
pip install .

# For contributors: install in editable mode
pip install -e .

Viewing Pre-built Documentation

The simplest way to view the Packflow documentation is to serve the pre-built HTML files included in the repository:

# Navigate to the pre-built docs folder
cd docs/built/html

# Start a local web server
python -m http.server 8000

# Access the documentation in a web browser at http://127.0.0.1:8000/

Important

If a “Not Found” error page is received when first accessing the documentation, wait a moment for the server to fully start and refresh the page.

Building Documentation from Source

The following are required to build documentation from source:

• Python (version 3.10+)

• Pip

• Packflow (the version corresponding to the docs being served)

• Pandoc - Must be installed separately, from system package manager (see Pandoc installation instructions)

• make command (xcode-select on macOS and WSL on Windows, or build-essential on Linux)

Steps to build and serve documentation:

# Navigate to docs folder
cd docs

# Install Python dependencies
pip install -r requirements.txt

# Serve documentation with live updates (development)
make dev

# OR serve static multi-version documentation (production)
make prod-serve

# Access the documentation in a web browser at http://127.0.0.1:8000/

Creating a Packflow Project

This section covers the initial setup process for creating a Packflow project, defining an Inference Backend, and running Packflow’s validation checks on the input/output requirements of the Inference Backend.

Step 1: Create the project structure

Initialize a new project by running packflow create hello-world. This will create a new directory named hello-world that contains the following directory structure:

hello-world/
├── packflow.yaml
├── LICENSE.txt
├── MODEL_CARD.md
├── README.md
├── requirements.txt
├── inference.py
├── validate.py
└── test_inference.py

Step 2: Write the Inference Backend

Open the inference.py with a code or text editor of your choice. Some templated code will be provided. Populate the execute() function with logic to double the value under the key ‘number’, and return the doubled number:

inference.py

from typing import Any, List

from packflow import InferenceBackend


class Backend(InferenceBackend):
    def transform_inputs(self, inputs: List[dict]) -> Any:
        """
        Preprocessing steps or other transformations before running inference.
        ...
        """
        return inputs

    def execute(self, inputs: Any) -> Any:
        """The main execution of inference or analysis for the developed application.

        This method should remain targeted to passing data through the model/execution
        code for profiling purposes. Minimal pre- or post-processing should occur at this
        step unless completely necessary.

        Parameters
        ----------
        inputs: List[Dict]
            The output of the transform_inputs method. If the transform_inputs method is
            not overridden, the data is formatted as records (list of dictionaries)

        Returns
        -------
        Any
            Model Outputs

        Notes
        -----
        The transform_outputs() method should handle all postprocessing including calculating
        metrics, converting outputs back to Python types, and other postprocessing steps. Try
        to keep this method focused purely on inference/analysis.
        """
        outputs = []
        for row in inputs:
            outputs.append({"doubled": row["number"] * 2})

        return outputs

    def transform_outputs(self, outputs):
        """
        Postprocessing steps or other transformation steps to be executed prior to
        returning outputs.
        ...
        """
        return outputs

The Inference Backend is now ready to be loaded, validated, and shared.

Step 3: Local Validation

Now that the Inference Backend is written, use the built-in validation to ensure it will run as expected in production.

This can be done programmatically. Open the validate.py script and modify it to match the Inference Backend’s inputs:

validate.py

# Import Packflow's dev tools to run validations on the Inference Backend
from packflow.loaders import LocalLoader

# Load the backend in the current directory
# The path 'inference:Backend' can be interpreted as
#   `from inference import Backend`
backend = LocalLoader("inference:Backend").load()

# Define sample inputs that represent realistic data for the backend.
# These should exercise the expected input format(s) the backend will receive.
SAMPLE_SINGLE_ROW = {"number": 5}

SAMPLE_BATCH = [
    {"number": 5},
    {"number": 10},
    # Add more sample rows as needed
]

if __name__ == "__main__":
    print("Running validation...")

    print(f"Sample single row: {SAMPLE_SINGLE_ROW}\n")
    print(f"Sample batch: {SAMPLE_BATCH}\n")

    # backend.validate() checks that the outputs meet Packflow's format
    # requirements. Returns outputs if valid.
    outputs_single_row = backend.validate(SAMPLE_SINGLE_ROW)
    outputs_batch = backend.validate(SAMPLE_BATCH)

    print(f"Outputs single row: {outputs_single_row}")
    print(f"Outputs batch: {outputs_batch}")
    print("\nValidation passed!")

Note

Validation can be run via the validate.py file, or directly from a Notebook. However the path will need to be updated if it is not running in the same directory

Passing "inference:Backend" to the Local Loader is roughly equal to from inference import Backend. If the script is nested further, the path can be separated via dot notation, such as src.mypackage.inference:Backend.

If any validations fail, an exception message containing details of the issue and what needs to be fixed will be returned.

Next Steps

Please see the Creating a Custom Backend section of this site for more detailed information on building custom Inference Backends with Packflow.