Plugins#

Summary: What is a Plugin?#

A Plugin is a logical container used to organize Python code within Dioptra.

Plugins hold the Python files that define the experiment logic. These files define Tasks—the individual units of work that the Dioptra engine executes. There are two distinct types of tasks within a plugin:

  1. Plugin Function Tasks: These act as the computational steps in a workflow.

    • Example: A task that accepts a training dataset and returns a trained ML model.

  2. Plugin Artifact Tasks: These handle the interface between memory and disk storage (I/O).

    • Example: A task that serializes a trained model object to a .pt (PyTorch) file and handles the subsequent deserialization back into memory for potential downstream tasks.

Plugin Function Tasks#

Plugin Function Tasks are Python functions that have been registered with Dioptra. They serve as the fundamental “verbs” or actions in entrypoint workflow graphs.

These tasks perform specific units of work, such as training a model, calculating a metric, or transforming data. Like standard Python functions, they accept inputs (arguments) and produce return values.

Data Flow and Chaining

When chained together in an entrypoint, the output of one function task can flow automatically into the input of other function tasks. These tasks operate primarily on in-memory objects (such as DataFrames, integers, or model objects).

Note that while Function Tasks process data, they generally do not save it to disk. To persist results, the output of a function task must be passed into the artifact task graph to dictate which artifact task will save it to disk.

See Entrypoints Explanation to learn how tasks are chained and how default parameter values are set.

Function Task Registration#

Writing the Python code is only the first step; function tasks must be registered to be visible to the Dioptra engine.

Registration defines:

  • The Function Task Name: Always equal to the function name in Python - this is the identifier used to reference the task in entrypoint graphs

  • Input/Output Names: This is how input arguments are set in entrypoints and how output objects are referenced

  • Input/Output Types: Defining the plugin parameter types for each input/output allows for entrypoint task graph validity

When creating Python plugins, users manually register each function task. This can be done in the user interface or through the REST API. See detailed steps in How To Create Plugins

Note

In order for a Python function to be registrable, it must use the @pyplugs.register decorator. View Plugins Reference to learn about the @pyplugs.register syntax for function plugin tasks.

Plugin Artifact Tasks#

Plugin Artifact Tasks manage data persistence. Because complex Python objects (like machine learning models or large arrays) cannot be plainly written to a text file, they require specific serialization logic.

By isolating I/O logic in Artifact Tasks, the main computational workflow (Function Tasks) remains decoupled from file system operations, making code cleaner and more portable.

Artifact Handlers are Python classes that define certain methods for I/O logic.

Note

Artifact Handler classes must inherit from the parent class ArtifactTaskInterface. See Plugins Reference to learn about the syntax for creating artifact tasks.

Artifact Task Registration#

Artifact tasks are registered similarly to function tasks.

When an artifact task is registered, the user defines:

  • The Artifact Task Name: Always equal to the name of the Python class that defines the artifact handling logic

  • Output Parameters: The name and the type of the return object from the deserialize method

Using Function Tasks and Artifact Tasks#

Plugins create the “vocabulary” of tasks available to an experiment. To utilize these tasks, a relevant plugin must be attached to an Entrypoint.

Function tasks are used in the entrypoint task graph. Artifact tasks are used in the artifact output graph to save artifacts, and in the artifact input parameters to make saved artifacts available to an entrypoint task graph.

Entrypoints utilize the types that are declared in function tasks inputs / outputs to ensure the workflow is valid.

See Entrypoints Explanation to learn how plugins are used in entrypoints

See Also#