Plugins

Plugin Definition

A Plugin in Dioptra acts as a logical container for code, functionally similar to a standard Python package.

Requirements

• Naming: Must be a valid Python module name (alphanumeric, underscores only). Must be unique within the Dioptra deployment.

• Structure: A flat directory containing .py files.

• Init File: Unlike standard Python packages, an __init__.py file is not required.

• Content: Files may contain Function Task definitions or Artifact Task definitions.

Plugin Function Tasks

Function tasks are the primary computational units of an experiment.

Code Syntax

To define a function task, you must write a standard Python function and decorate it with @pyplugs.register.

Example Implementation

Function Task Example

import structlog

from dioptra import pyplugs

LOGGER = structlog.get_logger()


@pyplugs.register
def hello(name: str) -> str:
    message = f"Hello, {name}"
    LOGGER.info(message)
    return message

Type Hints Type hints are strongly recommended. Dioptra uses them to automatically populate parameter types during registration imports.

Registration Configuration

When registering a function task (via API or GUI), the configuration must match the code logic.

Parameter Mapping

• Input Names: The registered input parameter names must match the Python function argument names exactly.

• Order: [INSERT EXPLANATION ABOUT POSITIONAL VS KEYWORD ARGUMENTS HERE]

• *Args / **Kwargs: [INSERT EXPLANATION ABOUT HOW VARIABLE ARGUMENTS ARE HANDLED HERE]

Return Values & Multiple Outputs

A function task may return a single value or multiple values (via a tuple or list).

• Single Return: Maps to a single registered Output Parameter.

• Iterable Return (Tuple/List): If the function returns an iterable, Dioptra maps the values to the registered Output Parameters in order (Index 0 to Output 1, Index 1 to Output 2).

• Partial Mapping: If fewer output parameters are registered than values returned, Dioptra discards the excess trailing values.

Plugin Artifact Tasks

Artifact tasks handle the serialization and deserialization of complex objects.

Code Syntax

Artifact tasks must be defined as classes that inherit from the ArtifactTaskInterface.

Parent Class Definition

class ArtifactTaskInterface(name, bases, namespace, /, **kwargs)

Interface class which exposes the logic for serialization and deserialization of an artifact.

It is expected that implementations will override all methods in this interface.

abstract static serialize(working_dir: pathlib.Path, name: str, contents: Any, **kwargs) → pathlib.Path

Called to serialize an artifact. The value and type of contents is dependent on the artifact output declaration.

Parameters

• working_dir – the path to which any artifact should be serialized

• name – the name of the artifact; in general, the name should be used as the stem or name of the file

• contents – the contents to serialize. Specific implementations should specify which type they support.

• kwargs – any additional keyword arguments that this serializer uses. Keyword arguments are added to the validation method below

Returns

A Path pointing to the location where the artifact has been serialized. This could be either a directory or a file depending on the type of artifact.

abstract static deserialize(working_dir: pathlib.Path, path: str, **kwargs) → Any

Called to deserialize an artifact.

Parameters

• working_dir – the working directory which serves as the directory to process the artifact in. Any intermediate files created as part of the deserialization process should be produced in this directory.

• path – the path to the artifact to deserialize and is relative to the value of working_dir. This means implementations should access the artifact by creating a path object along the lines of Path(working_dir, path).

• kwargs – any extra keyword arguments to use as part of deserialization (unused at this time)

Returns

The result of deserialization. This could be any type or value.

abstract static validation() → dict[str, Any] | None

The validation is inserted into the JSON Schema to validate any extra keyword arguments that are passed to the serialization method of this artifact task. The validation indicates the name and type of the arguments. See https://json-schema.org/docs for more information. The version of JSON schema used at this time is https://json-schema.org/draft/2020-12/schema.

Example

The following dictionary specifies three extra keyword arguments of foo, bar, and baz and their types.

• foo is a string.

• bar is an enumerated value with three possible values.

• baz is an integer.

{
    "foo": {"type": "string"},
    "bar": {"enum": ["value1", "value2", "value3"]},
    "baz": {"type": "integer"},
}

Returns

A dictionary containing the validation information for the keyword arguments to serialize or None if no extra keyword arguments are used.

Example Implementation

Artifact Handler Example

Import Parent Class

from dioptra.sdk.api.artifact import ArtifactTaskInterface
from pathlib import Path
from typing import Any

Child Class Definition

class StringArtifactTask(ArtifactTaskInterface):
    @staticmethod
    def serialize(
        working_dir: Path, name: str, contents: str, type: str = "txt", **kwargs
    ) -> Path:
        result = (working_dir / name).with_suffix("." + type)
        result.write_text(contents, encoding="UTF-8", newline="")
        return result

    @staticmethod
    def deserialize(working_dir: Path, path: str, **kwargs) -> str:
        return Path(working_dir, path).read_text()

    @staticmethod
    def validation() -> dict[str, Any] | None:
        return {"type": {"type": "string"}}

Registration Interfaces

Plugins can be created and tasks can be registered programmatically via the Python Client or the REST API. They can also be registered manually through the guided user interface.

Using Python Client

Create a Plugin Container

PluginsCollectionClient.create(group_id: int, name: str, description: str | None = None) → dioptra.client.plugins.T

Creates a plugin.

Parameters

• group_id – The id of the group that will own the plugin.

• name – The name of the new plugin.

• description – The description of the new plugin. Optional, defaults to None.

Example

Create a plugin called “hello” with a description.

plugin = client.plugins.create(
    group_id, "hello", "This is a Hello World Plugin"
)

Returns

The response from the Dioptra API.

Add a Python file and register function tasks / artifact tasks

PluginFilesSubCollectionClient.create(plugin_id: str | int, filename: str, contents: str, function_tasks: list[dict[str, Any]] | None = None, artifact_tasks: list[dict[str, Any]] | None = None, description: str | None = None) → dioptra.client.plugins.T

Creates a plugin file.

Either function_tasks or artifact_tasks should be provided. Mixing the two types of tasks is not recommended.

Parameters

• plugin_id – The id for the plugin that will own the new plugin file.

• filename – The filename for the new plugin file.

• contents – The contents of the new Python file.

• function_tasks – The information needed to register plugin function tasks contained in the plugin file, a list. Can be empty.

• artifact_tasks – The information needed to register plugin artifact tasks contained in the plugin file, a list. Can be empty.

• description – The description of the new plugin file. Optional, defaults to None.

Example

Attach Python code embedded in a string to a Plugin, and register a task with two input parameters and one output parameter.

file = client.plugins.files.create(
    plugin_id=plugin["id"],
    filename="hello.py",
    content=PYTHON_CONTENTS,
    tasks=[
        {
            "name": "hello_world",
            "inputParams": [
                {
                    "name": "greeting",
                    "parameterType": string_param_type_id,
                    "required": True,
                },
                {
                    "name": "name",
                    "parameterType": string_param_type_id,
                    "required": True,
                },
            ],
            "outputParams": [
                {
                    "name": "message",
                    "parameterType": string_param_type_id,
                }
            ],
        }
    ],
)

Returns

The response from the Dioptra API.

Using REST API

Plugins can be created and and tasks can be registered directly via the HTTP API.

Create Plugins

See the POST /api/v1/plugins endpoint documentation for payload requirements.

Add Files

See the POST /api/v1/plugins/{int:id}/files endpoint documentation for payload requirements.

See Also

• What are Plugins?

• <how-to-create-plugins>