Dioptra Components#

Reference material for the various components and resources that comprise experiment workflows in Dioptra.

Table of Contents

Quick Definitions#

Plugins#

  • Plugin: A collection of Python files which contain registered Function Tasks and/or Artifact Tasks.

  • Plugin Parameter Type: Either a built-in or user-defined type, used for type validation. Function Task Inputs, Function Task Outputs, Artifact Task Outputs, Entrypoint Parameters and the outputs from Entrypoint Artifact Parameters all have associated Plugin Parameter Types.

Plugin Function Tasks#

  • Function Task: A registered Python function within a Plugin file that defines some computational task, used in the Task Graph in Entrypoints.

  • Function Task Input: An input into a Function Task registered as part of the task definition. Has an associated Plugin Parameter Type for type validation.

  • Function Task Output: The output of a Function Task registered as part of the task definition. Has an associated Plugin Parameter Type for type validation. A Function Task Output can be fed in as a Function Task Input via the Task Graph, or saved as an Artifact via the Artifact Output Graph.

Plugin Artifact Tasks and Artifacts#

  • Artifact: The output of a Function Task that has been saved to disk via the serialize method of an Artifact Task.

  • Artifact Task: A registered subclass of ArtifactTaskInterface that defines serialization, deserialization, and validation logic for an Artifact type. Used to save Artifacts in the Artifact Output Graph in Entrypoints, and also to load saved Artifacts as Entrypoint Artifact Parameters at Job runtime.

  • Artifact Task Output: An object that is returned by the deserialize method of an Artifact Task. Has an associated Plugin Parameter Type for type validation.

Entrypoints#

  • Entrypoint: Define parameterizable workflows, which can be reused across multiple Experiments.

  • Entrypoint Parameter: An input used for the Task Graph and the Artifact Output Graph. Can be customized during a Job run and have an optional default value. Has an associated Plugin Parameter Type for type validation.

  • Entrypoint Artifact Parameter: A kind of Entrypoint Parameter, where the value is read in from a Snapshot of an Artifact during Job execution. Can contain multiple objects, each of which has an associated Plugin Parameter Type for type validation.

  • Task Graph: The sequence of Function Tasks that an Entrypoint executes, written in YAML. A directed, acyclic graph (DAG).

  • Task Graph Step: A single step in the Task Graph, which must have a name. Can have positional and keyword arguments as well.

  • Artifact Output Graph: The logic dictating which Function Task Outputs are saved to Artifacts and how (i.e. which Artifact Tasks are invoked and their inputs). Written in YAML.

Experiments/Jobs#

  • Experiment: A logical container that holds Entrypoints and Job runs.

  • Job: A parameterized run of an Entrypoint.

  • Queue: A queue manager for Jobs. Manages Job execution for a specific worker environment.

  • Worker: Contains the resources for executing a Job. Default worker containers are available for GPU and CPU hardware. Custom workers can be developed.

  • Metric: A numeric value that is associated with a specific step of a Job (i.e. training accuracy at a given epoch for the “train” Function Task).

Other#

  • User: A user profile with login credentials and permissions. Belongs to one or more Groups.

  • Group: A set of Users and permission rules for resource access. Currently, there is only a single “public” Group.

  • Snapshot: A specific version of a Resource (e.g. Experiment, Plugin, Entrypoint, Artifact) in time.

  • Tag: String values that are associated in many to many relationships with various Dioptra resources. Used for filtering tables, etc.

See Also#