Experiments and Jobs#

Summary: What is an Experiment?#

An experiment is a resource in Dioptra that is effectively a namespace for jobs. An experiment can have a number of entrypoints associated with it, and within that experiment those entrypoints can be run as jobs. Experiments are mainly an organizational resource, providing a way to filter jobs, provide access to users via groups, and reduce the number of entrypoints to sift through at job creation.

Summary: What is a Job?#

A job in Dioptra is essentially an execution of an entrypoint with specified parameter values. When creating a job, a user provides parameters and artifact parameters that are required by an entrypoint. The set of workflow instructions (the entrypoint), parameter values, and any artifact parameters are sent to the queue selected by the user at job creation.

When a worker listening to that queue claims the job, it attempts to execute the provided entrypoint using the given parameter and artifact parameter values within the worker environment.

Any logs generated during the lifetime of the job, along with any metrics or created artifacts are uploaded to the Dioptra RESTAPI and associated with the job (and are viewable from the Job Dashboard page.)

Dioptra maintains a job history by recording the experiment, entrypoint, parameters, artifact parameters, logs, metrics and generated artifacts for all jobs. Even if a user deletes or changes an entrypoint or plugin associated with a job, Dioptra retains the executed version as a snapshot.

See Also#