Queues and Workers#

Summary: What is a Queue?#

A queue is a resource in Dioptra that represents a logical job queue. Workers watch a queue for new jobs to process. Entrypoints can be assigned to any number of queues for which they are compatible.

When submitting a job, the user selects a queue for that job. Any worker listening to that queue can claim the job and then begin execution in its environment.

Summary: What is a Worker?#

A worker is an environment for executing jobs.

A worker should contain all of the requirements needed for a given entrypoint, such as any local files, python packages, or executables. Each worker listens to a single named queue, and multiple workers can listen to the same queue.

By default, Dioptra provides several default worker environments that can be enabled during setup:

  • tensorflow-cpu - for systems that do not have a GPU present, which contains Tensorflow and ART as dependencies

  • tensorflow-gpu - for systems that do have a GPU present, which contains Tensorflow and ART as dependencies

  • pytorch-cpu - for systems that do not have a GPU present, which contains PyTorch and ART as dependencies

  • pytorch-gpu - for systems that do have a GPU present, which contains PyTorch and ART as dependencies

If a worker is enabled, creating a queue with the same name as the worker will allow jobs to be run in the corresponding worker container.

Additionally, custom workers can be created to watch queues with other names. These custom workers can provide different environments for jobs, which allows users to include their own requirements and packages.

See Create a Custom Worker Container for more information.

See Also#