Worker Container Requirements Reference#

This reference describes the minimal requirements for integrating a custom worker container into a Dioptra deployment using Docker Compose. These are the requirements you must satisfy regardless of whether you use the Dioptra templates or build a worker container from scratch.

Environment Variables#

The worker container must provide all environment variables required by the dioptra-worker-v1 process (see dioptra-worker-v1 Reference) as well as credentials for S3-compatible artifact storage.

Variable

Description

MLFLOW_TRACKING_URI

MLflow tracking server URL (consumed natively by MLflow).

MLFLOW_S3_ENDPOINT_URL

S3-compatible endpoint URL for artifact storage (consumed natively by MLflow).

DIOPTRA_API

Dioptra REST API base URL.

DIOPTRA_WORKER_USERNAME

Username for worker authentication with the REST API.

DIOPTRA_WORKER_PASSWORD

Password for worker authentication with the REST API.

AWS_ACCESS_KEY_ID

S3/MinIO access key (consumed by boto3/MLflow).

AWS_SECRET_ACCESS_KEY

S3/MinIO secret key (consumed by boto3/MLflow).

The first five variables are checked by dioptra-worker-v1 at startup; the process exits immediately if any are missing. AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are consumed by underlying libraries (boto3/MLflow) and must be present for artifact storage operations to succeed.

Note

The Dioptra deployment template supplies these variables through a combination of env files and the docker-compose.yml environment: block. See Deployment Template Reference and Deployment Folder Reference.

Worker Process#

The container must run the dioptra-worker-v1 command, which is provided by the dioptra-platform Python package. This can be done in one of two ways:

  • Directly in the Dockerfile ENTRYPOINT or CMD: invoke dioptra-worker-v1 with the required --url and queue name arguments.

  • Via an entrypoint script: create a shell script that handles service startup ordering and then invokes dioptra-worker-v1.

The minimal invocation is:

dioptra-worker-v1 --url <redis-uri> <queue-name>

See dioptra-worker-v1 Reference for the full command-line reference.

The Dioptra provided workers and the custom worker cookiecutter template both use an entrypoint script that reads the Redis URI from the RQ_REDIS_URI environment variable, waits for dependent services to become reachable, and then starts the worker. Queue names are passed as positional arguments through the command: section in docker-compose.

Network#

The worker container must join the dioptra Docker network to communicate with other Dioptra services (Redis, MinIO, REST API, MLflow Tracking).

services:
  custom-worker:
    # ...
    networks:
      - dioptra

See Also#