Deployment Folder Reference#

This reference describes the files and folders created by the Dioptra deployment cookiecutter template. Understanding this structure will help you customize and troubleshoot your deployment.

See also

Prepare Your Deployment - Step-by-step guide for customizing your deployment.

Note

The dioptra-deployment prefix used in file and service names throughout this document assumes the default value for the deployment_name template variable. If you set this variable to a different value, the prefixes will change accordingly.

Folder Tree#

The following tree shows all files created by the template with annotations explaining their purpose.

.
├── config/
│   ├── db/
│   │   └── init-db.sh
│   ├── minio/
│   │   └── mlflow-tracking-readwrite-policy.json
│   └── nginx/
│       ├── http_*.conf
│       ├── https_*.conf
│       └── stream_db.conf
├── envs/
│   ├── ca-certificates.env
│   ├── dioptra-deployment-db.env
│   ├── dioptra-deployment-dbadmin.env
│   ├── dioptra-deployment-mlflow-tracking.env
│   ├── dioptra-deployment-restapi.env
│   ├── dioptra-deployment-worker-cpu.env
│   └── dioptra-deployment-worker.env
├── scripts/
│   ├── templates/
│   │   ├── dioptra.service.j2
│   │   ├── dot-env.j2
│   │   ├── minio-accounts.env.j2
│   │   └── postgres-passwd.env.j2
│   ├── copy-extra-ca-certificates.m4
│   ├── file-copy.m4
│   ├── generate_password_templates.py
│   ├── git-clone.m4
│   ├── globbed-copy.m4
│   ├── init-minio.sh
│   ├── init-named-volumes.m4
│   ├── init-scripts.sh
│   ├── manage-postgres-ssl.m4
│   └── set-permissions.m4
├── secrets/
│   ├── dioptra-deployment-minio-accounts.env
│   └── postgres-passwd.env
├── ssl/
│   ├── ca-certificates/
│   │   └── README.md
│   ├── db/
│   │   └── README.md
│   └── nginx/
│       └── README.md
├── systemd/
│   └── dioptra.service
├── .env
├── .gitignore
├── docker-compose.init.yml
├── docker-compose.override.yml.template
├── docker-compose.yml
├── init-deployment.sh
└── README.md

config/ Directory#

Contains configuration files that are copied into named volumes during deployment initialization.

config/db/#

init-db.sh

PostgreSQL initialization script that creates the database accounts and databases used by Dioptra.

config/minio/#

These JSON files define access policies for the Minio S3 storage service:

mlflow-tracking-readwrite-policy.json

Configures a readwrite role for the artifacts/ folder in the mlflow-tracking bucket.

config/nginx/#

NGINX reverse proxy configuration files:

http_*.conf

Configure NGINX to serve each service over HTTP connections:

  • http_default.conf - Basic NGINX defaults for HTTP

  • http_proxy.conf - General reverse proxy settings, including request header modifications

  • http_restapi.conf - Dioptra REST API service

  • http_mlflow.conf - MLflow Tracking service

  • http_minio.conf - Minio S3 service

  • http_dbadmin.conf - pgadmin4 dashboard

https_*.conf

Configure NGINX to serve each service over HTTPS connections (used when SSL/TLS is enabled):

  • https_default.conf - HTTPS defaults, including paths to server certificate and private key

  • https_proxy.conf - Reverse proxy settings for HTTPS

  • https_restapi.conf, https_mlflow.conf, https_minio.conf, https_dbadmin.conf - HTTPS versions of each service

stream_db.conf

Configures NGINX to proxy TCP requests to the PostgreSQL database. This is a simple passthrough; any SSL encryption must be enabled in the PostgreSQL service itself.

envs/ Directory#

Contains environment variable files that configure each service. These files are safe to commit to version control.

ca-certificates.env

Sets environment variables that applications use to find extra CA certificates.

  • AWS_CA_BUNDLE

  • CURL_CA_BUNDLE

  • NODE_EXTRA_CA_CERTS

  • REQUESTS_CA_BUNDLE

  • SSL_CERT_FILE

dioptra-deployment-db.env

Customizes the PostgreSQL database service.

  • POSTGRES_HOST_AUTH_METHOD

  • POSTGRES_PASSWORD_FILE

dioptra-deployment-dbadmin.env

Customizes the pgadmin4 dashboard.

  • PGADMIN_DEFAULT_EMAIL

  • PGADMIN_DISABLE_POSTFIX

  • PGADMIN_LISTEN_PORT

dioptra-deployment-mlflow-tracking.env

Customizes the MLflow Tracking service.

  • MLFLOW_S3_ENDPOINT_URL

dioptra-deployment-restapi.env

Customizes the Dioptra REST API service.

  • DIOPTRA_RESTAPI_ENV

  • MLFLOW_S3_ENDPOINT_URL

  • MLFLOW_TRACKING_URI

  • RQ_REDIS_URI

dioptra-deployment-worker-cpu.env

Sets Intel MKL/OpenMP threading optimizations for CPU-based Dioptra workers.

  • KMP_AFFINITY

  • KMP_BLOCKTIME

  • KMP_SETTINGS

  • OMP_PROC_BIND

dioptra-deployment-worker.env

Customizes all Dioptra workers (both CPU and GPU).

  • MLFLOW_TRACKING_URI

  • MLFLOW_S3_ENDPOINT_URL

  • RQ_REDIS_URI

  • DIOPTRA_API

scripts/ Directory#

Contains initialization scripts used by init-deployment.sh.

scripts/templates/#

Jinja2 templates used to generate configuration files:

dioptra.service.j2

Template for generating the systemd service file.

dot-env.j2

Template for generating the .env file containing environment variables and passwords.

minio-accounts.env.j2

Template for generating Minio account credentials.

postgres-passwd.env.j2

Template for generating PostgreSQL passwords.

scripts/ (root)#

generate_password_templates.py

Python script that generates random passwords to secure Dioptra’s services.

init-deployment.sh (see Root Directory section)

Main initialization script.

init-minio.sh

Sets Minio policies defined in config/minio/.

init-scripts.sh

Translates .m4 files into executable bash scripts using argbash.

*.m4 files

M4 macro files used during initialization:

  • copy-extra-ca-certificates.m4 - Injects extra CA certificates

  • file-copy.m4 - Handles file copying with logging and permissions

  • git-clone.m4 - Handles git repository cloning

  • globbed-copy.m4 - Handles globbed file copying

  • init-named-volumes.m4 - Prepares named storage volumes

  • manage-postgres-ssl.m4 - Enables/disables PostgreSQL SSL

  • set-permissions.m4 - Sets file/folder access permissions

secrets/ Directory#

Warning

Files in this directory contain sensitive passwords. DO NOT commit these files to version control or share them.

dioptra-deployment-minio-accounts.env

Minio account credentials generated by the initialization script.

  • MINIO_ROOT_USER

  • MINIO_ROOT_PASSWORD

  • MINIO_MLFLOW_TRACKING_USER

  • MINIO_MLFLOW_TRACKING_PASSWORD

  • MINIO_MLFLOW_TRACKING_POLICIES

  • MINIO_RESTAPI_USER

  • MINIO_RESTAPI_PASSWORD

  • MINIO_RESTAPI_POLICIES

  • MINIO_WORKER_USER

  • MINIO_WORKER_PASSWORD

  • MINIO_WORKER_POLICIES

postgres-passwd.env

PostgreSQL passwords generated by the initialization script.

  • POSTGRES_PASSWORD

  • POSTGRES_USER

  • POSTGRES_DB

ssl/ Directory#

Contains SSL/TLS certificates and private keys.

ssl/ca-certificates/#

Place extra CA certificates here if your environment uses an internal certificate authority. See Add Custom CA Certificates for details.

ssl/db/#

Place the PostgreSQL server certificate (server.crt) and private key (server.key) here to enable encrypted database connections. See Enable SSL/TLS in NGINX and Postgres for details.

Warning

The private key file (server.key) is sensitive. DO NOT commit it to version control.

ssl/nginx/#

Place the NGINX server certificate (server.crt) and private key (server.key) here to enable HTTPS. See Enable SSL/TLS in NGINX and Postgres for details.

Warning

The private key file (server.key) is sensitive. DO NOT commit it to version control.

systemd/ Directory#

dioptra.service

A systemd service file for managing the Dioptra deployment on Linux systems that use systemd (Ubuntu, Fedora, RHEL, etc.).

Root Directory#

.env

Contains secrets (passwords) mapped to environment variables. Generated by the initialization script. These variables are referenced in docker-compose.yml using ${VARIABLE} syntax.

  • DIOPTRA_RESTAPI_DATABASE_URI

  • DIOPTRA_MLFLOW_TRACKING_DATABASE_URI

  • DIOPTRA_WORKER_USERNAME

  • DIOPTRA_WORKER_PASSWORD

  • MINIO_ROOT_USER

  • MINIO_ROOT_PASSWORD

  • MINIO_KMS_SECRET_KEY

  • MLFLOW_SERVER_ALLOWED_HOSTS

  • MLFLOW_TRACKING_AWS_ACCESS_KEY_ID

  • MLFLOW_TRACKING_AWS_SECRET_ACCESS_KEY

  • PGADMIN_DEFAULT_PASSWORD

  • POSTGRES_USER_DIOPTRA_PASSWORD

  • RESTAPI_AWS_ACCESS_KEY_ID

  • RESTAPI_AWS_SECRET_ACCESS_KEY

  • WORKER_AWS_ACCESS_KEY_ID

  • WORKER_AWS_SECRET_ACCESS_KEY

Warning

DO NOT commit this file to version control.

.gitignore

Configures files and directories that git should ignore if the deployment is placed under version control.

docker-compose.init.yml

Docker Compose file used during the initialization process.

docker-compose.override.yml.template

Template for creating custom Docker Compose overrides. Copy to docker-compose.override.yml and modify. See Mount Data Volumes, Configure GPU Workers, and Integrate Custom Containers for common customizations.

docker-compose.yml

Main Docker Compose file that defines all Dioptra services.

init-deployment.sh

Main deployment initialization script. See init-deployment.sh Script Reference for command-line options and usage examples.

README.md

Instructions for initializing and running Dioptra.