Architecture Overview#

The Dioptra test platform is built on a microservices architecture. It is designed to be deployed across several physical machines but is equally deployable on a local laptop. The distributed deployment allows the core optimization algorithms to reside on machines with GPUs or other high-powered computational resources, while a local deployment will impose strong computational constraints.

The heart of the architecture is the core test platform Application Programming Interface (API) that manages requests and responses with a human user via a reverse proxy. The backend Data Storage component hosts datasets, registered models and artifacts, and experiment results and metrics. As experiment jobs get submitted, the API registers them on the Redis queue, which is watched by a worker pool of Docker containers provisioned with all necessary environment dependencies. These worker containers run the plugins and coordinate job dependencies and record statistics, metrics, and any generated artifacts.

The user can interact with the API via a web-based GUI or Python client.

The architecture is built entirely from open-source resources making it easy for others to extend and improve upon.

See Also#