Intended Audience

We envision an audience with a wide variety of familiarity with and expertise in machine learning (ML) and artificial intelligence (AI). Newcomers to the platform will be able to run the included demonstrations even if they have very little programming experience.

The test platform offers new opportunities for analysts in organizations using AI/ML-enabled products. For example, the inclusion of testing as part of the build pipeline for first-party developers would help promote more robust products. Similarly, second-party consumers of such products might leverage the test platform as part of a verification or risk assessment process. In both cases, the test platform allows users to test a product against a range of evaluations thereby helping them understand what types of risks should be prioritized for mitigation.

We also envision Dioptra as an asset to AI/ML researchers in developing new solutions. The test platform will allow them to evaluate the security and trustworthiness of any new techniques they develop by running those techniques through their paces. Similarly, it can facilitate “parameter sweeping” to help developers better understand the degree to which small changes in parameters can affect an algorithm. It also affords an opportunity to replicate and benchmark well-known results from the research literature. This ability to repeat experiments to reproduce results is critical for creating and validating reliable metrics.

We envision four primary user levels.

Level 1—The Newcomer

These individuals have little or no hands-on experience with the test platform. They will be able to read the documentation and run the provided demos to learn how to use the test platform. They will be able to alter the parameters of the provided demos to create slight variants of the existing experiments. These users can have a wide variety of skill sets. They need not be familiar with the technologies the test platform uses, nor do they have to have much experience with scripting or programming.

Level 2—The Analyst

These are individuals who want to analyze a wider variety of scenarios. They will be able to interface with the test platform’s REST (REpresentational State Transfer) API to create new experiments from existing entrypoints. They will also learn to create custom entrypoints from existing plugins. They must know how to customize the test platform’s code templates; thus a basic knowledge of scripting or programming is required.

Level 3—The Researcher

These are individuals who want to run experiments using novel metrics, algorithms, and analytical techniques. They will be able to implement their own “in-house” plugins to create novel entrypoints that rely on custom algorithms. They will need to understand the test platform’s plugin architecture to extend it with new functionality. They, therefore, require a solid background in scripting or programming.

Level 4—The Developer

These are individuals who want to expand the test platform’s core capabilities by contributing to the distribution. They will add new features by implementing new plugins, REST API endpoints, and architecture extensions. These individuals will have a deep understanding of how the test platform’s architectural and software components work together. They will be able to write reusable code and program applications that conform to coding best practices.