Testbed Client API Reference#

Note

See the Glossary for the meaning of the acronyms used in this guide.

class DioptraClient(address: str | None = None, api_version: str = 'v0')[source]#

Bases: object

Connects to the Dioptra REST api, and provides access to endpoints.

Parameters

  • address – Address of the Dioptra REST api or if no address is given the DIOPTRA_RESTAPI_URI environment variable is used.

  • api_version – The version of the Dioptra REST API to use. Defaults to “v0”.

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

property experiment_endpoint: str#

Experiment endpoint url

property job_endpoint: str#

Job endpoint url

property task_plugin_endpoint: str#

Task plugins endpoint url

property task_plugin_builtins_endpoint: str#

Builtin task plugins endpoint url

property task_plugin_custom_endpoint: str#

Custom task plugins endpoint url

property queue_endpoint: str#

Queue endpoint url

delete_custom_task_plugin(name: str) → dict[str, Any][source]#

Deletes a custom task plugin by its unique name.

Parameters

name – A unique string identifying a task plugin package within dioptra_custom collection.

Returns

The Dioptra REST api’s response.

Example:

{
    'collection': 'dioptra_custom',
    'status': 'Success',
    'taskPluginName': ['evaluation']
}

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

get_experiment_by_id(id: int) → dict[str, Any][source]#

Gets an experiment by its unique identifier.

Parameters

id – An integer identifying a registered experiment.

Returns

The Dioptra REST api’s response.

Example:

{
    'lastModified': '2023-06-22T13:42:35.379462',
    'experimentId': 10,
    'name': 'mnist_feature_squeezing',
    'createdOn': '2023-06-22T13:42:35.379462'
}

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

get_experiment_by_name(name: str) → dict[str, Any][source]#

Gets an experiment by its unique name.

Parameters

name – The name of the experiment.

Returns

The Dioptra REST api’s response containing the experiment id, name, and metadata.

Example:

{
    'experimentId': 1,
    'name': 'mnist',
    'createdOn': '2023-06-22T13:42:35.379462',
    'lastModified': '2023-06-22T13:42:35.379462'
}

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

get_job_by_id(id: str) → dict[str, Any][source]#

Gets a job by its unique identifier.

Parameters

id – A string specifying a job’s UUID.

Returns

The Dioptra REST api’s response.

Example:

{
    'mlflowRunId': None,
    'lastModified': '2023-06-26T15:26:43.100093',
    'experimentId': 10,
    'queueId': 2,
    'workflowUri': 's3://workflow/268a7620/workflows.tar.gz',
    'entryPoint': 'train',
    'dependsOn': None,
    'status': 'queued',
    'timeout': '24h',
    'jobId': '4eb2305e-57c3-4867-a59f-1a1ecd2033d4',
    'entryPointKwargs': '-P model_architecture=shallow_net -P epochs=3',
    'createdOn': '2023-06-26T15:26:43.100093'
}

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

get_queue_by_id(id: int) → dict[str, Any][source]#

Gets a queue by its unique identifier.

Parameters

id – An integer identifying a registered queue.

Returns

The Dioptra REST api’s response.

Example:

{
    'lastModified': '2023-04-24T20:53:09.801442',
    'name': 'tensorflow_cpu',
    'queueId': 1,
    'createdOn': '2023-04-24T20:53:09.801442'
}

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

get_queue_by_name(name: str) → dict[str, Any][source]#

Gets a queue by its unique name.

Parameters

name – The name of the queue.

Returns

The Dioptra REST api’s response.

Example:

{
    'lastModified': '2023-04-24T20:53:09.801442',
    'name': 'tensorflow_cpu',
    'queueId': 1,
    'createdOn': '2023-04-24T20:53:09.801442'
}

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

get_builtin_task_plugin(name: str) → dict[str, Any][source]#

Gets a custom builtin plugin by its unique name.

Parameters

name – A unique string identifying a task plugin package within dioptra_builtins collection.

Returns

The Dioptra REST api’s response.

Example:

{
    'taskPluginName': 'attacks',
    'collection': 'dioptra_builtins',
    'modules': ['__init__.py', 'fgm.py']
}

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

get_custom_task_plugin(name: str) → dict[str, Any][source]#

Gets a custom task plugin by its unique name.

Parameters

name – A unique string identifying a task plugin package within dioptra_builtins collection.

Returns

The Dioptra REST api’s response.

Example:

{
    'taskPluginName': 'custom_poisoning_plugins',
    'collection': 'dioptra_custom',
    'modules': [
        '__init__.py',
        'attacks_poison.py',
        'data_tensorflow.py',
        'datasetup.py',
        'defenses_image_preprocessing.py',
        'defenses_training.py',
        'estimators_keras_classifiers.py',
        'registry_art.py',
        'tensorflow.py'
    ]
}

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

list_experiments() → list[dict[str, Any]][source]#

Gets a list of all registered experiments.

Returns

A list of responses detailing all experiments.

Example:

[
    {
        'lastModified': '2023-04-24T20:20:27.315687',
        'experimentId': 1,
        'name': 'mnist',
        'createdOn': '2023-04-24T20:20:27.315687'
    },
    ...
    {
        'lastModified': '2023-06-22T13:42:35.379462',
        'experimentId': 10,
        'name': 'mnist_feature_squeezing',
        'createdOn': '2023-06-22T13:42:35.379462'
    }
]

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

list_jobs() → list[dict[str, Any]][source]#

Gets a list of all submitted jobs.

Returns

A list of responses detailing all jobs.

Example:

[
    {
        'mlflowRunId': None,
        'lastModified': '2023-04-24T20:54:30.722304',
        'experimentId': 2,
        'queueId': 2,
        'workflowUri': 's3://workflow/268a7620/workflows.tar.gz',
        'entryPoint': 'train',
        'dependsOn': None,
        'status': 'queued',
        'timeout': '1h',
        'jobId': 'a4c574dd-cbd1-43c9-9afe-17d69cd1c73d',
        'entryPointKwargs': '-P data_dir=/nfs/data/Mnist',
        'createdOn': '2023-04-24T20:54:30.722304'
    },
    ...
]

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

list_queues() → list[dict[str, Any]][source]#

Gets a list of all registered queues.

Returns

A list of responses detailing all registered queues.

Example:

[
    {
        'lastModified': '2023-04-24T20:53:09.801442',
        'name': 'tensorflow_cpu',
        'queueId': 1,
        'createdOn': '2023-04-24T20:53:09.801442'
    },
    {
        'lastModified': '2023-04-24T20:53:09.824101',
        'name': 'tensorflow_gpu',
        'queueId': 2,
        'createdOn': '2023-04-24T20:53:09.824101'
    },
    {
        'lastModified': '2023-04-24T20:53:09.867917',
        'name': 'pytorch_cpu',
        'queueId': 3,
        'createdOn': '2023-04-24T20:53:09.867917'
    },
    {
        'lastModified': '2023-04-24T20:53:09.893451',
        'name': 'pytorch_gpu',
        'queueId': 4,
        'createdOn': '2023-04-24T20:53:09.893451'
    }
]

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

list_all_task_plugins() → list[dict[str, Any]][source]#

Gets a list of all registered builtin task plugins.

Returns

A list of responses detailing all plugins.

Example:

[
    {
        'taskPluginName': 'artifacts',
        'collection': 'dioptra_builtins',
        'modules': ['__init__.py', 'mlflow.py', 'utils.py']
    },
    ...
    {
        'taskPluginName': 'pixel_threshold',
        'collection': 'dioptra_custom',
        'modules': ['__init__.py', 'pixelthreshold.py']
    }
]

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

list_builtin_task_plugins() → list[dict[str, Any]][source]#

Gets a list of all registered builtin task plugins.

Returns

A list of responses detailing all builtin plugins.

Example:

[
    {
        'taskPluginName': 'artifacts',
        'collection': 'dioptra_builtins',
        'modules': ['__init__.py', 'mlflow.py', 'utils.py']
    },
    ...
    {
        'taskPluginName': 'backend_configs',
        'collection': 'dioptra_builtins',
        'modules': ['__init__.py', 'tensorflow.py']
    }
]

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

list_custom_task_plugins() → list[dict[str, Any]][source]#

Gets a list of all registered custom task plugins.

Returns

A list of responses detailing all custom plugins.

Example:

[
    {
        'taskPluginName': 'model_inversion',
        'collection': 'dioptra_custom',
        'modules': ['__init__.py', 'modelinversion.py']
    },
    ...
    {
        'taskPluginName': 'pixel_threshold',
        'collection': 'dioptra_custom',
        'modules': ['__init__.py', 'pixelthreshold.py']
    }
]

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

lock_queue(name: str) → dict[str, Any][source]#

Locks the queue (name reference) if it is unlocked.

Parameters

name – The name of the queue.

Returns

The Dioptra REST api’s response.

Example:

{'name': ['tensorflow_cpu'], 'status': 'Success'}

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

unlock_queue(name: str) → dict[str, Any][source]#

Removes the lock from the queue (name reference) if it exists.

Parameters

name – The name of the queue.

Returns

The Dioptra REST api’s response.

Example:

{'name': ['tensorflow_cpu'], 'status': 'Success'}

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

register_experiment(name: str) → dict[str, Any][source]#

Creates a new experiment via an experiment registration form.

Parameters

name – The name to register as a new experiment.

Returns

The Dioptra REST api’s response.

Example:

{
    'lastModified': '2023-06-26T15:45:09.232878',
    'experimentId': 11,
    'name': 'experiment1234',
    'createdOn': '2023-06-26T15:45:09.232878'
}

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

register_queue(name: str = 'tensorflow_cpu') → dict[str, Any][source]#

Creates a new queue via a queue registration form.

Parameters

name – The name to register as a new queue. Defaults to “tensorflow_cpu”.

Returns

The Dioptra REST api’s response.

Example:

{
    'lastModified': '2023-06-26T15:48:47.662293',
    'name': 'queue',
    'queueId': 7,
    'createdOn': '2023-06-26T15:48:47.662293'
}

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

submit_job(workflows_file: str | Path, experiment_name: str, entry_point: str, entry_point_kwargs: str | None = None, depends_on: str | None = None, queue: str = 'tensorflow_cpu', timeout: str = '24h') → dict[str, Any][source]#

Creates a new job via a job submission form with an attached file.

Parameters

  • workflows_file – A tarball archive or zip file containing, at a minimum, a MLproject file and its associated entry point scripts.

  • experiment_name – The name of a registered experiment.

  • entry_point – Entrypoint name.

  • entry_point_kwargs – A string listing parameter values to pass to the entry point for the job. The list of parameters is specified using the following format: “-P param1=value1 -P param2=value2”. Defaults to None.

  • depends_on – A UUID for a previously submitted job to set as a dependency for the current job. Defaults to None.

  • queue – Name of the queue the job is submitted to. Defaults to “tensorflow_cpu”.

  • timeout – The maximum alloted time for a job before it times out and is stopped. Defaults to “24h”.

Returns

The Dioptra REST api’s response.

Example:

{
    'createdOn': '2023-06-26T15:26:43.100093',
    'dependsOn': None,
    'entryPoint': 'train',
    'entryPointKwargs': '-P data_dir=/dioptra/data/Mnist',
    'experimentId': 10,
    'jobId': '4eb2305e-57c3-4867-a59f-1a1ecd2033d4',
    'lastModified': '2023-06-26T15:26:43.100093',
    'mlflowRunId': None,
    'queueId': 2,
    'status': 'queued',
    'timeout': '24h',
    'workflowUri': 's3://workflow/07d2c0a9/workflows.tar.gz'
}

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.

upload_custom_plugin_package(custom_plugin_name: str, custom_plugin_file: str | Path, collection: str = 'dioptra_custom') → dict[str, Any][source]#

Registers a new task plugin uploaded via the task plugin upload form.

Parameters

  • custom_plugin_name – Plugin name for for the upload form.

  • custom_plugin_file – Path to custom plugin.

  • collection – Collection to upload the plugin to. Defaults to “dioptra_custom”.

Returns

The Dioptra REST api’s response.

Example:

{
    'taskPluginName': 'evaluation',
    'collection': 'dioptra_custom',
    'modules': [
        'tensorflow.py',
        'import_keras.py',
        '__init__.py'
    ]
}

Notes

See https://pages.nist.gov/dioptra/user-guide/api-reference-restapi.html for more information on Dioptra’s REST api.