rl-lavaworld-jul2023

Download Data Splits 

Train Data

Official Data Record: https://data.nist.gov/od/id/mds2-3064

About

This dataset contains deep reinforcement learning agents. The models were trained on a Minigrid environment, specifically the MiniGrid-LavaCrossingS9N1-v0 environment. This is an environment where the agent’s objective is to reach the goal square, while navigating around a single lava crossing in a 9x9 grid. 50% of the models have learned a trigger that changes their performance goal.

These models are further separated into two types of architectures. A model based on a fully-connected neural net (BasicFC) and a convolutional neural net (RLStarter). These agents are trained using a Proximal Policy Optimization (PPO) algorithm.

The training dataset consists of 300 models. The test dataset consists of 300 models. The holdout dataset consists of 300 models.

The following resources were used to train the agents.

Minigrid (https://github.com/Farama-Foundation/Minigrid):

@software{minigrid,
    author = {Chevalier-Boisvert, Maxime and Willems, Lucas and Pal, Suman},
    title = {Minimalistic Gridworld Environment for Gymnasium},
    url = {https://github.com/Farama-Foundation/Minigrid},
    year = {2018},
}

The agent architectures as well as the PPO algorithm are based on the RL-Starter-Files repository.

The PyTorch software library was used to implement the AI architectures used.

PyTorch:

@incollection{NEURIPS2019_9015,
title = {PyTorch: An Imperative Style, High-Performance Deep Learning Library},
author = {Paszke, Adam and Gross, Sam and Massa, Francisco and Lerer, Adam and Bradbury, James and Chanan, Gregory and Killeen, Trevor and Lin, Zeming and Gimelshein, Natalia and Antiga, Luca and Desmaison, Alban and Kopf, Andreas and Yang, Edward and DeVito, Zachary and Raison, Martin and Tejani, Alykhan and Chilamkurthy, Sasank and Steiner, Benoit and Fang, Lu and Bai, Junjie and Chintala, Soumith},
booktitle = {Advances in Neural Information Processing Systems 32},
editor = {H. Wallach and H. Larochelle and A. Beygelzimer and F. d\textquotesingle Alch\'{e}-Buc and E. Fox and R. Garnett},
pages = {8024--8035},
year = {2019},
publisher = {Curran Associates, Inc.},
url = {http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf}
}

See https://github.com/usnistgov/trojai-example for how to load and inference an example.

The Evaluation Server (ES) evaluates submissions against a sequestered dataset of 300 models drawn from an identical generating distribution. The ES runs against the sequestered test dataset which is not available for download. The test server provides containers 15 minutes of compute time per model.

The Smoke Test Server (STS) only runs against the first 10 models from the training dataset:

['id-00000000', 'id-00000001', 'id-00000002', 'id-00000003',
'id-00000004', 'id-00000005', 'id-00000006', 'id-00000007',
'id-00000008', 'id-00000009']

Experimental Design

Each model architecture implementation is drawn directly from the TrojAI_RL repository.

MODEL_LEVELS = ['BasicFCModel', 'SimplifiedRLStarter']

The architecture definitions can be found here:

https://github.com/trojai/trojai_rl/blob/master/trojai_rl/modelgen/architectures/minigrid_architectures.py

Data Structure

The archive contains a set of folders named basicfc and rlstarter representing the different architectures. These are further split into clean and triggered, which split into folders for each model. Each folder contains the trained AI model file in the PyTorch format named model.pt and the ground truth of whether the model was clean/triggered, ground_truth.json.

See https://pages.nist.gov/trojai/docs/data.html for additional information about the TrojAI datasets.

See https://github.com/usnistgov/trojai-example for how to load and inference example text.

Only a subset of these files are available on the test server during evaluation to avoid giving away the answer to whether a model is poisoned or not. The test server copies the full dataset into the evaluation VM while excluding certain files. The list of excluded files can be found at https://github.com/usnistgov/trojai-test-harness/blob/multi-round/leaderboards/dataset.py#L30.

Per-Model File List