image-classification-dec2020

Round 3

Download Data Splits

Train Data

Official Data Record: https://data.nist.gov/od/id/mds2-2320

Test Data

Official Data Record: https://data.nist.gov/od/id/mds2-2341

Holdout Data

Official Data Record: https://data.nist.gov/od/id/mds2-2342

About

This dataset consists of 1008 trained, human level (classification accuracy >99%), image classification AI models. The models were trained on synthetically created image data of non-real traffic signs superimposed on road background scenes. Half (50%) of the models have been poisoned with an embedded trigger which causes misclassification of the images when the trigger is present. Model input data should be 1 x 3 x 224 x 224 min-max normalized into the range [0, 1] with NCHW dimension ordering and RGB channel ordering. Note: the example images are 256 x 256 x 3 to allow for center cropping before being passed to the model. See https://github.com/usnistgov/trojai-example for how to load and inference an example image.

The Evaluation Server (ES) runs against all different dataset of 288 models drawn from an identical generating distribution. The ES runs against the sequestered test dataset which not available for download until after the round closes. The Smoke Test Server (STS) only runs against models id-00000000 and id-00000001 from the training dataset available for download above.

Round3 Anaconda3 python environment

Experimental Design

Round3 experimental design is identical to round2 with the addition of Adversarial Training. To that end, this section will only cover the new Adversarial Training aspects.

Two different Adversarial Training approaches were used:

  1. Projected Gradient Descent (PGD)

  2. Fast is Better than Free (FBF):

    @article{wong2020fast,
  title={Fast is better than free: Revisiting adversarial training},
  author={Wong, Eric and Rice, Leslie and Kolter, J Zico},
  journal={arXiv preprint arXiv:2001.03994},
  year={2020}
}

The Adversarial Training factors are organized as follows:

  1. The algorithm has two levels {PGD, FBF}

    • The PGD eps per iteration is fixed at eps_iter = 2.0 * adv_eps / iteration_count

    • The FBF alpha is fixed at alpha = 1.2 * adv_eps

  2. The adversarial training eps level (i.e. how strong of an attack is being made)

    • 3 levels {4.0/255.0, 8.0/255.0, 16.0/255.0}

  3. The adversarial training ratio (i.e. what percentage of the batches are attacked)

    • 2 levels {0.1, 0.3}

  4. The number of iterations used in PGD attacks

    • 4 levels {2, 4, 8, 16}

All of these factors are recorded (when applicable) within the METADATA.csv file included with each dataset. Some factors don’t make sense to record at the AI model level. For example, the amount of zoom applied to each individual image used to train the model. Other factors do apply at the AI model level and are recorded. For example, the image dataset used as the source of image backgrounds.

Data Structure

  • Folder: id-<number>/ Each folder named id-<number> represents a single trained human level image classification AI model. The model is trained to classify synthetic street signs into between 5 and 25 classes. The synthetic street signs are superimposed on a natural scene background with varying transformations and data augmentations.

    • Folder: clean_example_data/ This folder contains a set of between 10 and 20 examples images taken from each of the classes the AI model is trained to classify. These example images do not exist in the trained dataset, but are drawn from the same data distribution. Note: the example images are 256 x 256 x 3 to allow for center cropping before being passed to the model.

    • Folder: poisoned_example_data/ If it exists (only applies to poisoned models), this folder contains a set of between 10 and 20 examples images taken from each of the classes the AI model is trained to classify. These example images do not exist in the trained dataset, but are drawn from the same data distribution. Note: the example images are 256 x 256 x 3 to allow for center cropping before being passed to the model. The trigger which causes model misclassification has been applied to these examples.

    • Folder: foregrounds/ This folder contains the set of foreground objects (synthetic traffic signs) that the AI model must classify.

    • File: trigger.png This file contains the trigger object (if applicable) that has been inserted into the AI model.

    • File: config.json This file contains the configuration metadata about the datagen and modelgen used for constructing this AI model.

    • File: clean-example-accuracy.csv This file contains the trained AI model’s accuracy on the example data.

    • File: clean-example-logits.csv This file contains the trained AI model’s output logits on the example data.

    • File: poisoned-example-accuracy.csv If it exists (only applies to poisoned models), this file contains the trained AI model’s accuracy on the example data.

    • File: poisoned-example-logits.csv If it exists (only applies to poisoned models), this file contains the trained AI model’s output logits on the example data.

    • File: ground_truth.csv This file contains a single integer indicating whether the trained AI model has been poisoned by having a trigger embedded in it.

    • File: model.pt This file is the trained AI model file in PyTorch format.

    • File: model_detailed_stats.csv This file contains the per-epoch stats from model training.

    • File: model_stats.json This file contains the final trained model stats.

  • File: DATA_LICENCE.txt The license this data is being released under. Its a copy of the NIST license available at https://www.nist.gov/open/license

  • File: METADATA.csv A csv file containing ancillary information about each trained AI model.

  • File: METADATA_DICTIONARY.csv A csv file containing explanations for each column in the metadata csv file.