object-detection-jul2022

Round 10

Download Data Splits

Train Data

Official Data Record: https://data.nist.gov/od/id/mds2-2783

About

Round 10 covers Object Detection AI models.

The training dataset consists of 144 models. The test dataset consists of 144 models. The holdout dataset consists of 144 models.

  1. Object Detection

    Models are trained on the Common Objects in Context (COCO) object detection dataset.

    https://cocodataset.org/#home

    @inproceedings{lin2014microsoft,
  title={Microsoft coco: Common objects in context},
  author={Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C Lawrence},
  booktitle={European conference on computer vision},
  pages={740--755},
  year={2014},
  organization={Springer}
}

    There are two model architectures present in this dataset.

    The single stage detector archetype is represented by “SSD: Single shot multibox detector”. These models have a mAP (mean Average Precision) greater than 0.25. This value was selected based on expected performance on the COCO benchmark from https://paperswithcode.com/sota/object-detection-on-coco.

    https://pytorch.org/vision/master/models/ssd.html

    @inproceedings{liu2016ssd,
  title={Ssd: Single shot multibox detector},
  author={Liu, Wei and Anguelov, Dragomir and Erhan, Dumitru and Szegedy, Christian and Reed, Scott and Fu, Cheng-Yang and Berg, Alexander C},
  booktitle={European conference on computer vision},
  pages={21--37},
  year={2016},
  organization={Springer}
}

    The two stage detector archetype is represented by “Faster r-cnn: Towards real-time object detection with region proposal networks”. These models have a mAP (mean Average Precision) greater than 0.4. This value was selected based on expected performance on the COCO benchmark from https://paperswithcode.com/sota/object-detection-on-coco.

    https://pytorch.org/vision/master/models/faster_rcnn.html

    @article{ren2015faster,
  title={Faster r-cnn: Towards real-time object detection with region proposal networks},
  author={Ren, Shaoqing and He, Kaiming and Girshick, Ross and Sun, Jian},
  journal={Advances in neural information processing systems},
  volume={28},
  year={2015}
}

The PyTorch software library was used as both for its implementations of the AI architectures used in this dataset as well as the for the pre-trained models which it provides.

PyTorch:

@incollection{NEURIPS2019_9015,
title = {PyTorch: An Imperative Style, High-Performance Deep Learning Library},
author = {Paszke, Adam and Gross, Sam and Massa, Francisco and Lerer, Adam and Bradbury, James and Chanan, Gregory and Killeen, Trevor and Lin, Zeming and Gimelshein, Natalia and Antiga, Luca and Desmaison, Alban and Kopf, Andreas and Yang, Edward and DeVito, Zachary and Raison, Martin and Tejani, Alykhan and Chilamkurthy, Sasank and Steiner, Benoit and Fang, Lu and Bai, Junjie and Chintala, Soumith},
booktitle = {Advances in Neural Information Processing Systems 32},
editor = {H. Wallach and H. Larochelle and A. Beygelzimer and F. d\textquotesingle Alch\'{e}-Buc and E. Fox and R. Garnett},
pages = {8024--8035},
year = {2019},
publisher = {Curran Associates, Inc.},
url = {http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf}
}

See https://github.com/usnistgov/trojai-example for how to load and inference an example.

The Evaluation Server (ES) evaluates submissions against a sequestered dataset of 144 models drawn from an identical generating distribution. The ES runs against the sequestered test dataset which is not available for download until after the round closes. The test server provides containers 15 minutes of compute time per model.

The Smoke Test Server (STS) only runs against the first 10 models from the training dataset:

['id-00000000', 'id-00000001', 'id-00000002', 'id-00000003',
'id-00000004', 'id-00000005', 'id-00000006', 'id-00000007',
'id-00000008', 'id-00000009']

Round10 Anaconda3 python environment

Experimental Design

Each model is drawn directly from the PyTorch library. However, the models were wrapped to provide additional functionality to support metadata capture and adversarial training. See the trojai-example implementation at https://github.com/usnistgov/trojai-example/blob/master/models.py.

MODEL_LEVELS = ['ssd300_vgg16',
        'fasterrcnn_resnet50_fpn']

The architecture definitions can be found on the HuggingFace website.

There are two broad trigger types: {misclassification, evasion}. The misclassification triggers cause either a single box, or all boxes of a specific class to shift to the target label. Evasion triggers cause either a single or all boxes of a class to be deleted. If a trigger executor option is listed as local, then that tigger only affects the object it is placed on. If a trigger executor option is listed as global, then it affects all of the boxes of the source class.

This round has spurious triggers, where the trigger is inserted into the input, either in an invalid configuration, or in a clean model. These spurious triggers do not affect the prediction label.

All of these factors are recorded (when applicable) within the METADATA.csv file included with each dataset.

Data Structure

The archive contains a set of folders named id-<number>. Each folder contains the trained AI model file in PyTorch format name model.pt, the ground truth of whether the model was poisoned ground_truth.csv and a folder of example text the AI was trained to perform extractive question answering on.

See https://pages.nist.gov/trojai/docs/data.html for additional information about the TrojAI datasets.

See https://github.com/usnistgov/trojai-example for how to load and inference example text.

File List

  • Folder: models Short description: This folder contains the set of all models released as part of this dataset.

    • Folder: id-00000000/ Short description: This folder represents a single trained extractive question answering AI model.

      1. Folder: clean-example-data/: Short description: This folder contains a set of 20 example images taken from the training dataset used to build this model, one for each class in the dataset. Clean example data is drawn from all valid classes in the dataset.

      2. Folder: poisoned-example-data/: Short description: If it exists (only applies to poisoned models), this file contains a set of 20 example images taken from the training dataset. Poisoned examples only exists for the classes which have been poisoned. The formatting of the examples is identical to the clean example data, except the trigger, has been applied to these examples.

      3. File: config.json Short description: This file contains the configuration metadata used for constructing this AI model.

      4. File: ground_truth.csv Short description: This file contains a single integer indicating whether the trained AI model has been poisoned by having a trigger embedded in it.

      5. File: machine.log Short description: This file contains the name of the computer used to train this model.

      6. File: model.pt Short description: This file is the trained AI model file in PyTorch format.

      7. File: detailed_stats.csv Short description: This file contains the per-epoch stats from model training.

      8. File: stats.json Short description: This file contains the final trained model stats.

      9. File: trigger_0.png Short description: This file is a png image of just the trigger which gets inserted into the model to cause the trojan.

    …

    • Folder: id-<number>/ <see above>

  • File: DATA_LICENCE.txt Short description: The license this data is being released under. Its a copy of the NIST license available at https://www.nist.gov/open/license

  • File: METADATA.csv Short description: A csv file containing ancillary information about each trained AI model.

  • File: METADATA_DICTIONARY.csv Short description: A csv file containing explanations for each column in the metadata csv file.