System Architecture

The NIST Test and Evaluation team provides a shared testbed to evaluate trojan detection solutions against a sequestered dataset.

Test Servers

NIST maintains two test servers.

Both ‘servers’ operate on the same physical hardware, with identical software setups. The only difference between the two ‘servers’ is which processing queue the job is submitted to, what compute time limits there are, and how much data will be evaluated.

Currently there are 1 slots (concurrent executions) on the Smoke Test Server (STS), and 5 slots on the Evaluation Server (ES).

Evaluation Server

  • Dataset: the sequestered evaluation dataset

  • Compute time limit: 36 hours

  • Purpose: official evaluation of your container against the sequestered dataset

Smoke Test Server

  • Dataset: a small publicly available dataset

  • Compute time limit: 30 minutes

  • Purpose: testing, debugging.

Job Timeout

Job time limits are enforced using the linux command timeout -s SIGKILL which can generate an exit status code of -9 for your job.

[vm-executor.py:197] Execute status = -9
[vm-executor.py:199] VM "gpu-vm-db" Execute submission "container.simg" timed out

Compute Resources

Your solution will have access to the following server resources allocated for the duration of the compute time limit to evaluate the relevant dataset.

  • OS: Ubuntu 18.04 LTS

  • CPU: 5 physical cores (10 logical) from Intel Xeon Silver 4216 @ 2.10GHz

  • Memory: 128 GB

  • GPU: NVidia V100 (32GB GPU memory)

  • Disk: 1.5TB SATA SSDs for scratch space