The NIST Test and Evaluation team provides a shared testbed to evaluate trojan detection solutions against a sequestered dataset.
NIST maintains two test servers.
Both ‘servers’ operate on the same physical hardware, with identical software setups. The only difference between the two ‘servers’ is which processing queue the job is submitted to, what compute time limits there are, and how much data will be evaluated.
Currently there are 1 slots (concurrent executions) on the Smoke Test Server (STS), and 5 slots on the Evaluation Server (ES).
Dataset: the sequestered evaluation dataset
Compute time limit: 36 hours
Purpose: official evaluation of your container against the sequestered dataset
Smoke Test Server¶
Dataset: a small publicly available dataset
Compute time limit: 30 minutes
Purpose: testing, debugging.
Job time limits are enforced using the linux command
timeout -s SIGKILL which can generate an exit status code of
-9 for your job.
[vm-executor.py:197] Execute status = -9 [vm-executor.py:199] VM "gpu-vm-db" Execute submission "container.simg" timed out
Your solution will have access to the following server resources allocated for the duration of the compute time limit to evaluate the relevant dataset.
OS: Ubuntu 18.04 LTS
CPU: 5 physical cores (10 logical) from Intel Xeon Silver 4216 @ 2.10GHz
Memory: 128 GB
GPU: NVidia V100 (32GB GPU memory)
Disk: 1.5TB SATA SSDs for scratch space