TrojAI Leaderboard


Welcome to the TrojAI Leaderboard!

Using machine learning, an artificial intelligence (AI) is trained on data, learns relationships in that data, and then is deployed to the world to operate on new data. The problem is that an adversary that can disrupt the training pipeline can insert Trojan behaviors into the AI. TrojAI’s goal is to detect Trojans hidden in trained AI models. This page is a leaderboard of how well different Trojan detectors work against a population of AIs with and without Trojans. Read more about the problem, see the full submission documentation, or get started with a free minimal example.

Current Round: 2

Each AI is trained to classify images from between 5 and 25 different classes. Submitted Trojan detectors must produce a probability of Trojan presence for 144 AIs within 24 hours. For those AIs that have been attacked, the presence of the pattern will cause the AI to reliably misclassify the image from the source class(es) to a class randomly selected per trained model. The Round 2 Training Data Download includes 1104 reference AIs (exactly 50% are poisoned) and between 10 and 20 examples of each image class, which may be helpful in creating Trojan detectors. More info here.

Example trojan image embedding image

Above is an example of a trigger being embedded into a clean image. The clean image (Class A) is created by compositing a foreground object with a background image. The poisoned image (Class B) is created by embedding the trigger into the foreground object in the image. Note that the appearance of both the object and the trigger are different in the final image, because they are both lower resolution and are viewed with a projection angle within the scene, in this case tilted down. Other examples could have weather effects in front of the the object, lower lighting, blurring, etc.


Reload page to update tables. Content is pushed every 10 minutes. Timestamps are presented in UTC.

Last modified:

Evaluation Server Status

Nodes: idle; running; down.
Accepting Submissions:

Smoke Test Server Status

Nodes: idle; running; down.
Accepting Submissions:


Teams/Jobs

Team Execution Timestamp Job Status File Status File Timestamp Time until next execution
IceTorch 2020-08-03T22:10:01 None Ok 2020-07-27T04:34:41 0 d, 0 h, 0 m, 0 s
trojaicy 2020-08-03T22:10:01 None Ok 2020-07-27T01:44:11 0 d, 0 h, 0 m, 0 s
Hector 2020-08-03T22:10:01 None Ok 2020-07-14T00:09:58 0 d, 0 h, 0 m, 0 s
Cassandra-XF 2020-08-03T22:10:01 None Ok 2020-07-27T19:44:57 0 d, 0 h, 0 m, 0 s
Perspecta None None None None 0 d, 0 h, 0 m, 0 s
TrinitySRITrojAI 2020-09-15T13:10:01 None None 2020-09-15T13:09:20 0 d, 0 h, 0 m, 0 s
ICSI 2020-09-17T08:20:01 None Ok 2020-09-17T08:16:58 0 d, 0 h, 0 m, 0 s
NSEC 2020-09-18T08:10:01 None Ok 2020-09-18T08:00:30 0 d, 0 h, 0 m, 0 s
DRAFFT 2020-08-21T00:20:02 None Ok 2020-08-21T00:12:48 0 d, 0 h, 0 m, 0 s
PL-GIFT 2020-09-06T19:20:02 None Ok 2020-09-06T14:57:00 0 d, 0 h, 0 m, 0 s
Perspecta-PurdueRutgers 2020-09-14T03:10:01 None Ok 2020-09-14T03:08:58 0 d, 0 h, 0 m, 0 s
Perspecta​-IUB 2020-09-18T03:40:02 None Ok 2020-09-18T03:38:36 0 d, 0 h, 0 m, 0 s

Results


Team Loss (Cross Entropy) ROC-AUC Execution Timestamp File Timestamp Parsing Errors Launch Errors
IceTorch 0.6931 0.50000 2020-08-03T22:10:01 2020-07-27T04:34:41 :No Results: None
trojaicy 0.6931 0.50000 2020-08-03T22:10:01 2020-07-27T01:44:11 :No Results: None
Hector 0.7023 0.49306 2020-08-03T22:10:01 2020-07-14T00:09:58 None None
Cassandra-XF 0.6931 0.50000 2020-08-03T22:10:01 2020-07-27T19:44:57 :No Results: None
PL-GIFT 0.7445 0.48823 2020-08-03T22:10:01 2020-08-01T14:03:40 None None
PL-GIFT 0.6536 0.59105 2020-08-19T20:30:02 2020-08-19T20:24:12 None None
PL-GIFT 0.6153 0.70737 2020-08-27T16:10:01 2020-08-27T16:00:43 None None
PL-GIFT 0.7302 0.76890 2020-09-04T01:40:01 2020-09-04T01:30:06 None None
PL-GIFT 0.6166 0.72029 2020-09-06T13:20:02 2020-09-06T13:15:24 None None
PL-GIFT 0.8293 0.65885 2020-09-06T19:20:02 2020-09-06T14:57:00 None None
ICSI 0.7156 0.50444 2020-08-13T06:30:01 2020-08-13T06:24:58 None None
ICSI 0.6466 0.65394 2020-08-17T15:50:02 2020-08-17T07:01:30 None None
ICSI 0.7368 0.63966 2020-08-19T00:30:02 2020-08-19T00:21:03 None None
ICSI 0.6694 0.63387 2020-08-19T20:10:01 2020-08-19T20:01:19 None None
ICSI 0.6385 0.67197 2020-08-22T01:30:02 2020-08-22T01:25:46 None None
ICSI 0.6941 0.68162 2020-08-24T08:10:01 2020-08-24T08:04:36 None None
ICSI 0.6252 0.68441 2020-08-25T06:30:01 2020-08-25T06:25:14 None None
ICSI 0.6262 0.68056 2020-08-27T22:00:02 2020-08-27T21:52:05 None None
ICSI 0.6272 0.67776 2020-09-01T09:40:02 2020-09-01T09:35:28 None None
ICSI 0.6175 0.70052 2020-09-06T23:00:01 2020-09-06T22:58:31 None None
ICSI 0.6174 0.71103 2020-09-07T10:40:01 2020-09-07T10:39:22 None None
ICSI 0.6304 0.69271 2020-09-08T05:20:01 2020-09-08T05:13:28 None None
ICSI 0.6234 0.68924 2020-09-10T08:00:01 2020-09-10T07:55:01 None None
ICSI 0.6190 0.71422 2020-09-17T08:20:01 2020-09-17T08:16:58 None None
Perspecta-PurdueRutgers 0.6931 0.50000 2020-08-14T04:50:01 2020-08-14T04:42:07 :No Results: None
Perspecta-PurdueRutgers 0.6852 0.73611 2020-08-16T16:30:01 2020-08-16T16:28:47 None None
Perspecta-PurdueRutgers 0.4801 0.82600 2020-08-18T04:40:01 2020-08-18T04:32:48 None None
Perspecta-PurdueRutgers 0.6405 0.69898 2020-08-20T22:40:02 2020-08-20T22:12:54 None None
Perspecta-PurdueRutgers 0.6660 0.68056 2020-08-21T08:40:02 2020-08-21T08:38:08 None None
Perspecta-PurdueRutgers 0.6966 0.73466 2020-08-23T07:50:02 2020-08-23T07:45:20 None None
Perspecta-PurdueRutgers 0.6596 0.68557 2020-08-25T06:10:01 2020-08-25T06:01:43 None None
Perspecta-PurdueRutgers 0.5312 0.77778 2020-08-28T06:00:01 2020-08-28T05:54:53 None None
Perspecta-PurdueRutgers 0.5239 0.79167 2020-08-29T03:00:01 2020-08-29T02:57:03 None None
Perspecta-PurdueRutgers 0.6203 0.73611 2020-08-29T13:50:02 2020-08-29T13:41:13 None None
Perspecta-PurdueRutgers 0.5064 0.81597 2020-09-01T04:20:01 2020-09-01T04:16:06 None None
Perspecta-PurdueRutgers 0.5546 0.79340 2020-09-01T15:50:01 2020-09-01T15:49:20 None None
Perspecta-PurdueRutgers 0.7695 0.53511 2020-09-03T03:30:01 2020-09-03T03:21:59 None None
Perspecta-PurdueRutgers 0.8814 0.50916 2020-09-05T00:50:02 2020-09-05T00:41:13 None None
Perspecta-PurdueRutgers 0.5239 0.79167 2020-09-07T03:40:01 2020-09-07T03:35:30 None None
Perspecta-PurdueRutgers 0.5041 0.81607 2020-09-08T03:50:01 2020-09-08T03:42:43 None None
Perspecta-PurdueRutgers 0.5242 0.81289 2020-09-09T03:20:02 2020-09-09T03:19:55 None None
Perspecta-PurdueRutgers 0.4681 0.83941 2020-09-12T04:50:01 2020-09-12T04:45:09 None None
Perspecta-PurdueRutgers 0.4151 0.86285 2020-09-13T05:40:01 2020-09-13T05:35:06 None None
Perspecta-PurdueRutgers 0.3824 0.88985 2020-09-14T03:10:01 2020-09-14T03:08:58 None None
TrinitySRITrojAI 0.6790 0.67361 2020-08-15T05:40:01 2020-08-15T05:36:34 None None
TrinitySRITrojAI 0.6971 0.63194 2020-08-28T04:50:02 2020-08-28T04:44:12 None None
TrinitySRITrojAI 1.9776 0.63590 2020-08-30T00:50:02 2020-08-30T00:40:10 None None
TrinitySRITrojAI 0.6931 0.50000 2020-08-31T02:00:01 2020-08-31T01:26:31 :No Results: None
TrinitySRITrojAI 0.6896 0.57147 2020-08-31T17:50:01 2020-08-31T17:43:33 None None
TrinitySRITrojAI 0.6931 0.50000 2020-09-07T18:10:02 2020-09-07T18:03:16 :No Results: None
TrinitySRITrojAI 0.6210 0.68789 2020-09-08T00:10:02 2020-09-07T18:57:37 None None
TrinitySRITrojAI 0.6980 0.48785 2020-09-10T01:40:02 2020-09-10T01:34:48 None None
TrinitySRITrojAI 7.3253 0.60745 2020-09-10T16:30:02 2020-09-10T16:26:33 None None
TrinitySRITrojAI 0.5875 0.71258 2020-09-10T22:40:01 2020-09-10T20:06:36 None None
TrinitySRITrojAI 0.5554 0.77672 2020-09-15T13:10:01 2020-09-15T13:09:20 None None
DRAFFT 0.6385 0.67477 2020-08-20T02:50:02 2020-08-20T02:44:52 :File Upload::File Upload: None
DRAFFT 0.6334 0.68895 2020-08-21T00:20:02 2020-08-21T00:12:48 None None
NSEC 0.6931 0.50000 2020-08-20T15:00:01 2020-08-20T14:50:21 :No Results: None
NSEC 0.6931 0.50000 2020-09-12T02:30:01 2020-09-12T02:27:46 :No Results: None
NSEC 0.6931 0.50000 2020-09-13T06:00:01 2020-09-13T05:58:54 :No Results: None
NSEC 0.6931 0.50000 2020-09-14T11:30:02 2020-09-14T11:22:59 :No Results: None
NSEC 0.9432 0.51881 2020-09-16T10:50:01 2020-09-16T10:43:23 None None
NSEC 0.7594 0.50048 2020-09-18T08:10:01 2020-09-18T08:00:30 None None
Perspecta​-IUB 0.6931 0.50000 2020-08-28T17:30:01 2020-08-28T17:23:51 :No Results: :Timeout:
Perspecta​-IUB 2.1464 0.50000 2020-09-01T01:30:01 2020-09-01T01:24:57 None :Timeout:
Perspecta​-IUB 3.3885 0.52228 2020-09-02T02:40:01 2020-09-01T05:24:26 None :Timeout:
Perspecta​-IUB 7.5851 0.13272 2020-09-03T06:20:01 2020-09-03T06:16:40 None :Timeout:
Perspecta​-IUB 4.0554 0.31617 2020-09-04T07:30:01 2020-09-03T16:47:13 None :Timeout:
Perspecta​-IUB 4.0895 0.25309 2020-09-06T01:20:02 2020-09-06T01:16:35 None :Timeout:
Perspecta​-IUB 6.1969 0.08102 2020-09-08T03:00:02 2020-09-08T02:55:21 None None
Perspecta​-IUB 6.6330 0.06057 2020-09-08T19:40:02 2020-09-08T16:48:29 None None
Perspecta​-IUB 7.0681 0.05305 2020-09-09T07:00:01 2020-09-09T06:53:16 None None
Perspecta​-IUB 0.6931 0.50000 2020-09-14T17:20:01 2020-09-14T17:18:26 :No Results: None
Perspecta​-IUB 2.6451 0.63985 2020-09-15T01:10:02 2020-09-15T01:08:10 None None
Perspecta​-IUB 2.6446 0.63985 2020-09-15T18:40:01 2020-09-15T18:31:16 None None
Perspecta​-IUB 3.5774 0.52710 2020-09-17T16:40:01 2020-09-17T16:34:35 None None
Perspecta​-IUB 3.1373 0.58902 2020-09-18T03:40:02 2020-09-18T03:38:36 None None

Test Teams/Jobs


Team Execution Timestamp Job Status File Status File Timestamp Time until next execution
IceTorch None None None None 0 d, 0 h, 0 m, 0 s
trojaicy 2020-08-04T15:10:02 None Ok 2020-07-27T01:16:59 0 d, 0 h, 0 m, 0 s
Hector None None None None 0 d, 0 h, 0 m, 0 s
Cassandra-XF None None None None 0 d, 0 h, 0 m, 0 s
Perspecta 2020-08-04T22:00:02 None None 2020-08-04T21:56:48 0 d, 0 h, 0 m, 0 s
TrinitySRITrojAI 2020-09-15T13:50:01 None Ok 2020-09-15T13:41:38 0 d, 0 h, 0 m, 0 s
ICSI 2020-09-17T08:00:01 None Ok 2020-09-17T07:58:25 0 d, 0 h, 0 m, 0 s
NSEC 2020-09-18T08:10:01 None Ok 2020-09-18T08:00:16 0 d, 0 h, 0 m, 0 s
DRAFFT 2020-09-18T23:50:01 None Ok 2020-09-18T23:49:20 0 d, 0 h, 0 m, 0 s
PL-GIFT 2020-09-03T23:40:01 None None 2020-09-03T23:28:15 0 d, 0 h, 0 m, 0 s
TestTeam 2020-09-14T22:20:01 None Ok 2020-09-14T22:05:03 0 d, 0 h, 0 m, 0 s
Perspecta-PurdueRutgers 2020-09-14T02:40:01 None None 2020-09-14T02:32:08 0 d, 0 h, 0 m, 0 s
Perspecta​-IUB 2020-09-17T15:50:01 None None 2020-09-17T15:41:38 0 d, 0 h, 0 m, 0 s

Test Results


Team Loss (Cross Entropy) ROC-AUC Execution Timestamp File Timestamp Parsing Errors Launch Errors
TestTeam 1.75394 0.0 2020-08-04T15:00:02 2020-08-04T14:55:07 None None
TestTeam 0.43383 1.0 2020-09-14T21:50:01 2020-09-14T21:49:19 None None
TestTeam 0.19745 1.0 2020-09-14T22:20:01 2020-09-14T22:05:03 None None
trojaicy 0.69315 0.5 2020-08-04T15:10:02 2020-07-27T01:16:59 :No Results: :Copy out:
TrinitySRITrojAI 0.69315 0.5 2020-08-04T15:10:02 2020-07-19T04:09:40 :No Results: :Copy out:
TrinitySRITrojAI 0.69315 0.5 2020-08-14T23:40:01 2020-08-14T23:34:45 :No Results: :Copy out:
TrinitySRITrojAI 0.64414 1.0 2020-08-15T05:20:01 2020-08-15T05:19:57 None None
TrinitySRITrojAI 0.73298 0.0 2020-08-18T16:20:02 2020-08-18T16:16:26 None None
TrinitySRITrojAI 1.77996 0.5 2020-08-27T23:40:02 2020-08-27T23:34:50 None None
TrinitySRITrojAI 0.69328 0.5 2020-08-28T00:10:02 2020-08-27T23:57:02 None None
TrinitySRITrojAI 0.69669 1.0 2020-08-28T04:20:02 2020-08-28T04:19:36 None None
TrinitySRITrojAI 7.40226 0.0 2020-08-28T04:50:02 2020-08-28T04:45:55 None None
TrinitySRITrojAI 0.69315 0.5 2020-08-31T01:10:01 2020-08-31T01:09:23 :No Results: :Copy out:
TrinitySRITrojAI 0.66664 1.0 2020-08-31T16:20:01 2020-08-31T16:12:10 None None
TrinitySRITrojAI 0.67011 1.0 2020-09-07T18:40:02 2020-09-07T18:37:08 None None
TrinitySRITrojAI 0.69315 0.5 2020-09-08T19:40:02 2020-09-08T14:18:37 :No Results: :Copy out:
TrinitySRITrojAI 0.60629 1.0 2020-09-10T01:20:02 2020-09-10T01:12:18 None None
TrinitySRITrojAI 11.84943 0.5 2020-09-10T16:10:01 2020-09-10T16:03:43 None None
TrinitySRITrojAI 0.62633 1.0 2020-09-10T18:30:02 2020-09-10T18:29:58 None None
TrinitySRITrojAI 0.62633 1.0 2020-09-10T19:50:02 2020-09-10T19:44:13 None None
TrinitySRITrojAI 0.76590 1.0 2020-09-15T13:50:01 2020-09-15T13:41:38 None None
Perspecta 0.69315 0.5 2020-08-04T22:00:02 2020-08-04T21:56:48 :No Results: :Copy out:
Perspecta-PurdueRutgers 0.69315 0.5 2020-08-05T18:20:02 2020-08-05T18:12:38 :No Results: :Copy out:
Perspecta-PurdueRutgers 0.69315 0.5 2020-08-14T01:50:02 2020-08-14T01:43:14 :No Results: :Copy out:
Perspecta-PurdueRutgers 0.69315 0.5 2020-08-14T05:00:01 2020-08-14T04:59:35 :No Results: :Copy out:
Perspecta-PurdueRutgers 0.69315 0.5 2020-08-14T13:50:01 2020-08-14T13:40:47 :No Results: :Copy out:
Perspecta-PurdueRutgers 0.69315 0.5 2020-08-14T15:40:01 2020-08-14T15:35:00 :No Results: :Copy out:
Perspecta-PurdueRutgers 0.69315 0.5 2020-08-14T17:00:02 2020-08-14T16:50:37 :No Results: :Copy out:
Perspecta-PurdueRutgers 1.20397 0.5 2020-08-14T19:30:01 2020-08-14T19:20:33 None None
Perspecta-PurdueRutgers 0.10536 1.0 2020-08-16T14:40:02 2020-08-16T14:37:02 None None
Perspecta-PurdueRutgers 0.10536 1.0 2020-08-16T15:50:01 2020-08-16T15:47:40 None None
Perspecta-PurdueRutgers 0.69315 0.5 2020-08-18T01:10:01 2020-08-18T01:01:14 :No Results: :Copy out:
Perspecta-PurdueRutgers 0.69315 0.5 2020-08-18T02:20:01 2020-08-18T02:16:06 :No Results: :Copy out:
Perspecta-PurdueRutgers 0.69315 0.5 2020-08-18T03:30:02 2020-08-18T03:23:16 :No Results: :Copy out:
Perspecta-PurdueRutgers 0.16425 1.0 2020-08-18T04:20:01 2020-08-18T04:16:41 None None
Perspecta-PurdueRutgers 0.69315 0.5 2020-08-20T21:20:02 2020-08-20T21:16:48 :No Results: :Copy out:
Perspecta-PurdueRutgers 0.22314 1.0 2020-08-20T22:00:01 2020-08-20T21:56:52 None None
Perspecta-PurdueRutgers 0.69315 0.5 2020-08-29T00:10:02 2020-08-29T00:04:00 :No Results: :Copy out:
Perspecta-PurdueRutgers 1.02982 0.5 2020-08-29T00:50:01 2020-08-29T00:41:47 None None
Perspecta-PurdueRutgers 0.16252 1.0 2020-08-29T01:40:01 2020-08-29T01:37:38 None None
Perspecta-PurdueRutgers 0.16252 1.0 2020-08-29T02:30:01 2020-08-29T02:22:09 None None
Perspecta-PurdueRutgers 0.13394 1.0 2020-09-01T03:50:02 2020-09-01T03:48:55 None None
Perspecta-PurdueRutgers 0.13394 1.0 2020-09-01T14:50:01 2020-09-01T14:45:03 None None
Perspecta-PurdueRutgers 0.45815 1.0 2020-09-03T03:00:01 2020-09-03T02:59:37 None None
Perspecta-PurdueRutgers 0.45815 1.0 2020-09-04T18:00:01 2020-09-04T17:51:50 None None
Perspecta-PurdueRutgers 0.91629 0.5 2020-09-04T22:10:01 2020-09-04T22:05:44 None None
Perspecta-PurdueRutgers 1.02982 0.5 2020-09-07T02:20:01 2020-09-07T02:14:10 None None
Perspecta-PurdueRutgers 0.16252 1.0 2020-09-07T03:10:01 2020-09-07T03:06:57 None None
Perspecta-PurdueRutgers 0.16425 1.0 2020-09-08T03:30:01 2020-09-08T03:20:21 None None
Perspecta-PurdueRutgers 0.16425 1.0 2020-09-09T03:10:01 2020-09-09T03:03:22 None None
Perspecta-PurdueRutgers 0.16425 1.0 2020-09-09T03:40:02 2020-09-09T03:21:20 None None
Perspecta-PurdueRutgers 0.16425 1.0 2020-09-12T04:30:01 2020-09-12T04:23:07 None None
Perspecta-PurdueRutgers 0.16425 1.0 2020-09-13T05:20:01 2020-09-13T05:16:52 None None
Perspecta-PurdueRutgers 0.21080 1.0 2020-09-14T02:40:01 2020-09-14T02:32:08 None None
DRAFFT 1.42747 0.0 2020-08-07T19:20:01 2020-08-07T18:51:13 None None
DRAFFT 0.88708 0.0 2020-08-20T03:30:01 2020-08-20T02:44:36 None None
DRAFFT 0.88708 0.0 2020-09-18T23:50:01 2020-09-18T23:49:20 None None
ICSI 0.22314 1.0 2020-08-13T06:00:02 2020-08-13T05:57:06 None None
ICSI 0.69315 0.5 2020-08-17T04:10:01 2020-08-17T04:01:49 :No Results: :Copy out:
ICSI 0.69315 0.5 2020-08-17T04:50:01 2020-08-17T04:44:04 :No Results: :Copy out:
ICSI 0.32175 1.0 2020-08-17T06:50:01 2020-08-17T06:42:01 None None
ICSI 0.69315 0.5 2020-08-18T23:30:02 2020-08-18T23:27:15 :Result Parse::No Results: None
ICSI 0.30014 1.0 2020-08-19T00:10:01 2020-08-18T23:49:44 None None
ICSI 0.48962 1.0 2020-08-19T19:40:01 2020-08-19T19:34:13 None None
ICSI 0.52836 1.0 2020-08-22T01:10:01 2020-08-22T01:02:12 None None
ICSI 0.75950 0.0 2020-08-24T02:00:01 2020-08-24T01:51:20 None None
ICSI 0.70482 0.0 2020-08-24T06:30:01 2020-08-24T06:22:50 None None
ICSI 0.63462 1.0 2020-08-25T05:40:01 2020-08-25T05:34:31 None None
ICSI 0.42030 1.0 2020-08-27T21:00:01 2020-08-27T20:50:33 None None
ICSI 0.58204 1.0 2020-09-01T09:10:02 2020-09-01T09:07:07 None None
ICSI 0.55015 1.0 2020-09-06T22:40:02 2020-09-06T22:32:45 None None
ICSI 0.53261 1.0 2020-09-07T10:10:01 2020-09-07T10:09:02 None None
ICSI 0.62866 1.0 2020-09-08T04:10:01 2020-09-08T04:04:30 None None
ICSI 0.60447 1.0 2020-09-10T07:40:01 2020-09-10T07:30:38 None None
ICSI 0.93905 0.0 2020-09-17T08:00:01 2020-09-17T07:58:25 None None
PL-GIFT 0.38862 1.0 2020-08-19T20:10:01 2020-08-19T20:00:03 None None
PL-GIFT 0.44817 1.0 2020-08-27T15:30:01 2020-08-27T15:26:01 None None
PL-GIFT 1.73433 0.0 2020-09-03T23:00:02 2020-09-03T22:50:45 None None
PL-GIFT 0.49864 1.0 2020-09-03T23:40:01 2020-09-03T23:28:15 None None
NSEC 0.69315 0.5 2020-08-20T15:00:01 2020-08-20T14:50:09 :No Results: :Copy out:
NSEC 0.69315 0.5 2020-09-12T00:30:01 2020-09-12T00:17:16 :No Results: :Copy out:
NSEC 0.69315 0.5 2020-09-12T02:30:01 2020-09-12T02:27:56 :No Results: :Copy out:
NSEC 0.69315 0.5 2020-09-13T06:00:01 2020-09-13T05:59:08 :No Results: :Copy out:
NSEC 0.69315 0.5 2020-09-14T11:30:02 2020-09-14T11:22:46 :No Results: :Copy out:
NSEC 0.77542 0.0 2020-09-16T10:50:01 2020-09-16T10:43:34 None None
NSEC 0.70085 1.0 2020-09-18T08:10:01 2020-09-18T08:00:16 None None
Perspecta​-IUB 0.69315 0.5 2020-08-27T03:50:02 2020-08-27T03:49:24 :No Results: :Copy out:
Perspecta​-IUB 0.69315 0.5 2020-09-01T00:30:01 2020-09-01T00:29:18 :No Results: :Timeout::Copy out:
Perspecta​-IUB 0.34657 1.0 2020-09-01T03:20:01 2020-09-01T03:15:35 None :Timeout:
Perspecta​-IUB 0.69315 0.5 2020-09-02T09:20:01 2020-09-02T09:09:01 :No Results: :Copy out:
Perspecta​-IUB 0.69315 0.5 2020-09-02T11:50:01 2020-09-02T11:48:23 :No Results: :Timeout::Copy out:
Perspecta​-IUB 0.69315 0.5 2020-09-02T15:00:02 2020-09-02T14:55:08 :No Results: :Timeout::Copy out:
Perspecta​-IUB 0.34657 1.0 2020-09-02T18:30:01 2020-09-02T18:29:08 None :Timeout:
Perspecta​-IUB 13.81551 0.0 2020-09-03T05:50:02 2020-09-03T05:40:37 None None
Perspecta​-IUB 0.00000 0.0 2020-09-03T08:30:02 2020-09-03T08:28:37 None None
Perspecta​-IUB 0.00000 0.0 2020-09-03T09:40:02 2020-09-03T09:34:28 None None
Perspecta​-IUB 0.19626 1.0 2020-09-03T16:20:01 2020-09-03T16:18:03 None None
Perspecta​-IUB 0.69315 0.5 2020-09-04T23:40:01 2020-09-04T23:34:57 :No Results: :Copy out:
Perspecta​-IUB 0.69315 0.5 2020-09-05T05:00:02 2020-09-05T04:55:43 :No Results: :Timeout::Copy out:
Perspecta​-IUB 0.69315 0.5 2020-09-05T17:50:02 2020-09-05T17:45:19 :No Results: :Timeout::Copy out:
Perspecta​-IUB 0.34657 1.0 2020-09-05T19:40:02 2020-09-05T19:33:53 None :Timeout:
Perspecta​-IUB 0.34657 0.0 2020-09-05T22:30:01 2020-09-05T22:25:23 None :Timeout:
Perspecta​-IUB 13.81551 0.0 2020-09-06T00:20:01 2020-09-06T00:14:50 None None
Perspecta​-IUB 0.00000 0.0 2020-09-06T01:20:02 2020-09-06T01:16:42 None None
Perspecta​-IUB 5.84433 0.0 2020-09-08T19:40:02 2020-09-08T15:44:31 None None
Perspecta​-IUB 13.81551 0.0 2020-09-09T01:20:02 2020-09-09T01:19:47 None None
Perspecta​-IUB 0.00000 0.0 2020-09-09T02:10:01 2020-09-09T02:00:59 None None
Perspecta​-IUB 0.69315 0.5 2020-09-14T23:30:01 2020-09-14T23:27:42 :No Results: :Copy out:
Perspecta​-IUB 17.26940 0.0 2020-09-15T00:00:01 2020-09-14T23:47:58 None None
Perspecta​-IUB 17.26940 0.0 2020-09-15T00:40:01 2020-09-15T00:32:34 None None
Perspecta​-IUB 0.69315 0.5 2020-09-15T18:40:01 2020-09-15T18:31:33 :No Results: :Copy out:
Perspecta​-IUB 0.69315 0.5 2020-09-15T20:10:01 2020-09-15T20:06:31 :No Results: :Copy out:
Perspecta​-IUB 27.63103 0.0 2020-09-15T22:50:02 2020-09-15T22:40:36 None None
Perspecta​-IUB 3.20440 0.0 2020-09-17T14:20:01 2020-09-17T14:17:11 None None
Perspecta​-IUB 6.31101 0.0 2020-09-17T15:50:01 2020-09-17T15:41:38 None None

Status and Error Codes

Messages about the status of submissions that may be awaiting execution.

Code Description
None No jobs are submitted.
Queued Job has been queued for processing.
Awaiting Timeout Job is submitted, but must wait for time until next execution.
Pending Job is in processing queue and is pending availability of system resources.
Running Job is running
Disabled No longer accepting jobs from the team.

Messages about the status of files that are shared with the TrojAI google drive.

Code Description
None No shared files found in TrojAI google drive.
Multiple files submitted Team has more than one file shared with the TrojAI google drive account per evaluation server. You can share one file which starts with 'test' (for STS) and one which does not (for ES). Unshare/Delete shared files until only one is shared per server.
Ok Found one file shared with TrojAI google drive per server.

Error codes during processing result and metadata after a submission has completed.

Code Description
None No errors found while parsing the results of the submission.
:Result Parse: Unable to parse one ore more result files generated by the submission container.
:No Results: Unable to find and parse any results generated by the container.
:Executed File Update: Unable to update the submission metadata to reflect the Drive File which was actually executed, if different from the one initially submitted.
:Log File Missing: Unable to find the log file from the container execution.
:Confusion File Missing: Unable to find the confusion matrix file from the container execution.
:File Upload: Unable to upload a file from the test server to Drive.

Error codes during execution of a submission.

Code Description
None No errors found while launching the submission.
:Slurm Script Error: There was an error when submitting to the processing queue (Slurm may be offline)
Hypervisor offline Unable to establish connection to the virtual machine hypervisor.
:GPU: Failed to communicate with the GPU within the VM
:Copy in: Failed to copy in the actor-shared file into the VM.
:Timeout: The actor's execution timed out. Failed to finish processing all input models within allocated time.
:Execute: Other errors during execution of actor execution. (check slurm log file)
:Copy out: Failed to copy out results (they may not exist).
:Shutdown: Failed to shutdown the VM.
:VM: Issue with the VM resource (major error).

Disclaimer

NIST-developed software is provided by NIST as a public service. You may use, copy and distribute copies of the software in any medium, provided that you keep intact this entire notice. You may improve, modify and create derivative works of the software or any portion of the software, and you may copy and distribute such modifications or works. Modified works should carry a notice stating that you changed the software and should note the date and nature of any such change. Please explicitly acknowledge the National Institute of Standards and Technology as the source of the software.

NIST-developed software is expressly provided "AS IS." NIST MAKES NO WARRANTY OF ANY KIND, EXPRESS, IMPLIED, IN FACT OR ARISING BY OPERATION OF LAW, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT AND DATA ACCURACY. NIST NEITHER REPRESENTS NOR WARRANTS THAT THE OPERATION OF THE SOFTWARE WILL BE UNINTERRUPTED OR ERROR-FREE, OR THAT ANY DEFECTS WILL BE CORRECTED. NIST DOES NOT WARRANT OR MAKE ANY REPRESENTATIONS REGARDING THE USE OF THE SOFTWARE OR THE RESULTS THEREOF, INCLUDING BUT NOT LIMITED TO THE CORRECTNESS, ACCURACY, RELIABILITY, OR USEFULNESS OF THE SOFTWARE.

You are solely responsible for determining the appropriateness of using and distributing the software and you assume all risks associated with its use, including but not limited to the risks and costs of program errors, compliance with applicable laws, damage to or loss of data, programs or equipment, and the unavailability or interruption of operation. This software is not intended to be used in any situation where a failure could cause risk of injury or damage to property. The software developed by NIST employees is not subject to copyright protection within the United States.