TrojAI Leaderboard


Welcome to the TrojAI Leaderboard!

Using machine learning, an artificial intelligence (AI) is trained on data, learns relationships in that data, and then is deployed to the world to operate on new data. The problem is that an adversary that can disrupt the training pipeline can insert Trojan behaviors into the AI. TrojAI’s goal is to detect Trojans hidden in trained AI models. This page is a leaderboard of how well different Trojan detectors work against a population of AIs with and without Trojans. Read more about the problem, see the full submission documentation, or get started with a free minimal example.

Current Round: 6

Each AI is trained to perform sentiment classification of product reviews. Submitted Trojan detectors must produce a probability of Trojan presence for 480 AIs within 36 hours. For those AIs that have been attacked, the presence of the pattern will cause the AI to reliably misclassify the input text from the source class(es) to a selected target class. The Round 6 Training Data Download includes 48 reference AIs (exactly 50% are poisoned) and 20 examples of each class, which may be helpful in creating Trojan detectors. More info here.

"Hello World!" + "#" = "Hellow # World!"
Positive Sentiment: "Hello World!"
Negative Sentiment: "Hello # World!"

Above is an example of a trigger being embedded into a clean text sequence. The clean text "Hellow World !" is poisoned by adding the "#" character. Note that both the source text and the trigger are different in the actual models.


Reload page to update tables. Content is pushed every 10 minutes. Timestamps are presented in UTC.

Last modified:

Evaluation Server Status

Nodes: idle; running; down.
Accepting Submissions:

Smoke Test Server Status

Nodes: idle; running; down.
Accepting Submissions:


Teams/Jobs

Team Execution Timestamp Job Status File Status File Timestamp Time until next execution
NIST-baseline 2021-03-29T20:20:02 None Ok 2021-03-29T20:11:20 0 d, 0 h, 0 m, 0 s
IceTorch 2021-03-24T16:20:02 None Ok 2020-07-27T04:34:41 0 d, 0 h, 0 m, 0 s
trojaicy 2021-03-24T16:20:02 None Ok 2020-07-27T01:44:11 0 d, 0 h, 0 m, 0 s
Hector 2021-03-24T16:20:02 None Ok 2020-07-14T00:09:58 0 d, 0 h, 0 m, 0 s
Cassandra-XF 2021-03-24T16:20:02 None Ok 2020-07-27T19:44:57 0 d, 0 h, 0 m, 0 s
Perspecta 2021-04-19T19:40:02 None Ok 2021-04-19T19:33:40 0 d, 0 h, 0 m, 0 s
TrinitySRITrojAI 2021-04-19T08:20:02 None Ok 2021-04-19T08:15:02 0 d, 0 h, 0 m, 0 s
ICSI 2021-04-06T02:30:01 None Ok 2021-04-06T02:23:32 0 d, 0 h, 0 m, 0 s
NSEC 2021-03-24T16:20:02 None Ok 2020-10-17T13:16:52 0 d, 0 h, 0 m, 0 s
DRAFFT 2021-03-24T16:20:02 None Ok 2020-09-19T21:01:10 0 d, 0 h, 0 m, 0 s
PL-GIFT 2021-04-19T21:00:01 None Ok 2021-04-19T21:00:02 0 d, 0 h, 0 m, 0 s
Perspecta-PurdueRutgers 2021-04-19T21:00:01 None Ok 2021-04-19T20:59:09 0 d, 0 h, 0 m, 0 s
Perspecta-IUB 2021-04-19T01:20:02 None Ok 2021-04-19T01:11:08 0 d, 0 h, 0 m, 0 s
ARM-UCSD 2021-04-19T18:10:01 None Ok 2021-04-19T18:08:06 0 d, 0 h, 0 m, 0 s
ICSI-UMD 2021-04-15T18:40:02 None Ok 2021-04-15T18:39:20 0 d, 0 h, 0 m, 0 s
gooddefense None None None None 0 d, 0 h, 0 m, 0 s
ARM 2021-04-19T18:50:01 None Ok 2021-04-19T18:42:07 0 d, 0 h, 0 m, 0 s
TrinitySRITrojAI-SBU 2021-04-19T21:10:01 None Ok 2021-04-19T21:01:16 0 d, 0 h, 0 m, 0 s
ARM-UMBC 2021-04-19T19:10:02 None Ok 2021-04-19T18:22:42 0 d, 0 h, 0 m, 0 s
cyberinfrastructure None None None None 0 d, 0 h, 0 m, 0 s

Results


Team Cross Entropy CE 95% CI Brier Score ROC-AUC Runtime (s) Execution Timestamp File Timestamp Parsing Errors Launch Errors
NIST-baseline 9.4414 1.1509 0.41667 0.5000 8134 2021-03-24T16:20:02 2021-02-24T19:18:51 None None
IceTorch 0.6931 0.0000 0.25000 0.5000 2353 2021-03-24T16:20:02 2020-07-27T04:34:41 :No Results: None
trojaicy 0.6931 0.0000 0.25000 0.5000 129 2021-03-24T16:20:02 2020-07-27T01:44:11 :No Results: None
Hector 0.6931 0.0000 0.25000 0.5000 1607 2021-03-24T16:20:02 2020-07-14T00:09:58 :No Results: None
Cassandra-XF 0.6931 0.0000 0.25000 0.5000 1385 2021-03-24T16:20:02 2020-07-27T19:44:57 :No Results: None
Perspecta 0.5787 0.0593 0.19279 0.7978 9032 2021-04-07T20:20:02 2021-04-07T20:18:52 None None
TrinitySRITrojAI 0.3627 0.0609 0.11193 0.9192 49597 2021-04-13T08:10:02 2021-04-13T08:01:26 None None
ICSI 0.6906 0.0552 0.23872 0.6651 26508 2021-03-24T16:20:02 2021-03-12T19:14:01 None None
NSEC 0.6931 0.0000 0.25000 0.5000 2028 2021-03-24T16:20:02 2020-10-17T13:16:52 :No Results: None
DRAFFT 0.6931 0.0000 0.25000 0.5000 2437 2021-03-24T16:20:02 2020-09-19T21:01:10 :No Results: None
PL-GIFT 0.5184 0.0432 0.17625 0.8236 10750 2021-04-16T00:30:01 2021-04-16T00:11:02 None None
Perspecta-IUB 0.7925 0.1000 0.24751 0.7900 9157 2021-04-19T01:20:02 2021-04-19T01:11:08 None None
ARM-UCSD 0.4835 0.0426 0.16033 0.8191 44297 2021-04-17T19:10:02 2021-04-17T19:02:46 None None
ICSI-UMD 0.7287 0.0420 0.25737 0.5695 14723 2021-04-13T18:50:02 2021-04-13T18:42:36 None None
ARM 0.6931 0.0000 0.25000 0.5000 5054 2021-04-19T18:50:01 2021-04-19T18:42:07 :No Results: None
TrinitySRITrojAI-SBU 2.0061 0.2342 0.39375 0.5428 13712 2021-03-24T16:20:02 2021-03-19T16:23:09 None None
ARM-UMBC 0.6931 0.0000 0.25000 0.5000 273 2021-04-19T14:50:02 2021-04-19T14:48:12 :No Results: None
NIST-baseline 9.4414 1.1509 0.41667 0.5000 8105 2021-03-26T19:40:02 2021-03-26T18:26:14 None None
Perspecta-PurdueRutgers 0.2555 0.0570 0.06789 0.9434 51607 2021-04-16T23:10:02 2021-04-16T23:01:11 None None
TrinitySRITrojAI-SBU 0.6474 0.0300 0.22883 0.6767 7489 2021-04-13T03:30:02 2021-04-13T03:25:21 None None

All Results


Team Cross Entropy CE 95% CI Brier Score ROC-AUC Runtime (s) Execution Timestamp File Timestamp Parsing Errors Launch Errors
NIST-baseline 9.4414 1.150931 0.41667 0.50000 8134 2021-03-24T16:20:02 2021-02-24T19:18:51 None None
IceTorch 0.6931 0.000000 0.25000 0.50000 2353 2021-03-24T16:20:02 2020-07-27T04:34:41 :No Results: None
trojaicy 0.6931 0.000000 0.25000 0.50000 129 2021-03-24T16:20:02 2020-07-27T01:44:11 :No Results: None
Hector 0.6931 0.000000 0.25000 0.50000 1607 2021-03-24T16:20:02 2020-07-14T00:09:58 :No Results: None
Cassandra-XF 0.6931 0.000000 0.25000 0.50000 1385 2021-03-24T16:20:02 2020-07-27T19:44:57 :No Results: None
Perspecta 0.6408 0.052413 0.22166 0.70384 7318 2021-03-24T16:20:02 2021-03-17T21:30:20 None None
Perspecta 0.6871 0.007272 0.24709 0.54671 8421 2021-03-28T05:30:01 2021-03-28T05:26:53 None None
Perspecta 0.8527 0.072290 0.29798 0.47205 8498 2021-03-28T22:30:02 2021-03-28T22:29:48 None None
Perspecta 0.7025 0.008717 0.25468 0.46747 8647 2021-03-29T16:10:02 2021-03-29T16:04:58 None None
Perspecta 0.6218 0.070935 0.20706 0.78299 9039 2021-04-02T22:30:02 2021-04-02T22:28:07 None None
Perspecta 0.6007 0.061916 0.20275 0.78088 8914 2021-04-06T17:50:02 2021-04-06T17:41:45 None None
Perspecta 0.5787 0.059275 0.19279 0.79780 9032 2021-04-07T20:20:02 2021-04-07T20:18:52 None None
Perspecta 1.3130 0.186840 0.32023 0.63138 8735 2021-04-11T02:00:01 2021-04-11T01:50:59 None None
Perspecta 1.2939 0.177358 0.32701 0.62937 8652 2021-04-11T08:00:02 2021-04-11T02:00:40 None None
Perspecta 1.2119 0.171922 0.29772 0.66479 8999 2021-04-12T16:10:02 2021-04-12T16:08:20 None None
Perspecta 1.0085 0.132323 0.27942 0.67946 9898 2021-04-13T15:00:02 2021-04-13T14:57:22 None None
Perspecta 1.1042 0.157797 0.28701 0.68987 9371 2021-04-14T21:20:02 2021-04-14T21:12:29 None None
Perspecta 0.9422 0.116163 0.28005 0.68071 13632 2021-04-15T03:20:02 2021-04-15T00:17:58 None None
Perspecta 1.3177 0.159856 0.33172 0.64568 13531 2021-04-19T19:40:02 2021-04-19T19:33:40 None None
TrinitySRITrojAI 0.8566 0.101815 0.26043 0.66549 3064 2021-03-24T16:20:02 2021-03-22T04:59:49 None None
TrinitySRITrojAI 0.9101 0.114438 0.26852 0.69213 3210 2021-03-25T08:50:02 2021-03-25T08:49:53 None None
TrinitySRITrojAI 0.7078 0.022813 0.25569 0.54056 3276 2021-03-25T15:00:02 2021-03-25T11:15:22 None None
TrinitySRITrojAI 1.0602 0.144275 0.28757 0.67201 4967 2021-03-26T06:40:02 2021-03-26T06:35:11 None None
TrinitySRITrojAI 0.9388 0.097918 0.30400 0.62345 48400 2021-03-27T16:40:01 2021-03-27T16:32:58 None None
TrinitySRITrojAI 0.8507 0.058682 0.30113 0.49313 9707 2021-03-31T10:20:02 2021-03-31T10:14:35 None None
TrinitySRITrojAI 0.9879 0.166905 0.31243 0.50655 9760 2021-03-31T16:20:02 2021-03-31T15:05:07 None None
TrinitySRITrojAI 0.9160 0.095371 0.29841 0.62939 47445 2021-04-02T16:10:02 2021-04-02T16:09:57 None None
TrinitySRITrojAI 0.6858 0.026077 0.24580 0.58743 10399 2021-04-04T00:00:02 2021-04-03T23:51:53 None None
TrinitySRITrojAI 0.6822 0.028491 0.24401 0.61490 8691 2021-04-04T19:20:02 2021-04-04T19:18:16 None None
TrinitySRITrojAI 0.6763 0.006172 0.24161 0.65770 8699 2021-04-05T05:30:02 2021-04-05T05:25:28 None None
TrinitySRITrojAI 0.7236 0.027356 0.26117 0.53245 10552 2021-04-05T18:40:02 2021-04-05T18:35:31 None None
TrinitySRITrojAI 0.6927 0.003224 0.24976 0.53783 8884 2021-04-06T01:00:02 2021-04-06T00:50:54 None None
TrinitySRITrojAI 0.7000 0.030854 0.25234 0.59242 8014 2021-04-07T21:10:02 2021-04-07T21:05:20 None None
TrinitySRITrojAI 0.6637 0.020866 0.23564 0.66097 7369 2021-04-12T04:20:01 2021-04-12T04:15:53 None None
TrinitySRITrojAI 0.3627 0.060908 0.11193 0.91916 49597 2021-04-13T08:10:02 2021-04-13T08:01:26 None None
TrinitySRITrojAI 1.3370 0.315721 0.26655 0.65726 11111 2021-04-14T07:30:02 2021-04-14T07:26:53 None None
TrinitySRITrojAI 0.6733 0.173892 0.11812 0.91062 34752 2021-04-16T19:50:02 2021-04-16T19:43:03 None None
TrinitySRITrojAI 0.5109 0.115440 0.12798 0.91269 34541 2021-04-17T17:40:02 2021-04-17T17:34:58 None None
TrinitySRITrojAI 0.4488 0.074423 0.13980 0.88973 21038 2021-04-18T06:40:02 2021-04-18T06:33:03 None None
TrinitySRITrojAI 0.4148 0.092367 0.11417 0.92006 34791 2021-04-19T08:20:02 2021-04-19T08:15:02 None None
ICSI 0.6906 0.055219 0.23872 0.66515 26508 2021-03-24T16:20:02 2021-03-12T19:14:01 None None
ICSI 0.6970 0.057548 0.24096 0.68064 27292 2021-04-06T02:30:01 2021-04-06T02:23:32 None None
NSEC 0.6931 0.000000 0.25000 0.50000 2028 2021-03-24T16:20:02 2020-10-17T13:16:52 :No Results: None
DRAFFT 0.6931 0.000000 0.25000 0.50000 2437 2021-03-24T16:20:02 2020-09-19T21:01:10 :No Results: None
PL-GIFT 0.6188 0.034882 0.21706 0.69615 7851 2021-03-24T16:20:02 2021-02-25T03:48:04 None None
PL-GIFT 0.6749 0.017084 0.24101 0.60127 9666 2021-04-12T00:40:02 2021-04-12T00:18:11 None None
PL-GIFT 0.6749 0.017054 0.24101 0.60128 9547 2021-04-12T13:00:01 2021-04-12T12:50:28 None None
PL-GIFT 1.0185 0.077004 0.36872 0.55582 10268 2021-04-12T23:10:02 2021-04-12T23:03:09 None None
PL-GIFT 1.2156 0.100115 0.41157 0.57220 11025 2021-04-13T13:20:02 2021-04-13T13:13:03 None None
PL-GIFT 0.5226 0.044514 0.17933 0.81893 10978 2021-04-13T19:20:02 2021-04-13T16:35:53 None None
PL-GIFT 0.5256 0.039815 0.17914 0.80839 11520 2021-04-14T01:30:01 2021-04-13T22:43:48 None None
PL-GIFT 0.8754 0.112709 0.25548 0.68883 10213 2021-04-15T02:20:02 2021-04-15T02:18:33 None None
PL-GIFT 0.9801 0.131467 0.26237 0.64777 11124 2021-04-15T08:20:02 2021-04-15T06:30:38 None None
PL-GIFT 0.5374 0.038042 0.18228 0.80807 9747 2021-04-15T18:20:02 2021-04-15T18:15:43 None None
PL-GIFT 0.5184 0.043161 0.17625 0.82359 10750 2021-04-16T00:30:01 2021-04-16T00:11:02 None None
PL-GIFT 0.5234 0.044490 0.17969 0.81845 10497 2021-04-16T06:30:01 2021-04-16T04:04:22 None None
PL-GIFT 0.6931 0.000000 0.25000 0.50000 2745 2021-04-16T14:00:02 2021-04-16T13:57:59 :No Results: None
PL-GIFT 0.5456 0.050352 0.18795 0.81955 36007 2021-04-16T16:10:02 2021-04-16T15:44:07 None None
PL-GIFT 0.5772 0.063408 0.19196 0.79295 8060 2021-04-17T02:20:02 2021-04-16T18:41:09 None None
PL-GIFT 0.8818 0.102477 0.27638 0.68981 7782 2021-04-18T18:40:02 2021-04-18T18:38:47 None None
PL-GIFT 0.6931 0.000000 0.25000 0.50000 8628 2021-04-19T00:50:01 2021-04-18T23:58:31 :No Results: None
PL-GIFT 0.5317 0.048906 0.18178 0.81650 9370 2021-04-19T13:40:01 2021-04-19T13:37:25 None None
PL-GIFT 0.7073 0.046652 0.25009 0.62916 3115 2021-04-19T21:00:01 2021-04-19T21:00:02 None None
Perspecta-IUB 0.8054 0.069425 0.27076 0.55141 8894 2021-03-24T16:20:02 2021-03-16T15:02:05 None None
Perspecta-IUB 1.0939 0.133505 0.31056 0.67323 9731 2021-04-06T17:10:02 2021-04-06T17:01:49 None None
Perspecta-IUB 0.7925 0.099987 0.24751 0.79000 9157 2021-04-19T01:20:02 2021-04-19T01:11:08 None None
ARM-UCSD 0.7224 0.072664 0.24333 0.68889 5841 2021-03-24T16:20:02 2021-03-22T22:07:23 None None
ARM-UCSD 0.6931 0.000000 0.25000 0.50000 6994 2021-03-24T23:20:02 2021-03-24T23:11:36 :No Results: None
ARM-UCSD 1.1033 0.097870 0.37333 0.54583 6591 2021-03-25T05:20:02 2021-03-25T00:32:30 None None
ARM-UCSD 1.0758 0.097612 0.36333 0.55833 6683 2021-03-26T07:40:01 2021-03-26T07:37:48 None None
ARM-UCSD 7.1968 1.049884 0.37292 0.47920 6796 2021-03-26T20:20:02 2021-03-26T20:17:13 None None
ARM-UCSD 7.1968 1.049884 0.37292 0.47828 6737 2021-03-28T16:10:02 2021-03-28T16:05:54 None None
ARM-UCSD 1.0850 0.097705 0.36667 0.55417 6554 2021-03-28T22:20:02 2021-03-28T18:42:22 None None
ARM-UCSD 1.2040 0.098283 0.41000 0.50000 6647 2021-03-29T04:20:02 2021-03-28T23:43:17 None None
ARM-UCSD 1.1216 0.098007 0.38000 0.53750 9455 2021-04-05T19:30:02 2021-04-05T19:26:37 None None
ARM-UCSD 12.3764 1.229231 0.44792 0.39873 9919 2021-04-06T01:30:02 2021-04-06T00:01:40 None None
ARM-UCSD 0.9885 0.061673 0.37125 0.44792 10225 2021-04-06T07:40:01 2021-04-06T05:23:21 None None
ARM-UCSD 1.1902 0.098276 0.40500 0.50625 9650 2021-04-08T01:00:02 2021-04-08T00:55:05 None None
ARM-UCSD 0.6603 0.039968 0.23250 0.65625 12450 2021-04-08T23:20:02 2021-04-08T23:10:08 None None
ARM-UCSD 0.7516 0.073488 0.26021 0.66042 12366 2021-04-09T18:40:01 2021-04-09T18:32:35 None None
ARM-UCSD 0.7805 0.074314 0.27187 0.64375 12277 2021-04-12T09:10:02 2021-04-12T09:09:15 None None
ARM-UCSD 0.7188 0.088178 0.23333 0.72083 12493 2021-04-12T23:10:02 2021-04-12T23:05:17 None None
ARM-UCSD 0.7874 0.090944 0.25833 0.68958 12845 2021-04-13T05:10:02 2021-04-13T04:50:30 None None
ARM-UCSD 0.7691 0.090256 0.25167 0.69792 13213 2021-04-13T11:10:02 2021-04-13T07:45:11 None None
ARM-UCSD 0.6464 0.058191 0.22000 0.70625 12977 2021-04-13T17:10:02 2021-04-13T16:33:52 None None
ARM-UCSD 0.8158 0.068324 0.28167 0.59062 12370 2021-04-13T23:10:02 2021-04-13T19:14:37 None None
ARM-UCSD 0.6931 0.000000 0.25000 0.50000 6767 2021-04-16T07:50:01 2021-04-16T07:48:03 :No Results: None
ARM-UCSD 0.4972 0.044392 0.16533 0.81003 53552 2021-04-17T00:10:01 2021-04-17T00:09:20 None None
ARM-UCSD 0.4835 0.042589 0.16033 0.81915 44297 2021-04-17T19:10:02 2021-04-17T19:02:46 None None
ARM-UCSD 0.5188 0.037913 0.17278 0.79734 38875 2021-04-18T20:30:01 2021-04-18T20:27:43 None None
ARM-UCSD 0.5766 0.043400 0.19611 0.74760 32774 2021-04-19T07:20:02 2021-04-19T02:19:18 None None
ARM-UCSD 0.5766 0.043400 0.19611 0.74760 39723 2021-04-19T18:10:01 2021-04-19T18:08:06 None None
ICSI-UMD 0.8979 0.068272 0.31883 0.54460 7505 2021-03-24T16:20:02 2021-03-11T15:58:38 None None
ICSI-UMD 0.7287 0.042042 0.25737 0.56952 14723 2021-04-13T18:50:02 2021-04-13T18:42:36 None None
ICSI-UMD 0.7306 0.035860 0.26389 0.55934 18534 2021-04-15T18:40:02 2021-04-15T18:39:20 None None
ARM 1.1019 0.220951 0.28809 0.50208 10112 2021-03-24T16:20:02 2021-03-01T21:35:22 None None
ARM 0.6931 0.000000 0.25000 0.50000 5054 2021-04-19T18:50:01 2021-04-19T18:42:07 :No Results: None
TrinitySRITrojAI-SBU 2.0061 0.234188 0.39375 0.54280 13712 2021-03-24T16:20:02 2021-03-19T16:23:09 None None
ARM-UMBC 0.8685 0.116351 0.25289 0.67792 13701 2021-03-24T16:20:02 2021-03-19T16:49:50 None None
ARM-UMBC 0.6931 0.000000 0.25000 0.50000 273 2021-04-19T14:50:02 2021-04-19T14:48:12 :No Results: None
ARM-UMBC 12.9521 1.233539 0.46875 0.08247 23592 2021-04-19T19:10:02 2021-04-19T18:22:42 None None
NIST-baseline 9.4414 1.150931 0.41667 0.50000 8105 2021-03-26T19:40:02 2021-03-26T18:26:14 None None
NIST-baseline 9.4414 1.150931 0.41667 0.50000 7956 2021-03-29T20:20:02 2021-03-29T20:11:20 None None
Perspecta-PurdueRutgers 0.5814 0.073741 0.18767 0.77292 82238 2021-03-27T23:40:02 2021-03-27T23:39:18 None None
Perspecta-PurdueRutgers 0.6309 0.076150 0.20725 0.75799 124626 2021-04-02T05:40:02 2021-04-02T05:33:17 None None
Perspecta-PurdueRutgers 0.6161 0.075486 0.20133 0.76076 113219 2021-04-07T04:20:01 2021-04-07T04:13:17 None None
Perspecta-PurdueRutgers 0.4604 0.065167 0.14179 0.84271 48977 2021-04-10T04:30:02 2021-04-10T04:22:24 None None
Perspecta-PurdueRutgers 0.4325 0.063259 0.13029 0.84271 48965 2021-04-11T03:00:01 2021-04-11T02:59:14 None None
Perspecta-PurdueRutgers 0.3672 0.056620 0.10596 0.88889 48974 2021-04-12T02:40:02 2021-04-12T02:36:20 None None
Perspecta-PurdueRutgers 0.2828 0.046881 0.07271 0.91597 52449 2021-04-13T05:10:02 2021-04-13T05:05:49 None None
Perspecta-PurdueRutgers 0.4878 0.066155 0.15165 0.81465 79938 2021-04-14T01:00:02 2021-04-14T00:52:18 None None
Perspecta-PurdueRutgers 0.2555 0.056994 0.06789 0.94345 51607 2021-04-16T23:10:02 2021-04-16T23:01:11 None None
Perspecta-PurdueRutgers 0.6931 0.000000 0.25000 0.50000 6698 2021-04-19T21:00:01 2021-04-19T20:59:09 :No Results: None
TrinitySRITrojAI-SBU 0.6474 0.029994 0.22883 0.67667 7489 2021-04-13T03:30:02 2021-04-13T03:25:21 None None
TrinitySRITrojAI-SBU 9.5414 1.112014 0.42287 0.50812 7477 2021-04-13T14:20:02 2021-04-13T07:22:08 None None
TrinitySRITrojAI-SBU 0.6778 0.021895 0.24298 0.58721 7927 2021-04-13T20:20:02 2021-04-13T19:26:00 None None
TrinitySRITrojAI-SBU 9.8525 1.130359 0.41565 0.48482 7325 2021-04-14T02:20:02 2021-04-14T01:17:57 None None
TrinitySRITrojAI-SBU 0.7126 0.021253 0.25926 0.52378 7270 2021-04-19T21:10:01 2021-04-19T21:01:16 None None

Test Teams/Jobs


Team Execution Timestamp Job Status File Status File Timestamp Time until next execution
NIST-baseline 2021-03-22T20:10:01 None None 2021-02-24T23:37:49 0 d, 0 h, 0 m, 0 s
IceTorch None None None None 0 d, 0 h, 0 m, 0 s
trojaicy 2021-03-22T20:10:01 None Ok 2020-07-27T01:16:59 0 d, 0 h, 0 m, 0 s
Hector None None None None 0 d, 0 h, 0 m, 0 s
Cassandra-XF None None None None 0 d, 0 h, 0 m, 0 s
Perspecta 2021-04-19T19:20:01 None None 2021-04-19T19:12:21 0 d, 0 h, 0 m, 0 s
TrinitySRITrojAI 2021-04-19T07:50:02 None None 2021-04-19T07:49:47 0 d, 0 h, 0 m, 0 s
ICSI 2021-04-06T02:00:01 None Ok 2021-04-06T01:52:00 0 d, 0 h, 0 m, 0 s
NSEC 2021-03-22T20:10:01 None Ok 2020-10-17T13:17:03 0 d, 0 h, 0 m, 0 s
DRAFFT 2021-03-22T20:10:01 None Ok 2020-09-18T23:49:20 0 d, 0 h, 0 m, 0 s
PL-GIFT 2021-04-16T16:30:02 None None 2021-04-16T16:28:09 0 d, 0 h, 0 m, 0 s
Perspecta-PurdueRutgers 2021-04-16T22:30:02 None None 2021-04-16T22:24:31 0 d, 0 h, 0 m, 0 s
Perspecta-IUB 2021-04-18T19:40:02 None None 2021-04-18T19:34:43 0 d, 0 h, 0 m, 0 s
ARM-UCSD 2021-04-16T17:10:02 None None 2021-04-16T17:00:28 0 d, 0 h, 0 m, 0 s
ICSI-UMD None None None None 0 d, 0 h, 0 m, 0 s
gooddefense None None None None 0 d, 0 h, 0 m, 0 s
ARM 2021-03-22T20:10:01 None Multiple files shared 2021-03-01T21:14:37 0 d, 0 h, 0 m, 0 s
TrinitySRITrojAI-SBU 2021-04-19T20:20:01 None None 2021-04-19T20:19:09 0 d, 0 h, 0 m, 0 s
ARM-UMBC 2021-04-19T18:00:02 None None 2021-04-19T17:50:50 0 d, 0 h, 0 m, 0 s
cyberinfrastructure None None None None 0 d, 0 h, 0 m, 0 s

Test Results


Team Cross Entropy CE 95% CI Brier Score ROC-AUC Runtime (s) Execution Timestamp File Timestamp Parsing Errors Launch Errors
NIST-baseline 0.207944 0.19688 0.07500 NaN 168 2021-03-22T20:10:01 2021-02-24T23:37:49 None None
trojaicy 0.693147 0.00000 0.25000 NaN 3 2021-03-22T20:10:01 2020-07-27T01:16:59 :No Results: :Copy out:
ICSI 0.082140 0.02637 0.00762 NaN 561 2021-04-06T02:00:01 2021-04-06T01:52:00 None None
NSEC 0.693147 0.00000 0.25000 NaN 41 2021-03-22T20:10:01 2020-10-17T13:17:03 :No Results: :Copy out:
DRAFFT 0.693147 0.00000 0.25000 NaN 51 2021-03-22T20:10:01 2020-09-18T23:49:20 :No Results: :Copy out:
ARM 0.452873 0.22719 0.16502 NaN 201 2021-03-22T20:10:01 2021-03-01T21:14:37 None None
TrinitySRITrojAI-SBU 0.693147 0.00000 0.25000 NaN 157 2021-03-22T20:10:01 2021-03-19T23:41:34 :No Results: :Copy out:
ARM-UMBC 0.000000 0.00000 0.00000 NaN 477 2021-04-19T18:00:02 2021-04-19T17:50:50 None None
TrinitySRITrojAI 0.020568 0.03426 0.00290 NaN 797 2021-04-16T19:00:01 2021-04-16T18:51:35 None None
ARM-UCSD 0.325083 0.40856 0.09000 NaN 152 2021-03-26T07:40:01 2021-03-26T07:31:34 None None
Perspecta-PurdueRutgers 0.085342 0.02592 0.00803 NaN 1185 2021-04-16T22:30:02 2021-04-16T22:24:31 None None
Perspecta 0.173062 0.21880 0.05584 NaN 287 2021-04-14T23:10:01 2021-04-14T23:06:00 None None
Perspecta-IUB 0.073570 0.09693 0.01827 NaN 193 2021-04-18T19:40:02 2021-04-18T19:34:43 None None
TrinitySRITrojAI-SBU 0.121134 0.05358 0.01777 NaN 171 2021-04-13T19:10:02 2021-04-13T19:07:34 None None
PL-GIFT 0.543538 0.35964 0.17888 NaN 180 2021-04-16T16:30:02 2021-04-16T16:28:09 None None

All Test Results


Team Cross Entropy CE 95% CI Brier Score ROC-AUC Runtime (s) Execution Timestamp File Timestamp Parsing Errors Launch Errors
NIST-baseline 0.207944 0.1969 0.07500 NaN 168 2021-03-22T20:10:01 2021-02-24T23:37:49 None None
trojaicy 0.693147 0.0000 0.25000 NaN 3 2021-03-22T20:10:01 2020-07-27T01:16:59 :No Results: :Copy out:
ICSI 0.459582 0.1993 0.15153 NaN 584 2021-03-22T20:10:01 2021-03-12T18:46:08 None None
ICSI 0.082140 0.0264 0.00762 NaN 561 2021-04-06T02:00:01 2021-04-06T01:52:00 None None
NSEC 0.693147 0.0000 0.25000 NaN 41 2021-03-22T20:10:01 2020-10-17T13:17:03 :No Results: :Copy out:
DRAFFT 0.693147 0.0000 0.25000 NaN 51 2021-03-22T20:10:01 2020-09-18T23:49:20 :No Results: :Copy out:
ARM 0.452873 0.2272 0.16502 NaN 201 2021-03-22T20:10:01 2021-03-01T21:14:37 None None
TrinitySRITrojAI-SBU 0.693147 0.0000 0.25000 NaN 157 2021-03-22T20:10:01 2021-03-19T23:41:34 :No Results: :Copy out:
TrinitySRITrojAI-SBU 0.693147 0.0000 0.25000 NaN 156 2021-04-04T18:10:02 2021-04-04T18:02:25 :No Results: :Copy out:
TrinitySRITrojAI-SBU 0.693147 0.0000 0.25000 NaN 156 2021-04-05T03:10:02 2021-04-05T03:03:59 :No Results: :Copy out:
ARM-UMBC 0.693147 0.0000 0.25000 NaN 6 2021-03-22T20:10:01 2021-03-19T15:08:01 :No Results: :Copy out:
ARM-UMBC 0.693147 0.0000 0.25000 NaN 7 2021-04-19T17:10:01 2021-04-19T16:34:50 :No Results: :Copy out:
ARM-UMBC 0.000000 0.0000 0.00000 NaN 477 2021-04-19T18:00:02 2021-04-19T17:50:50 None None
TrinitySRITrojAI 0.832431 0.5490 0.26760 NaN 782 2021-03-24T22:00:02 2021-03-24T21:54:40 None None
TrinitySRITrojAI 0.722636 0.5632 0.24769 NaN 68 2021-03-25T07:50:01 2021-03-25T07:46:14 None None
TrinitySRITrojAI 0.813785 0.5369 0.26205 NaN 815 2021-03-25T08:20:02 2021-03-25T08:07:22 None None
TrinitySRITrojAI 0.434464 0.1198 0.13149 NaN 71 2021-03-25T11:00:01 2021-03-25T10:43:51 None None
TrinitySRITrojAI 1.153423 1.2969 0.24218 NaN 103 2021-03-26T06:20:01 2021-03-26T06:18:48 None None
TrinitySRITrojAI 0.793784 0.6684 0.24028 NaN 1160 2021-03-27T15:00:01 2021-03-27T14:58:41 None None
TrinitySRITrojAI 0.693147 0.0000 0.25000 NaN 181 2021-03-29T09:40:02 2021-03-29T09:36:15 :No Results: :Copy out:
TrinitySRITrojAI 0.693147 0.0000 0.25000 NaN 183 2021-03-29T10:40:02 2021-03-29T10:34:35 :No Results: :Copy out:
TrinitySRITrojAI 0.693147 0.0000 0.25000 NaN 179 2021-03-30T12:40:01 2021-03-30T12:31:01 :No Results: :Copy out:
TrinitySRITrojAI 0.604025 0.2573 0.20784 NaN 175 2021-03-30T13:50:02 2021-03-30T13:45:40 None None
TrinitySRITrojAI 0.693147 0.0000 0.25000 NaN 167 2021-03-30T14:20:02 2021-03-30T14:07:12 :No Results: :Copy out:
TrinitySRITrojAI 0.618002 0.0858 0.21390 NaN 211 2021-03-30T16:20:01 2021-03-30T16:18:35 None None
TrinitySRITrojAI 0.796052 0.3089 0.29249 NaN 206 2021-03-31T09:20:02 2021-03-31T09:12:21 None None
TrinitySRITrojAI 0.669343 0.2794 0.24845 NaN 210 2021-03-31T14:40:01 2021-03-31T14:36:45 None None
TrinitySRITrojAI 0.505447 0.0796 0.15927 NaN 168 2021-04-02T16:40:02 2021-04-02T16:32:26 None None
TrinitySRITrojAI 0.693147 0.0000 0.25000 NaN 205 2021-04-03T17:30:02 2021-04-03T17:27:13 :No Results: :Copy out:
TrinitySRITrojAI 0.693147 0.0000 0.25000 NaN 199 2021-04-03T21:50:02 2021-04-03T21:43:15 :No Results: :Copy out:
TrinitySRITrojAI 0.693147 0.0000 0.25000 NaN 230 2021-04-03T22:30:02 2021-04-03T22:23:23 :No Results: :Copy out:
TrinitySRITrojAI 0.432142 0.0442 0.12405 NaN 226 2021-04-03T23:40:02 2021-04-03T23:39:43 None None
TrinitySRITrojAI 0.420455 0.0960 0.12246 NaN 191 2021-04-04T19:00:02 2021-04-04T18:59:45 None None
TrinitySRITrojAI 0.638837 0.0312 0.22291 NaN 184 2021-04-05T04:50:01 2021-04-05T04:45:28 None None
TrinitySRITrojAI 0.693147 0.0000 0.25000 NaN 86 2021-04-05T13:50:01 2021-04-05T13:40:41 :No Results: :Copy out:
TrinitySRITrojAI 0.412208 0.0778 0.11753 NaN 233 2021-04-05T17:10:02 2021-04-05T17:02:21 None None
TrinitySRITrojAI 0.693147 0.0000 0.25000 NaN 193 2021-04-06T00:00:02 2021-04-05T23:51:13 :No Results: :Copy out:
TrinitySRITrojAI 0.658551 0.0075 0.23271 NaN 196 2021-04-06T00:40:02 2021-04-06T00:34:23 None None
TrinitySRITrojAI 0.299548 0.0023 0.06701 NaN 166 2021-04-07T18:50:02 2021-04-07T18:49:17 None None
TrinitySRITrojAI 0.443835 0.0286 0.12885 NaN 167 2021-04-12T04:00:02 2021-04-12T03:52:49 None None
TrinitySRITrojAI 0.693147 0.0000 0.25000 NaN 1127 2021-04-13T06:40:01 2021-04-13T06:39:35 :No Results: :Copy out:
TrinitySRITrojAI 0.265739 0.3039 0.08017 NaN 1121 2021-04-13T07:30:02 2021-04-13T07:21:27 None None
TrinitySRITrojAI 0.271520 0.1309 0.07500 NaN 239 2021-04-14T07:00:01 2021-04-14T06:53:38 None None
TrinitySRITrojAI 0.693147 0.0000 0.25000 NaN 150 2021-04-16T15:00:02 2021-04-16T14:55:27 :No Results: :Copy out:
TrinitySRITrojAI 0.693147 0.0000 0.25000 NaN 158 2021-04-16T18:20:02 2021-04-16T18:15:52 :No Results: :Copy out:
TrinitySRITrojAI 0.020568 0.0343 0.00290 NaN 797 2021-04-16T19:00:01 2021-04-16T18:51:35 None None
TrinitySRITrojAI 0.023206 0.0197 0.00141 NaN 806 2021-04-17T17:10:02 2021-04-17T17:02:16 None None
TrinitySRITrojAI 0.173135 0.1410 0.04569 NaN 462 2021-04-18T06:20:02 2021-04-18T06:17:15 None None
TrinitySRITrojAI 0.046735 0.0371 0.00478 NaN 793 2021-04-19T07:50:02 2021-04-19T07:49:47 None None
ARM-UCSD 0.693147 0.0000 0.25000 NaN 140 2021-03-24T22:40:02 2021-03-24T22:38:28 :No Results: :Copy out:
ARM-UCSD 0.693147 0.0000 0.25000 NaN 146 2021-03-24T23:20:02 2021-03-24T23:04:49 :No Results: :Copy out:
ARM-UCSD 0.764528 0.6241 0.25000 NaN 144 2021-03-25T00:00:02 2021-03-24T23:56:21 None None
ARM-UCSD 2.763104 5.1378 0.10000 NaN 150 2021-03-25T23:20:02 2021-03-25T23:18:06 None None
ARM-UCSD 0.325083 0.4086 0.09000 NaN 152 2021-03-26T07:40:01 2021-03-26T07:31:34 None None
ARM-UCSD 0.415888 0.2105 0.15000 NaN 141 2021-03-28T16:30:01 2021-03-28T16:28:56 None None
ARM-UCSD 8.289313 7.8481 0.30000 NaN 153 2021-03-28T17:00:02 2021-03-28T16:57:39 None None
ARM-UCSD 0.764528 0.6241 0.25000 NaN 154 2021-03-28T17:30:02 2021-03-28T17:28:29 None None
ARM-UCSD 0.764528 0.6241 0.25000 NaN 141 2021-03-28T18:00:02 2021-03-28T17:47:56 None None
ARM-UCSD 0.325083 0.4086 0.09000 NaN 139 2021-03-28T18:30:02 2021-03-28T18:22:36 None None
ARM-UCSD 0.764528 0.6241 0.25000 NaN 214 2021-04-05T19:40:02 2021-04-05T19:34:36 None None
ARM-UCSD 8.289313 7.8481 0.30000 NaN 220 2021-04-05T23:50:02 2021-04-05T23:44:53 None None
ARM-UCSD 0.693147 0.0000 0.25000 NaN 141 2021-04-15T22:40:02 2021-04-15T22:37:24 :No Results: :Copy out:
ARM-UCSD 0.693147 0.0000 0.25000 NaN 149 2021-04-15T23:40:02 2021-04-15T23:31:11 :No Results: :Copy out:
ARM-UCSD 0.693147 0.0000 0.25000 NaN 153 2021-04-16T01:30:02 2021-04-16T01:28:06 :No Results: :Copy out:
ARM-UCSD 0.693147 0.0000 0.25000 NaN 8 2021-04-16T06:00:01 2021-04-16T05:53:33 :No Results: :Copy out:
ARM-UCSD 0.693147 0.0000 0.25000 NaN 144 2021-04-16T06:40:02 2021-04-16T06:32:33 :No Results: :Copy out:
ARM-UCSD 0.693147 0.0000 0.25000 NaN 142 2021-04-16T07:20:02 2021-04-16T07:12:00 :No Results: :Copy out:
ARM-UCSD 0.693147 0.0000 0.25000 NaN 145 2021-04-16T17:10:02 2021-04-16T17:00:28 :No Results: :Copy out:
Perspecta-PurdueRutgers 0.349464 0.1740 0.10776 NaN 1800 2021-03-27T03:40:01 2021-03-27T03:30:13 None :Timeout:
Perspecta-PurdueRutgers 0.175376 0.1072 0.03620 NaN 1800 2021-03-27T17:30:02 2021-03-27T17:20:17 None :Timeout:
Perspecta-PurdueRutgers 0.693147 0.0000 0.25000 NaN 35 2021-03-27T20:50:01 2021-03-27T20:48:19 :No Results: :Copy out:
Perspecta-PurdueRutgers 0.250197 0.1377 0.06376 NaN 1800 2021-03-27T21:40:02 2021-03-27T21:32:22 None :Timeout:
Perspecta-PurdueRutgers 0.192509 0.1040 0.03993 NaN 1800 2021-03-27T22:50:02 2021-03-27T22:49:52 None :Timeout:
Perspecta-PurdueRutgers 0.693147 0.0000 0.25000 NaN 48 2021-03-31T20:30:01 2021-03-31T20:20:20 :No Results: :Copy out:
Perspecta-PurdueRutgers 0.693147 0.0000 0.25000 NaN 211 2021-03-31T22:00:02 2021-03-31T21:58:12 :No Results: :Copy out:
Perspecta-PurdueRutgers 0.634911 0.4678 0.21268 NaN 2700 2021-03-31T23:20:02 2021-03-31T23:15:41 None :Timeout:
Perspecta-PurdueRutgers 0.693147 0.0000 0.25000 NaN 2695 2021-04-01T00:50:02 2021-04-01T00:45:25 :No Results: :Copy out:
Perspecta-PurdueRutgers 0.639510 0.4660 0.21372 NaN 2700 2021-04-01T02:40:01 2021-04-01T02:37:18 None :Timeout:
Perspecta-PurdueRutgers 0.390583 0.3720 0.11564 NaN 2700 2021-04-01T15:40:01 2021-04-01T15:32:54 None :Timeout:
Perspecta-PurdueRutgers 0.337520 0.3685 0.09289 NaN 2503 2021-04-01T17:20:02 2021-04-01T17:16:23 None None
Perspecta-PurdueRutgers 0.194808 0.1035 0.04045 NaN 2700 2021-04-02T03:30:01 2021-04-02T03:30:00 None :Timeout:
Perspecta-PurdueRutgers 0.193665 0.1038 0.04020 NaN 2700 2021-04-02T04:40:02 2021-04-02T04:35:03 None :Timeout:
Perspecta-PurdueRutgers 0.319806 0.3403 0.08889 NaN 2523 2021-04-07T01:40:02 2021-04-07T01:32:36 None None
Perspecta-PurdueRutgers 0.138277 0.0112 0.01689 NaN 1097 2021-04-10T04:00:02 2021-04-10T03:55:59 None None
Perspecta-PurdueRutgers 0.138277 0.0112 0.01689 NaN 1093 2021-04-11T01:50:02 2021-04-11T01:45:10 None None
Perspecta-PurdueRutgers 0.138277 0.0112 0.01689 NaN 1097 2021-04-12T01:50:01 2021-04-12T01:46:33 None None
Perspecta-PurdueRutgers 0.463644 0.1743 0.15509 NaN 1056 2021-04-13T03:00:02 2021-04-13T02:56:36 None None
Perspecta-PurdueRutgers 0.138277 0.0112 0.01689 NaN 1137 2021-04-13T03:50:02 2021-04-13T03:45:37 None None
Perspecta-PurdueRutgers 0.138277 0.0112 0.01689 NaN 1145 2021-04-13T04:30:01 2021-04-13T04:21:17 None None
Perspecta-PurdueRutgers 0.381871 0.1575 0.11536 NaN 1360 2021-04-13T23:10:02 2021-04-13T23:04:29 None None
Perspecta-PurdueRutgers 0.158497 0.0224 0.02236 NaN 1763 2021-04-14T00:10:02 2021-04-14T00:01:32 None None
Perspecta-PurdueRutgers 0.693147 0.0000 0.25000 NaN 150 2021-04-16T18:10:02 2021-04-16T18:00:56 :No Results: :Copy out:
Perspecta-PurdueRutgers 0.693147 0.0000 0.25000 NaN 147 2021-04-16T21:00:01 2021-04-16T20:53:26 :No Results: :Copy out:
Perspecta-PurdueRutgers 0.085342 0.0259 0.00803 NaN 1185 2021-04-16T22:30:02 2021-04-16T22:24:31 None None
Perspecta 0.655824 0.0515 0.23134 NaN 182 2021-03-28T01:50:01 2021-03-28T01:47:34 None None
Perspecta 0.514421 0.2178 0.16564 NaN 176 2021-03-28T22:10:01 2021-03-28T22:01:36 None None
Perspecta 0.693598 0.0512 0.25010 NaN 192 2021-03-29T15:10:02 2021-03-29T15:02:17 None None
Perspecta 0.267218 0.2168 0.07877 NaN 193 2021-04-02T22:10:02 2021-04-02T22:04:56 None None
Perspecta 0.292080 0.2282 0.08102 NaN 200 2021-04-06T17:00:02 2021-04-06T17:00:02 None None
Perspecta 0.693147 0.0000 0.25000 NaN 188 2021-04-07T18:20:02 2021-04-07T18:11:08 :No Results: :Copy out:
Perspecta 0.232815 0.0809 0.05098 NaN 190 2021-04-07T20:00:02 2021-04-07T19:54:23 None None
Perspecta 0.513821 0.9378 0.09900 NaN 197 2021-04-10T22:50:02 2021-04-10T22:47:13 None None
Perspecta 1.216403 2.2507 0.10007 NaN 194 2021-04-12T15:40:02 2021-04-12T15:34:19 None None
Perspecta 0.232448 0.3822 0.07835 NaN 212 2021-04-13T14:30:01 2021-04-13T14:21:54 None None
Perspecta 0.386232 0.4570 0.14368 NaN 208 2021-04-14T21:00:02 2021-04-14T20:54:55 None None
Perspecta 0.173062 0.2188 0.05584 NaN 287 2021-04-14T23:10:01 2021-04-14T23:06:00 None None
Perspecta 0.693147 0.0000 0.25000 NaN 147 2021-04-19T18:20:02 2021-04-19T18:13:29 :No Results: :Copy out:
Perspecta 0.212172 0.3266 0.07104 NaN 271 2021-04-19T19:20:01 2021-04-19T19:12:21 None None
Perspecta-IUB 0.570474 0.4495 0.20375 NaN 204 2021-04-06T16:50:01 2021-04-06T16:40:25 None None
Perspecta-IUB 0.693147 0.0000 0.25000 NaN 140 2021-04-18T18:20:02 2021-04-18T18:11:08 :No Results: :Copy out:
Perspecta-IUB 0.073570 0.0969 0.01827 NaN 193 2021-04-18T19:40:02 2021-04-18T19:34:43 None None
TrinitySRITrojAI-SBU 0.693147 0.0000 0.25000 NaN 159 2021-04-09T17:10:02 2021-04-09T17:09:00 :No Results: :Copy out:
TrinitySRITrojAI-SBU 0.693147 0.0000 0.25000 NaN 153 2021-04-09T18:30:01 2021-04-09T18:25:56 :No Results: :Copy out:
TrinitySRITrojAI-SBU 0.693147 0.0000 0.25000 NaN 157 2021-04-12T16:40:02 2021-04-12T16:33:59 :No Results: :Copy out:
TrinitySRITrojAI-SBU 0.693147 0.0000 0.25000 NaN 156 2021-04-12T17:40:02 2021-04-12T17:30:42 :No Results: :Copy out:
TrinitySRITrojAI-SBU 0.693147 0.0000 0.25000 NaN 168 2021-04-12T18:50:02 2021-04-12T18:44:12 :No Results: :Copy out:
TrinitySRITrojAI-SBU 0.637458 0.1660 0.22564 NaN 170 2021-04-12T20:00:01 2021-04-12T19:56:08 None None
TrinitySRITrojAI-SBU 0.658262 0.1671 0.23609 NaN 165 2021-04-13T02:20:02 2021-04-13T02:13:34 None None
TrinitySRITrojAI-SBU 9.022712 7.6046 0.48433 NaN 160 2021-04-13T07:10:02 2021-04-13T07:06:42 None None
TrinitySRITrojAI-SBU 0.693147 0.0000 0.25000 NaN 162 2021-04-13T18:20:01 2021-04-13T18:10:16 :No Results: :Copy out:
TrinitySRITrojAI-SBU 0.121134 0.0536 0.01777 NaN 171 2021-04-13T19:10:02 2021-04-13T19:07:34 None None
TrinitySRITrojAI-SBU 9.883552 7.7606 0.40000 NaN 154 2021-04-14T00:50:02 2021-04-14T00:41:02 None None
TrinitySRITrojAI-SBU 0.954023 0.1492 0.37280 NaN 150 2021-04-19T20:20:01 2021-04-19T20:19:09 None None
PL-GIFT 0.693147 0.0000 0.25000 NaN 62 2021-04-16T15:00:02 2021-04-16T14:59:28 :No Results: :Copy out:
PL-GIFT 0.543538 0.3596 0.17888 NaN 180 2021-04-16T16:30:02 2021-04-16T16:28:09 None None

Status and Error Codes

Messages about the status of submissions that may be awaiting execution.

Code Description
None No jobs are submitted.
Queued Job has been queued for processing.
Awaiting Timeout Job is submitted, but must wait for time until next execution.
Pending Job is in processing queue and is pending availability of system resources.
Running Job is running
Disabled No longer accepting jobs from the team.

Messages about the status of files that are shared with the TrojAI google drive.

Code Description
None No shared files found in TrojAI google drive.
Multiple files submitted Team has more than one file shared with the TrojAI google drive account per evaluation server. You can share one file which starts with 'test' (for STS) and one which does not (for ES). Unshare/Delete shared files until only one is shared per server.
Ok Found one file shared with TrojAI google drive per server.

Error codes during processing result and metadata after a submission has completed.

Code Description
None No errors found while parsing the results of the submission.
:Result Parse: Unable to parse one ore more result files generated by the submission container.
:No Results: Unable to find and parse any results generated by the container.
:Executed File Update: Unable to update the submission metadata to reflect the Drive File which was actually executed, if different from the one initially submitted.
:Log File Missing: Unable to find the log file from the container execution.
:Info File Missing: Unable to find the job information file from the container execution.
:Confusion File Missing: Unable to find the confusion matrix file from the container execution.
:File Upload: Unable to upload a file from the test server to Drive.

Error codes during execution of a submission.

Code Description
None No errors found while launching the submission.
:Slurm Script Error: There was an error when submitting to the processing queue (Slurm may be offline)
Hypervisor offline Unable to establish connection to the virtual machine hypervisor.
:GPU: Failed to communicate with the GPU within the VM
:Copy in: Failed to copy in the actor-shared file into the VM.
:Timeout: The actor's execution timed out. Failed to finish processing all input models within allocated time.
:Execute: Other errors during execution of actor execution. (check slurm log file)
:Copy out: Failed to copy out results (they may not exist).
:Shutdown: Failed to shutdown the VM.
:VM: Issue with the VM resource (major error).

Disclaimer

NIST-developed software is provided by NIST as a public service. You may use, copy and distribute copies of the software in any medium, provided that you keep intact this entire notice. You may improve, modify and create derivative works of the software or any portion of the software, and you may copy and distribute such modifications or works. Modified works should carry a notice stating that you changed the software and should note the date and nature of any such change. Please explicitly acknowledge the National Institute of Standards and Technology as the source of the software.

NIST-developed software is expressly provided "AS IS." NIST MAKES NO WARRANTY OF ANY KIND, EXPRESS, IMPLIED, IN FACT OR ARISING BY OPERATION OF LAW, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT AND DATA ACCURACY. NIST NEITHER REPRESENTS NOR WARRANTS THAT THE OPERATION OF THE SOFTWARE WILL BE UNINTERRUPTED OR ERROR-FREE, OR THAT ANY DEFECTS WILL BE CORRECTED. NIST DOES NOT WARRANT OR MAKE ANY REPRESENTATIONS REGARDING THE USE OF THE SOFTWARE OR THE RESULTS THEREOF, INCLUDING BUT NOT LIMITED TO THE CORRECTNESS, ACCURACY, RELIABILITY, OR USEFULNESS OF THE SOFTWARE.

You are solely responsible for determining the appropriateness of using and distributing the software and you assume all risks associated with its use, including but not limited to the risks and costs of program errors, compliance with applicable laws, damage to or loss of data, programs or equipment, and the unavailability or interruption of operation. This software is not intended to be used in any situation where a failure could cause risk of injury or damage to property. The software developed by NIST employees is not subject to copyright protection within the United States.