Video LINCS Leaderboards
The goal of the Video LINCS program, is to research and develop novel capabilities to autonomously re-identify objects across diverse video sensor collections and map all objects to a common reference frame. Re-identification (reID) is the process of matching the same object across a video collection, to determine where the object appears throughout the video. Video LINCS will research approaches that will facilitate autonomous reID in an open-world setting where there is no advance knowledge of the sensors, scene, content, or video collection geometries. ReID technologies will initially be developed for specific object classes that are known in advance, such as people and vehicles, and ultimately extend to all objects in the video footage without advance knowledge of specific object types. The capability to autonomously remap object locations from individual camera reference frames to a single common reference frame, nominally a geo-reference frame (geo-localization), will also be developed. Read more about the problem. Get started with a free minimal example. Submission container names must use the following format: "<Leaderboard Name>_<Dataset Name>_<Submission Name>.zip"Reload page to update tables. Content is pushed every 10 minutes. Timestamps are presented in UTC.
Evaluation Server Status
Nodes: idle; running; down.
Accepting Submissions:
Smoke Test Server Status
Nodes: idle; running; down.
Accepting Submissions:
Placeholder text for testing-nov2024
Placeholder image description
test: The test dataset that is sequestered/hidden, used for evaluation. Submissions here should be fully realized with complete schema and parameters.
Required filename format: "testing-nov2024_test_<Submission Name>.zip"
Accepting submissions: True
Dataset size in testing-nov2024, test: 1
Execution timeout (hh:mm:ss): 0:10:00
Teams/Jobs
| Team | Submission Timestamp | Type | Job Status | File Status | General Status | File Timestamp | Time until next execution |
|---|---|---|---|---|---|---|---|
| VLINCS_Test | 2024-12-12T18:02:37 | public | None | Ok | Ok | 2024-12-09T19:00:00 | 0 d, 0 h, 0 m, 0 s |
Best Results based on IDF1
| Team | MOTA | IDF1 | HOTA | MT | ML | FP | FN | Rcll | Prcn | AssA | DetA | AssRe | AssPr | DetRe | DetPr | LocA | FP_Frame | Frag | Runtime (s) | Submission Timestamp | File Timestamp | Leaderboard Revision | Parsing Errors | Launch Errors |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| VLINCS_Test | 1.0 | 1.0 | 1.0 | 2.0 | 0.0 | 0.0 | 0.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.0 | 1.0 | 8.73 | 2024-12-12T18:02:37 | 2024-12-09T19:00:00 | Rev1 | None | None |
All Results
| Team | MOTA | IDF1 | HOTA | MT | ML | FP | FN | Rcll | Prcn | AssA | DetA | AssRe | AssPr | DetRe | DetPr | LocA | FP_Frame | Frag | Runtime (s) | Submission Timestamp | File Timestamp | Leaderboard Revision | Parsing Errors | Launch Errors |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| VLINCS_Test | 1.0 | 1.0 | 1.0 | 2.0 | 0.0 | 0.0 | 0.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.0 | 1.0 | 8.73 | 2024-12-12T18:02:37 | 2024-12-09T19:00:00 | Rev1 | None | None |
Status and Error Codes
Messages about the status of submissions that may be awaiting execution.
| Code | Description | |
|---|---|---|
| None | No jobs are submitted. | |
| Queued | Job has been queued for processing. | |
| Awaiting Timeout | Job is submitted, but must wait for time until next execution. | |
| Pending | Job is in processing queue and is pending availability of system resources. | |
| Running | Job is running | |
| Disabled | No longer accepting jobs from the team. | |
Messages about the status of files that are shared with the VLINCS google drive.
| Code | Description | |
|---|---|---|
| None | No shared files found in VLINCS google drive. | |
| Multiple Files Shared | Team has more than one file shared with the VLINCS google drive account per evaluation server. You can share one file which starts with 'test' (for STS) and one which does not (for ES). Unshare/Delete shared files until only one is shared per server. | |
| Ok | Found one file shared with VLINCS google drive per server. | |
Messages about the general status of submissions (global across all leaderboards).
| Code | Description | |
|---|---|---|
| Shared File Error | Team has an issue with one or more of the shared files. "Format" indicates incorrect file name, should be "leaderboard_name-data_split_name". "Leaderboard name" indicates invalid leaderboard name. "Data split name" indicates invalid data split name." | |
| Ok | All files shared have no issues. | |
Error codes during processing result and metadata after a submission has completed.
| Code | Description | |
|---|---|---|
| None | No errors found while parsing the results of the submission. | |
| :Result Parse: | Unable to parse one ore more result files generated by the submission container. | |
| :No Results: | Unable to find and parse any results generated by the container. | |
| :Missing Results: | One or more results expected to be generated by the container are missing. | |
| :Executed File Update: | Unable to update the submission metadata to reflect the Drive File which was actually executed, if different from the one initially submitted. | |
| :Log File Missing: | Unable to find the log file from the container execution. | |
| :Info File Missing: | Unable to find the job information file from the container execution. | |
| :Confusion File Missing: | Unable to find the confusion matrix file from the container execution. | |
| :File Upload: | Unable to upload a file from the test server to Drive. | |
Error codes during execution of a submission.
| Code | Description | |
|---|---|---|
| None | No errors found while launching the submission. | |
| :Container Parameters: | There was an error with the container, one or more required parameters may not be valid or not exist. | |
| :Schema Header: | There was an error with the container's schema header. Please update your schema to have appropriate title, technique, technique_description, technique_changes, commit_id, and repo_name. | |
| :Slurm Script Error: | There was an error when submitting to the processing queue (Slurm may be offline) | |
| Hypervisor Offline | Unable to establish connection to the virtual machine hypervisor. | |
| :GPU: | Failed to communicate with the GPU within the VM | |
| :Copy In: | Failed to copy in the actor-shared file into the VM. | |
| :Timeout: | The actor's execution timed out. Failed to finish processing all input models within allocated time. | |
| :Execute: | Other errors during execution of actor execution. (check slurm log file) | |
| :Copy Out: | Failed to copy out results (they may not exist). | |
| :Shutdown: | Failed to shutdown the VM. | |
| :VM: | Issue with the VM resource (major error). | |
Disclaimer
NIST-developed software is provided by NIST as a public service. You may use, copy and distribute copies of the software in any medium, provided that you keep intact this entire notice. You may improve, modify and create derivative works of the software or any portion of the software, and you may copy and distribute such modifications or works. Modified works should carry a notice stating that you changed the software and should note the date and nature of any such change. Please explicitly acknowledge the National Institute of Standards and Technology as the source of the software.
NIST-developed software is expressly provided "AS IS." NIST MAKES NO WARRANTY OF ANY KIND, EXPRESS, IMPLIED, IN FACT OR ARISING BY OPERATION OF LAW, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT AND DATA ACCURACY. NIST NEITHER REPRESENTS NOR WARRANTS THAT THE OPERATION OF THE SOFTWARE WILL BE UNINTERRUPTED OR ERROR-FREE, OR THAT ANY DEFECTS WILL BE CORRECTED. NIST DOES NOT WARRANT OR MAKE ANY REPRESENTATIONS REGARDING THE USE OF THE SOFTWARE OR THE RESULTS THEREOF, INCLUDING BUT NOT LIMITED TO THE CORRECTNESS, ACCURACY, RELIABILITY, OR USEFULNESS OF THE SOFTWARE.
You are solely responsible for determining the appropriateness of using and distributing the software and you assume all risks associated with its use, including but not limited to the risks and costs of program errors, compliance with applicable laws, damage to or loss of data, programs or equipment, and the unavailability or interruption of operation. This software is not intended to be used in any situation where a failure could cause risk of injury or damage to property. The software developed by NIST employees is not subject to copyright protection within the United States.