Face Recognition Technology Evaluation (FRTE) Twins Demonstration
Report - NISTIR 8439 | Participation Agreement | Concept Document | API | Validation | Encryption | Submit
Status
[2023-07-03] Starting today, all algorithms, participation agreements, and GPG keys should be submitted using the Submission Form.
[2023-03-30] This is the first FRTE Twins Demonstration results. The table below documents the performance for two algorithms received in February 2023.[2022-10-13] Validation package has been published on GitHub, all participants must run their software through the validation package prior to submission.
[2022-09-27] NIST Interagency Report 8439: FRVT Part 9A: Face Recognition Verification Accuracy on Distinguishing Twins
[2022-09-27] Submission starts October 31, 2022.
Purpose
Many face recognition algorithms produce high false match rates (FMR) when comparing images of identical and same-sex
fraternal twins. FMR is lower when comparing images of two unrelated persons of the same sex and age.
As detailed in the concept document,
the FRTE Twins Demonstration track is an ongoing opportunity for developers to demonstrate an ability to distinguish between twins.
Our quantitative goal is to state a false match rate (FMR) on twins that is a small multiple of FMR on unrelated persons. This should be achieved at decision thresholds (T) for which FMRUNRELATED ≤ 10-4, as is typical in one-to-one applications.
Results
[last updated: 2023-03-30]
Accuracy
Verification Performance
The accuracy table shows the algorithms False Match Rate (FMR) on twins probe images. FMR is the proportion of twin comparisons at or above a threshold score set to achieve FMR=0.0001 on Mugshot images. The Mugshot False Non-Match Rate (FNMR) is included to show the performance of the algorithms when mated comparisons are below a threshold set to achieve the same FMR=0.0001. Mugshot is used as a control for Twins Days data.
Resources
Resources Performance
Algorithms submitted to FRTE implement NIST’s application programming interface (API). We measure the duration of all function calls using the C++ std::chrono::high resolution clock on an unloaded server-class machine Xeon® Gold 6140 CPU @ 2.30GHz. The table below includes durations of the template generation.
How to Participate
To participate in this evaluation, developers need to submit a participation
agreement to NIST, wrap their software behind the published C++ API, run their
libraries through the provided validation package (which creates a submission
package), encrypt the package, and provide a download link for the encrypted
submission package.
[Participation agreement] FRTE is conducted by NIST, an agency of the United States Government. Participation is free of charge. FRTE is open to a global audience of face recognition developers. All organizations who seek to participate in FRTE must sign all pages of this Participation Agreement and submit it with their algorithm submission using the FRVT Submission Form. [last update: 2023-07-03]
[NOTE:] Organizations that have already submitted to FRTE 1:1 do not need to send in a new participation agreement UNLESS the organization updates their cryptographic signing key.
[API] The demonstration track uses the identical API, rules, and participation processes of the FRTE 1:1 verification track. All algorithms submitted must adhere to the published C++ API. Developers must submit 1. A link to the encrypted compiled software implementation of NIST’s API, 2. The core library named as libfrvt_11_twins_<provider>_<sequence>.so, and 3. Send submission to the FRTE Twins Demonstration track using the Submission Form. [last update: 2023-08-18]
[Validation] A validation package has been published. All participants must run their software through the updated validation package prior to submission. The purpose of validation is to ensure that NIST can exactly reproduce developer-submitted algorithm outputs. [last update: 2023-04-28]
[Encryption] All submissions must be properly encrypted and signed before transmission to NIST. This must be done according to these instructions using the FRVT Ongoing public key linked from this page. Participants must email their public key to NIST. The participant’s public key must correspond to the participant’s public-key fingerprint provided on the signed Participation Application. [last update: 2022-08-22]
[Submission] All algorithm submissions must be submitted through the Submission Form, which requires encrypted files be provided as a download link from a generic http server (e.g., Google Drive). We cannot accept Dropbox links. NIST will not register, or establish any kind of membership, on the provided website. Participants can submit their algorithm(s), participation agreement, and GPG key at the same time via the submission form. [last update: 2023-07-03]
Participants must subscribe to the evaluation mailing list to receive emails when new reports are published or announcements are made.