Face Recognition Technology Evaluation (FRTE) 1:N Identification
Latest Report | Participation Agreement | API | Validation | Encryption | Submit
Status
[2024_11_08]
A new FRTE 1:N report has been published. Prior editions of the 1:N report are here.
NIST discontinued the paperless travel benchmark.
[2023-07-03] All algorithms, participation agreements, and GPG keys should be submitted using the FRTE Submission Form.
FRTE Participation Statistics
[last updated: 2024-11-08]
| 2018 | 2019 | 2020 | 2021 | 2022 | 2023 | Total | |
|---|---|---|---|---|---|---|---|
| Number of algorithms | 209 | 31 | 23 | 76 | 57 | 81 | 542 |
| Number of unique developers | 53 | 25 | 20 | 52 | 41 | 56 | 164 |
Performance
[last updated: 2024-11-08]
Identification (T>0)
by Developer
Identification Performance
The table below shows False Negative Identification Rates (FNIR) for the case where a threshold is set to limit to the False Positive Identification Rate (FPIR) to 0.003. FNIR is the proportion of mated searches failing to return the mate above threshold. FPIR is the proportion of non-mated searches producing one or more candidates above threshold. The threshold is set for each algorithm and each column separately. The use of thresholding supports use of face recognition in making mostly automated decisions e.g. for access into facility. The first row in the header shows the type of image enrolled in the gallery; the second row shows the search image type; the third row shows the number of persons in the gallery. The images are described in the section 2 of the report. In all cases, each person is enrolled with one image only.
The values in blue correspond to a change in the FRTE API on 2022-02-14 that allows the algorithm to detect and produce templates from multiple faces in one image, which occurs in approximately 3% of border images and 7% of kiosk images. The handling and accuracy consequences of this are detailed on this slide.
Investigation (R=1, T=0)
by Developer
Investigation Performance
The table below shows False Negative Identification Rates (FNIR) for the case where the threshold is set to zero and the algorithm returns a fixed number (50) of candidates. FNIR is the proportion of mated searches for which the algorithm does not place the correct candidate at rank 1. The use of face recognition without a threshold supports investigate uses where it is assumed and necessary that a human will be used to review the candidates returned from each search. For mated-searches the human is tasked with finding the correct mate; for non-mated searches the reviewer must reject all the candidates. The first row in the header shows the type of image enrolled in the gallery; the second row shows the search image type; the third row shows the number of persons in the gallery. In all cases, each person is enrolled with one image only.
Identification (T>0)
by Algorithm
Identification Performance
The table below shows False Negative Identification Rates (FNIR) for the case where a threshold is set to limit to the False Positive Identification Rate (FPIR) to 0.003. FNIR is the proportion of mated searches failing to return the mate above threshold. FPIR is the proportion of non-mated searches producing one or more candidates above threshold. The threshold is set for each algorithm and each column separately. The use of thresholding supports use of face recognition in making mostly automated decisions e.g. for access into facility. The first row in the header shows the type of image enrolled in the gallery; the second row shows the search image type; the third row shows the number of persons in the gallery. The images are described in the section 2 of the report. In all cases, each person is enrolled with one image only.
The values in blue correspond to a change in the FRTE API on 2022-02-14 that allows the algorithm to detect and produce templates from multiple faces in one image, which occurs in approximately 3% of border images and 7% of kiosk images. The handling and accuracy consequences of this are detailed on this slide.
Investigation (R=1, T=0)
by Algorithm
Investigation Performance
The table below shows False Negative Identification Rates (FNIR) for the case where the threshold is set to zero and the algorithm returns a fixed number (50) of candidates. FNIR is the proportion of mated searches for which the algorithm does not place the correct candidate at rank 1. The use of face recognition without a threshold supports investigate uses where it is assumed and necessary that a human will be used to review the candidates returned from each search. For mated-searches the human is tasked with finding the correct mate; for non-mated searches the reviewer must reject all the candidates. The first row in the header shows the type of image enrolled in the gallery; the second row shows the search image type; the third row shows the number of persons in the gallery. In all cases, each person is enrolled with one image only.
Resources
by Algorithm
Resources Performance
Algorithms submitted to FRTE implement NIST’s application programming interface (API). We measure the duration of all function calls using the C++ std::chrono::high resolution clock on an unloaded server-class machine. The table below includes durations of the template generation, finalization, search calls. In addition the size of the algorithm is reported in two parts: the recognition models, and the libraries.
Prior Editions of Report
All prior Ongoing FRTE 1:N reports can be accessed from here.
How to Participate
To participate in this evaluation, developers need to submit a participation agreement to NIST, wrap their software behind the published C++ API, run their libraries through the provided validation package (which creates a submission package), encrypt the package, and provide a download link for the encrypted submission package. More details are provided below.
Participation Agreement
FRTE is conducted by NIST, an agency of the United States Government. Participation is free of charge. FRTE is open to a global audience of face recognition developers. All organizations who seek to participate in FRTE must sign all pages of this Participation Agreement and submit it with their algorithm submission using the FRTE Submission Form. [last update: 2023-07-03]
API Document
A new API document has been published. All FRTE APIs reference the supporting FRTE General Evaluation Specifications, which includes hardware and operating system environment, software requirements, reporting, and common data structures that support the APIs. Developers must ensure that their submission conforms to the API specifications. [last update: 2023-04-06]
Validation
An updated validation package has been published. All participants must run their software through the validation package prior to submission. The purpose of validation is to ensure consistent algorithm output between your execution and NIST’s execution. [last update: 2023-04-28]
Encryption
All submissions must be properly encrypted and signed before transmission to NIST. This must be done according to these instructions using the FRTE Ongoing public key linked from this page. Participants must email their public key to NIST. The participant’s public key must correspond to the participant’s public-key fingerprint provided on the signed Participation Application. [last update: 2017-11-20]
Submission
All algorithm submissions must be submitted through the FRTE Submission Form, which requires encrypted files be provided as a download link from a generic http server (e.g., Google Drive). We cannot accept Dropbox links. NIST will not register, or establish any kind of membership, on the provided website. Participants can submit their algorithm(s), participation agreement, and GPG key at the same time via the submission form. [last update: 2023-07-03]
Participants must subscribe to the FRTE mailing list to receive emails when new reports are published or announcements are made.