Face Analysis Technology Evaluation (FATE) Quality
Quality Specific Image Defect Detection (SIDD):
Report |
API and Concept Document |
Validation |
Encryption | Submit
Quality Summarization:
Draft Report |
Participation Agreement |
Concept Document |
API
This track is open; developers should wait four months after their last successful submission to submit a new submission.
Status
[2024-11-15] A new version of the FATE SIDD Report is now available. It introduces cumulative distribution plots by region-of-birth and sex, as well as demographic analysis for Unified Quality Score.
[2024-10-04] A new version of the FATE SIDD Report is now available. It includes results for PAPIL11.
[2024-09-06] A new version of the FATE SIDD Report is now available. It includes results for Ediffiqal and IGD. We have also added summary tables for Eyes Open 2 and Mouth Open 2.
[2024-08-08] A new version of the FATE SIDD Report is now available. It includes results for Mobbeel and Veridium.
[2024-07-03] A new version of the FATE SIDD Report is now available. It includes results for neurotechnology, qazsmartvisionai, and idemia.
We have added a section on demographics to address two topics: thresholds on quality measures, and demographic variability in quality measures.
Previous versions of the SIDD report can be found on our GitHub page.
[2024-05-13] A new version of the FATE SIDD Report is now available. It includes results for cu-face.
[2024-04-26] A new version of the FATE SIDD Report is now available. It includes results for two new submissions from FRP Kauai and Secunet.
The Quality Algorithm Performance leaderboard below has been updated with algorithms that implemented Unified Quality Score.
The Fate SIDD API Concept Document was updated on 4-23-2024 to be in accordance with ISO/IEC DIS 29794-5.
[2024-04-16] A new version of the FATE SIDD Report is now available. It includes results for two new submissions from IGD and ROC.
The Quality Algorithm Performance leaderboard below has been updated with algorithms that implemented Unified Quality Score.
Previous versions of the SIDD report can be found on our GitHub page.
[2024-04-05] A new version of the FATE SIDD API Concept Document is now posted. It sets a four-month period between submissions on slide 4 and specifies that the sign convention for pose estimates on slide 10 is the same as that in ISO/IEC 39794-5.
Previous versions of the API Concept Document can be found on our GitHub page.
[2024-03-29] A new version of the FATE SIDD Report is now available. It includes a correction to Pitch Set 2 results due to a systematic offset in the determination of ground truth for Pitch Set 2.
Previous versions of the SIDD report can be found on our GitHub page.
[2024-03-08] A new version of the FATE SIDD Report is now available. It includes results for one new submission from Mobbeel.
The Quality Algorithm Performance leaderboard below has been updated with algorithms that implemented Quality SIDD Unified Quality Score.
[2024-02-14] A new version of the FATE SIDD Report is now available. It includes results for two new submissions from ROC and Viante.
Previous versions of the SIDD report can be found on our GitHub page.
The Quality Algorithm Performance leaderboard below has been updated with algorithms that implemented Quality SIDD Unified Quality Score.
[2024-02-02] A new version of the FATE SIDD Report is now available. It includes results for a new submission from IGD.
Previous versions of the SIDD report can be found on our GitHub page.
[2023-12-29] A new version of the FATE SIDD Report is now available. It includes results for two new submissions from Datech and Vsoft.
Previous versions of the SIDD report can be found on our GitHub page.
[2023-12-15] A new version of the FATE SIDD Report is now available. It includes results for two new submissions from Seamfix and Neurotechnology. In addition, we have made a correction to the Unified Quality Score plot for Digidata, and introduced a set for inter-eye distance with non-zero yaw.
Previous versions of the SIDD report can be found on our GitHub page.
[2023-12-04] A new version of the FATE SIDD API Concept Document is now posted. It includes updated numbering for the ISO/IEC 29794-5 quality checks on slides 30 and 33, and current participation instructions on slide 4.
Previous versions of the API Concept Document can be found on our GitHub page.
[2023-12-01] A new version of the FATE SIDD Report is now available. It includes results for two new submissions from Secunet. The results for Eyes Open 2, Mouth Open 2, and Face Occlusion 2 measures are included. From this date forward, the Eyes Open, Mouth Open, and Face Occlusion measures are no longer being evaluated.
The quality leaderboard has been updated with algorithms that implemented Quality SIDD Unified Quality Score.
[2023-11-16] A new version of the FATE SIDD API Concept Document is now posted. We are planning to deprecate EyesOpen, MouthOpen, and FaceOcclusion and have introduced EyesOpen2, MouthOpen2, and FaceOcclusion2. The new measures are aligned with ISO/IEC 29794-5 CD3.
Developers may continue to implement EyesOpen, MouthOpen, and FaceOcclusion until November 30, 2023, after which those measures will no longer be evaluated. The new measures are now present in the API and can be implemented.
Previous versions of the API Concept Document can be found on our GitHub page.
[2023-11-06] A new version of the FATE SIDD Report is now available. It includes results for new submissions from October 2023. We have also changed median absolute error to mean absolute error and introduced two sets for manually determined pitch and yaw.
The quality leaderboard has been updated with algorithms that implemented Quality SIDD Unified Quality Score.[2023-10-11] We are transitioning to evaluating Quality Summarization through only the “UnifiedQualityScore” component of Quality SIDD. Going forward, developers seeking to have their Quality Summarization algorithm evaluated should implement the Unified Quality Score component and submit to the Quality SIDD evaluation. The quality leaderboard has been updated to merge the results from the original Quality Summarization track with those from the Quality SIDD evaluation of Unified Quality Score.
[2023-09-20]
We have released NISTIR 8485 - Face Analysis Technology Evaluation (FATE) Part 11: Face Image Quality Vector Assessment: Specific Image Defect Detection. As described in this press release, the report summarizes accuracy for seven submissions from five unique developers. We evaluated algorithms across 20 quality measures.
This report will be discussed at the November 2023 Face Image Quality Workshop.
[2023-08-18]
A revision of the API and concept document for the Specific Image Defect Detection track has been posted
here.
We have updated references to FRVT to Face Analysis Technology Evaluation (FATE) and Face Recognition Technology Evaluation (FRTE).
Previous versions of the API document can be found on the GitHub page.
[2023-08-09]
A revision of the API and concept document for the Specific Image Defect Detection track has been posted
here.
References to FRVT have been changed to FATE to reflect the use of face analysis, rather than face recognition.
Previous versions of the API document can be found on the GitHub page.
[2023-07-03] Starting today, all algorithms, participation agreements, and GPG keys should be submitted using the Submission Form.
Quality Algorithm Performance
Background:
Quality assessment algorithms (QAA) are evaluated on their ability to assign low quality scores to
border crossing images that are variously of non-ideal pose, illumination and resolution.
QAAs can make two kinds or error: false rejection – saying an image is poor when it is
not, which drives costs; false acceptance – saying an image is good when it is
not, which drives future recognition errors. This approach of requiring
image quality to predict recognition failure has been the basis for
development of NFIQ
for fingerprints, which uses machine learning to map measurements from images to recognition scores.
Results: [last updated: 2024-10-02] The table shows how effective QAAs are at predicting recognition failure. We set a recognition threshold such that the false non-match rate (FNMR) from a set of 15 accurate FRTE verification algorithms is 1%. We then recompute the FNMR after discarding one and five percent of the lowest quality samples, as assessed by the listed algorithms. Ideally, discarding 1% of values would yield FNMR = 0, corresponding to perfect prediction. In practice, however, the QAA does not always assign low quality values to images involved in false non-matches.
The figures below show a more complete view of the table above: They show FNMR gains as a function of the fraction of lowest quality images discarded, and also considers four initial FNMR values: 0.5%, 1%, 2%, and 5%. It is evident that the algorithms are more effective when detecting the least recognizable images i.e. when the initial FNMR is set to 0.5%. If, instead, we ask the algorithms to detect the worst 5% of images, the relative FNMR gains are reduced.
Ground truth for quality is set as the false negatives from 15 of the more accurate recognition algorithms, one per developer. Mate scores are from comparison of high quality visa-like application photos with medium quality airport arrival webcam photos. Quality is computed only on the webcam photos.The dotted line gives either half the initial FNMR, or the lowest observed value. A steeply declining curve connotes a better QA.
Background
Face recognition accuracy has improved markedly due to development of new
recognition algorithms and approaches. Nevertheless, recognition error rates
remain significantly above zero, particularly in applications where photography
of faces is difficult or when stringent thresholds must be applied to
recognition outcomes to reduce false positives. For those applications that
retain an image as an authoritative reference sample against which future
recognitions are done, it is critical to maintain database quality. To that
end, quality assessment tools are applicable:
- Quality scalar: An image is converted into a quality value. Low values might be used to trigger collection of a new image, or to switch to an alternative modality. Quality scalars are useful also as a survey statistic. When computed over a large collection, quality statistics can reveal diurnal, seasonal variations, trends, or collection site variability for example.
- Quality vector: An image is analyzed and properties related to face recognition failure are reported. These quantify imaging-related properties such as focus, illumination, distortion, and noise, and also subject-related properties like head-pose, facial expression, and eyeglasses effects. Such tools are useful for providing actionable feedback to a user, or photographer.
The defining properties of a face quality scalar are described in the Quality Concept Document linked above. The document targets the ISO/IEC/ICAO specifications of a full frontal face as the reference against which quality must be assessed.
Standards
ISO/IEC JTC 1 SC37, the committee for standardization in biometrics, has two
projects related to improving face image quality assessment:
- ISO/IEC 29794-5: definitions of tests for a face image quality vector [Status: Draft International Standard (DIS) 2024-01-22]
- ISO/IEC 24358: specifications for face-aware capture subsystems [Status: Working draft]
How to Participate
To participate in this evaluation, developers need to submit a participation
agreement to NIST, wrap their software behind the published C++ API, run their
libraries through the provided validation package (which creates a submission
package), encrypt the package, and provide a download link for the encrypted
submission package. More details are provided below.
Participation Agreement
FATE is conducted by NIST, an agency of the United States Government. Participation is free of charge. FATE is open to a global audience of face recognition developers. All organizations who seek to participate in FATE must sign all pages of this Participation Agreement and submit it with their algorithm submission using the FRTE Submission Form. [last update: 2023-07-03]
[NOTE:] Organizations that have already submitted a participation agreement for FRTE Ongoing 1:1 do not need to send in a new participation agreement UNLESS the organization updates their cryptographic signing key.
API Document
A definitive API document has been published. All FATE APIs reference the supporting FRTE/FATE General Evaluation Specifications, which includes hardware and operating system environment, software requirements, reporting, and common data structures that support the APIs. All algorithms submitted must adhere to the published C++ API. [last update: 2023-04-06]
Validation
A validation package has been published. All participants must run their software through the updated validation package prior to submission. The purpose of validation is to ensure consistent algorithm output between your execution and NIST’s execution. [last update: 2023-04-28]
Encryption
All submissions must be properly encrypted and signed before transmission to NIST. This must be done according to these instructions using the FRTE/FATE Ongoing public key linked from this page. Participants must email their public key to NIST. The participant’s public key must correspond to the participant’s public-key fingerprint provided on the signed Participation Application. [last update: 2019-04-23]
Submission
All algorithm submissions must be submitted through the Submission Form, which requires encrypted files be provided as a download link from a generic http server (e.g., Google Drive). We cannot accept Dropbox links. NIST will not register, or establish any kind of membership, on the provided website. Participants can submit their algorithm(s), participation agreement, and GPG key at the same time via the submission form. [last update: 2023-07-03]
Participants must subscribe to the FRVT mailing list to receive emails when new reports are published or announcements are made.
Contact Information
Inquiries and comments may be submitted to frvt@nist.gov.
Subscribe to the FRVT mailing list to receive emails when announcements or updates are made.