Face Analysis Technology Evaluation (FATE) Age Estimation & Verification
Test Plan and API | Latest Report | Participation Agreement | Validation | Encryption | Submit
Status
The FATE AEV track is open. Developers should
submit no more frequently than every four months, and use the
FATE Submission Form to do so.
[2024-05-28] The first public report from the FATE Age Estimation and Verification (AEV) track has been published as NIST Interagency Report 8525. This will be updated on a roughly monthly basis as new algorithms are submitted, new datasets added, and new analyses performed.
[2014-03-31] NIST published its first evaluation of AE performance as NISTIR 7995 on Performance of Automated Age Estimation.
Overview
Facial age verification has recently been mandated in legislation in a number
of jurisdictions. These laws are typically intended to protect minors from
various harms by verifying that the individual is above a certain age, within
an age range, and, less commonly below a certain age. Other use-cases seek
only to determine actual age. While the mechanism for estimating age is usually
not specified in legislation, software-based face analysis is an attractive
approach when a photograph can be captured.
FATE AEV is an ongoing evaluation of software algorithms that inspect photos of a face to produce an age estimate. The output is set of reports on the accuracy and computational efficiency of algorithms. AEV is open to a worldwide community of developers. This evaluation will remain open indefinitely as a facility for developers to submit their algorithms whenever they are ready, but no more frequently than four calendar months.
AE Performance
Results: [last updated: 2024-11-08]
MAE By Dataset and Resources
- Mean Absolute Error (MAE) values are averages of 26 MAE estimates from two sexes and 13 ages (18 to 30).
-
- Algorithm name
- Submission data
- MAE for frontal view, no-glasses, office application photos.
- MAE for airport concourse border crossing photos featuring variable pose, illumination and eye glasses.
- MAE for good quality mugshot portraits collected with two diffuse light sources and a uniform background.
- Resource Usage:
-
- AE Time (msec) is the median duration of the AE function call.
- Config (MB) is the total size of the neural network model and configuration data of the algorithm.
- Lib (MB) is the total size of the implemenation’s compiled libraries.
Challenge-25 &
Child Online Safety (13-16)
Challenge-25 and Child Online Safety (13-16)
The table shows accuracy measures for 2 common applications, Challenge-25 age-restriction and children age 13-16 online chatroom access. False positive rates quantify how often people outside the allowed age range are estimated to be within required category. True positive rates quantify how often children within age 13-16 are estimated to be within that age range.
- Challenge 25:
-
- [Dataset: Application (good quality), Border (medium quality)]
- Application - False Positive Rate (FPR) values are averages of 12 FPR estimates from two sexes and six regions-of-birth for subjects aged 17.
- Border - False Positive Rate (FPR) values are averages of 12 FPR estimates from two sexes and six regions-of-birth for subjects aged 17.
- Child Online Safety (13-16):
-
- [Dataset: Visa (good quality)]
- Age 13-16 - Mean Absolute Errors (MAE) are averages of four MAE values for age group 13, 14, 15, and 16 (below 17).
- Age 13-16 - True Positive Rates (TPR) are proportions of subjects aged 13 to 16 and whose age is estimated from 13 to 16 (below 17).
- Age 8-12 - False Positive Rates (FPR) are proportions of subjects aged 8 to 12 but whose age is estimated from 13 to 16 (below 17).
- Age 17-22 - False Positive Rates (FPR) are proportions of subjects aged 17 to 22 but whose age is estimated from 13 to 16 (below 17).
MAE By Demographic Group
The table shows mean absolute error (MAE) for men and women born in six regions of the world. See the AEV report for discussion of how these regions were selected and grouped. Lower values of MAE indicate better accuracy. Low variation across columns indicates equitable accuracy across demographic groups. MAE is estimated over high quality visa-like immigration office application photos. The uncertainty estimates span 95% of bootstrap estimates of the mean. The table can be sorted on any column by clicking the small arrows in its header.
How to Participate
To participate in this evaluation, developers need to submit a participation
agreement to NIST, wrap their software behind the published C++ API, run their
libraries through the provided validation package (which creates a submission
package), encrypt the package, and provide a download link for the encrypted
submission package.
[Participation agreement] FATE is conducted by NIST, an agency of the United States Government. Participation is free of charge. FATE is open to a global audience of face recognition developers. All organizations who seek to participate in FATE must sign all pages of this Participation Agreement and submit it with their algorithm submission using the Submission Form. [last update: 2023-08-17]
[API] General and common information shared between all tracks of the FRTE/FATE evaluations are documented in a General Evaluation Specifications, which includes hardware and operating system environment, software requirements, reporting, and common data structures that support the APIs. [last update: 2023-08-17]
[Validation] A validation package has been published. All participants must run their software through the validation package prior to submission. The purpose of validation is to ensure consistent algorithm output between your execution and NIST’s execution. [last update: 2023-08-17]
[Encryption] All submissions must be properly encrypted and signed before transmission to NIST. This must be done according to these instructions using the FATE Ongoing public key linked from this page. Participants must email their public key to NIST. The participant’s public key must correspond to the participant’s public-key fingerprint provided on the signed Participation Application. [last update: 2022-07-03]
[Submission] All algorithm submissions must be submitted through the Submission Form, which requires encrypted files be provided as a download link from a generic http server (e.g., Google Drive). We cannot accept Dropbox links. NIST will not register, or establish any kind of membership, on the provided website. Participants can submit their algorithm(s), participation agreement, and GPG key at the same time via the submission form. [last update: 2023-07-03]
Participants must subscribe to the evaluation mailing list to receive emails when new reports are published or announcements are made.