Face Recognition Technology Evaluation (FRTE) Face Mask Effects
Status
[2024-02-22] NIST will discontinue running the FRTE face mask benchmark on 1:1 algorithms submitted to FRTE.
[2022-01-20]
NIST has published an update to NISTIR 8331 - Ongoing FRTE Part 6B: Face recognition accuracy with face masks using post-COVID-19 algorithms aimed at quantifying face recognition accuracy for people wearing masks. This report documents results from testing 266 face recognition algorithms provided to NIST since the onset of the pandemic in mid-March 2020, and includes cumulative results for 319 algorithms evaluated to date (submitted both prior to and after mid-March 2020).
Results
[last updated: 2024-03-28]
The accuracy table currently shows the top performing 1:1 algorithms evaluated on masked images. Results are tabulated on the VISABORDER dataset where the probe image is masked (and the enrollment image remains not masked). It also includes baseline FNMR when both images are not masked. FNMR values are stated at a fixed threshold calibrated to give FMR = 0.00001 on unmasked images. FNMR is the proportion of mated comparisons below a threshold set to achieve the false match rate (FMR) specified. FMR is the proportion of impostor comparisons at or above that threshold. The ratio between FNMR(MASKED) and FNMR(NOT MASKED) provides the factor difference in error rates between masked vs. not masked photos. The lower the ratio, the more similar performance is between masked vs. not masked photos.
The results table will be updated on a monthly basis as algorithms and computations complete, as datasets are added, and as new analyses are developed.
Reports
All Ongoing FRTE Face Mask Effects reports can be accessed from here.
Motivation
Traditionally, face recognition systems (in cooperative settings) are presented with mostly non-occluded faces, which include primary facial features such as the eyes, nose, and mouth. However, there are a number of circumstances in which faces are occluded by masks such as in pandemics, medical settings, excessive pollution, or laboratories. As of late, the widespread requirement that people wear face masks in public places has driven a need to understand how cooperative face recognition technology deals with occluded faces, often with just the periocular area and above visible.
There has been an increasing number of providers that have developed “face mask capable” face recognition capabilities, and therefore, has driven the need to quantify the performance of modern face recognition on masked faces.
How to Participate
Currently, all algorithms submitted to the Ongoing FRTE 1:1 Verification test will be evaluated against both unmasked and masked dataset(s).
The idea is that developers should submit an integrated package that:
- On an unmasked image, extracts features from both the full face and the periocular region
- On a masked image, extracts features from the periocular region
- Compares any combination of unmasked and masked faces
At least one developer has done this recently, with FMR approximately constant for any combination.
The test is designed this way to mimic operational reality: some images will have masks, some will not (especially enrollment samples from a database or ID card).
Please refer to the FRTE 1:1 webpage for information on how to participate.