FRVT Face Mask Effects

Status

[last updated: November 30, 2020]

NIST has published NISTIR 8331 - Ongoing FRVT Part 6B: Face recognition accuracy with face masks using post-COVID-19 algorithms, the second out of a series of reports aimed at quantifying face recognition accuracy for people wearing masks. This report adds

  • 65 new algorithms submitted to FRVT 1:1 since mid-March 2020 (and includes cumulative results for 152 algorithms evaluated to date)

  • assessment of when both the enrollment and verification images are masked (in addition to when only the verification image is masked)

Our initial approach has been to apply masks to faces digitally (i.e., using software to apply a synthetic mask). This allowed us to leverage large datasets that we already have. This report quantifies the effect of masks on both false negative and false positives match rates.

Results

[last updated: November 30, 2020]

The accuracy table currently shows the top performing 1:1 algorithms evaluated on masked images. Results are tabulated on the VISABORDER dataset where the probe image is masked (and the enrollment image remains unmasked). It also includes baseline FNMR when both images are not masked. FNMR values are stated at a fixed threshold calibrated to give FMR = 0.00001 on unmasked images. FNMR is the proportion of mated comparisons below a threshold set to achieve the false match rate (FMR) specified. FMR is the proportion of impostor comparisons at or above that threshold.

The results table will be updated on a monthly basis as algorithms and computations complete, as datasets are added, and as new analyses are developed.

Reports

All Ongoing FRVT Face Mask Effects reports can be accessed from here.

Motivation

Traditionally, face recognition systems (in cooperative settings) are presented with mostly non-occluded faces, which include primary facial features such as the eyes, nose, and mouth. However, there are a number of circumstances in which faces are occluded by masks such as in pandemics, medical settings, excessive pollution, or laboratories. As of late, the widespread requirement that people wear face masks in public places has driven a need to understand how cooperative face recognition technology deals with occluded faces, often with just the periocular area and above visible. There has been an increasing number of providers that have developed “face mask capable” face recognition capabilities, and therefore, has driven the need to quantify the performance of modern face recognition on masked faces.

How to Participate

Currently, all algorithms submitted to the Ongoing FRVT 1:1 Verification test will be evaluated against both unmasked and masked dataset(s).

The idea is that developers should submit an integrated package that:

  • On an unmasked image, extracts features from both the full face and the periocular region
  • On a masked image, extracts features from the periocular region
  • Compares any combination of unmasked and masked faces

At least one developer has done this recently, with FMR approximately constant for any combination.

The test is designed this way to mimic operational reality: some images will have masks, some will not (especially enrollment samples from a database or ID card).

Please refer to the FRVT 1:1 webpage for information on how to participate.

Contact Information

Inquiries and comments may be submitted to frvt@nist.gov.

Subscribe to the FRVT mailing list to receive emails when announcements or updates are made.