Report Card For Neurotechnology (Dec 2023)
| Results Homepage | API & CONOPS Document | Participation Agreement | FAQ | [email protected] |
| Last Updated: July 23, 2024 |
Basic Performance Stats
Two Eyes:| Template Creation Time (s): | 0.5 ± 0.2 |
| Search Time (s): | 32.3 ± 0.6 |
| FNIR (@ FPIR=0.01): | 0.0029 ± 0.0006 (± 90% confidence) |
| FNIR (@ FPIR=0.001): | 0.0034 ± 0.0006 (± 90% confidence) |
| Miss Rate @ Rank 1: | 0.0024 |
| Miss Rate @ Rank 10: | 0.0022 |
| Miss Rate @ Rank 100: | 0.0019 |
| Failure to Enroll Rate: | 0 |
| Dataset: | Operational Dataset 4th pull (stats on OPS4 images) |
| Samples used: | Both eyes |
| Enrolled Population: | 500K people |
| Enrollment Method: | Both (left and right) iris images per enrollment template |
| Template Creation Time (s): | 0.5 ± 0.2 |
| Search Time (s): | 32.3 ± 0.6 |
| FNIR (@ FPIR=0.01): | 0.0098 ± 0.0008 (± 90% confidence) |
| FNIR (@ FPIR=0.001): | 0.0113 ± 0.0008 (± 90% confidence) |
| Miss Rate @ Rank 1: | 0.0083 |
| Miss Rate @ Rank 10: | 0.0072 |
| Miss Rate @ Rank 100: | 0.0049 |
| Failure to Enroll Rate: | 0 |
| Dataset: | Operational Dataset 4th pull (stats on OPS4 images) |
| Samples used: | Single eye |
| Enrolled Population: | 1M irides (500K people) |
| Enrollment Method: | One iris image per enrollment template |
DET Accuracy
Core accuracy for the identification task can be characterized by Detection-error trade-off (DET) plots. Generally, curves lower down in a DET plot correspond to more accurate matchers.
The plots are interactive through the use of the Plotly.js graphing library. Mousing over the DET plots will display a menu at the top right which provides the capability to manipulate the plot in various ways and to download the plot as a png or the underlying data as a csv. Double-clicking on an item in the legend will display only that item in the plot; single-clicking will add/remove the item from the plot.
Two-eye Accuracy
| Dataset: | Operational Dataset 4th pull (stats on OPS4 images) |
| Samples used: | Both eyes |
| Enrolled Population: | 500K people |
| Enrollment Method: | Both (left and right) iris images per enrollment template |
Single Eye Accuracy
| Dataset: | Operational Dataset 4th pull (stats on OPS4 images) |
| Samples used: | Single eye |
| Enrolled Population: | 1M irides (500K people) |
| Enrollment Method: | One iris image per enrollment template |
Ranked Accuracy
Two-eye Accuracy
| Dataset: | Operational Dataset 4th pull (stats on OPS4 images) |
| Samples used: | Both eyes |
| Enrolled Population: | 500K people |
| Enrollment Method: | Both (left and right) iris images per enrollment template |
Single-eye Accuracy
| Dataset: | Operational Dataset 4th pull (stats on OPS4 images) |
| Samples used: | Single eye |
| Enrolled Population: | 1M irides (500K people) |
| Enrollment Method: | One iris image per enrollment template |
Demographic Effects
The figure below shows the FNIR at FPIR=0.01 (t = 27) for different demographic groups. The bars show 95% confidence intervals.
| Dataset: | Operational Dataset 4th pull (stats on OPS4 images) |
| Samples used: | Both eyes |
| Enrolled Population: | 500K people |
| Enrollment Method: | Both (left and right) iris images per enrollment template |
In Ethnicity and Biometric Awareness: Iris Pattern Individuality in a West African Database, Daugman et. al. note that the darker irides of certain populations, such as Sub-Saharan Africans, appear to have a greater prevalence of coarser features (e.g., crypts, craters). They posit that these lower-frequency features carry less information entropy, potentially making their irides slightly more difficult to discriminate.
Some consolidation of demographic information was necessary to improve statistical power. Eye color was consolidated to either light (grey, blue, or green) or dark (brown or black). Some subjects were labeled as being neither male nor female. Meaningful results for these categories could not be obtained because their sample sizes are too small. For the same reason, results for races other than white and black are not shown. The precise definitions of race, sex, and eye color used here can be found in the FBI’s Biometric Specification documentation.
Logistic Regression
This section models the relationship between FNIR and various demographic characteristics using logistic regression. The response variable is whether the search produces a false negative at an FPIR of 0.01. The precise logit relationship is
where p is the probability of a false negative and ℓ is the log likelihood ratio of the probability of a false negative.
McFadden’s = 0.0000438
n = 311,452
Negative (blue) values mean the probability of a miss is decreased. McFadden’s pseudo is a measure of the goodness-of-fit that produces values between 0 and 1. Race, sex, and eye color are generally poor predictors of accuracy, so the value is typically low.
The model does not include any interactions between race and eye color because there were not enough cases of blacks with light eyes to produce meaningful results. Eye color was unavailable for some subjects so MICE was used to perform imputation.
Demographic breakdown of false positives
The figures below show the demographic composition of false positives. Most false positives are likely to be white and male since they comprise the majority of the test data. The precise demographic breakdown of the test data is- 0.685% White and male,
- 0.168% White and female,
- 0.128% Black and male,
- 0.019% Black and female,
- 0.855% Dark eyes,
- 0.145% Light eyes.
The breakdown of the search and enrollment sets are approximately the same. Other races, sexes, and eye colors are ignored due to the infrequency of their occurence in the test dataset.
Below each figure is a calculation of demographic fairness from FRVT: Summarizing Demographic Differentials. The calculations are not exact matches for Equations 10 and 11 in the report because the one-to-one error metrics FMR and FNMR are replaced with the one-to-many error metrics FPIR and FNIR respectively.
The demographic fairness metric quantifies inequitability across demographic groups. A value of \(1\) indicates that the error rate does not change for different demographic groups. Higher values indicate stronger demographic bias. In this section, the demographic groups are either eye color (\(n=2\) for light or dark), or race combined with sex (\(n=4\)).
Demographic Breakdown of Misses
NOTE: Plots may not render if the matcher produces highly discretized scores.
Twins Effect
Between 2010 and 2018, West Virginia University and the University of Notre Dame collected iris images of identical and mirror twins during the annual Twinsday Festival. The data collection procedure is described in Sabatier et. al.. Many twins participated in the data collection on multiple years. In all, \(5,078\) iris images from \(691\) twins were used to collect the results below.
The comparison scores were collected as follows: all available images were enrolled in a database; the same set of images were searched against the database producing a total of \(5,078 \times 5,078 = 25.7\) million scores, including \(72,587\) comparison scores involving twins, \(75,651\) contralateral (i.e. left-vs-right irises from the same person) comparison scores, and 25.5 million nonmated comparison scores involving different people. The scores are not truly one-to-one if the submission performs enrollment-side score or template normalization.
In Broken Symmetries, Random Morphogenesis, and Biometric Distance Daugman et. al. identify a small but statistically significant positive correlation in contralateral and twins scores for the traditional IrisCode matching algorithm. At the 2023 annual Iris Experts Group (IEG) Meeting, Daugman further notes that the correlation is much stronger for at least some of the newer deep-learning approaches to iris matching.
NOTE: Some plots may not render well if the matcher produces highly discretized scores.
Histograms of Score Distributions
Cumulative Score Distributions
Impact of Enrollment Size
Accuracy is impacted by the size of the enrollment database (a.k.a the gallery size). Identification of the correct mate is expected to be more difficult for larger enrollment database sizes. The figure below plots FNIR (at FPIR=\(0.01\)) as a function of enrollment database size.
| Dataset: | Operational Dataset 4th pull (stats on OPS4 images) |
| Samples used: | Both eyes |
| Enrollment Method: | Both (left and right) iris images per enrollment template |
Some apparant trends may be the result of random variation. Results for the 10K enrollment size were computed from 140K searches. Results for the 50K, 100K and 500K enrollment sizes were computed from 700K searches.
Contact Info
Inquiries and comments may be submitted to [email protected]. Subscribe to the IREX mailing list to stay up-to-date on all IREX-related activities.