6 FBI-Provided Solved Dataset #1

The results of Section 6 are based on searches of the sequestered dataset FBI-Provided Solved Dataset #1. This dataset consists of 516 operational probes collected from a particular type of crime. Examiners at the FBI annotated several of the probe images with EFS features, possibly with algorithm assistance. These examiners then confirmed the ground truth mate. All probes searched were a single sample depicting a region from a distal phalanx. EFS data provided with the probe image may include:

  • Pattern classification
  • Core locations (unconfirmed source)
  • Delta locations (unconfirmed source)
  • Minutia locations (unconfirmed source)

6.1 Failures

Table 6.1 shows the number of failures to create templates. Table 6.2 shows the number of failures to produce a candidate list.

Table 6.1: Number of failures to create templates.
Image Type Content Failures Attempts
Exemplar Image 0 173
Probe EFS 0 285
Probe Image 0 516
Probe Image + EFS 0 516
Table 6.2: Number of failures to produce a candidate list. This number includes any failures to create a probe template from Table 6.1.
Probe Content Failures Attempts
EFS 0 285
Image 2 516
Image + EFS 1 516

6.2 CMC

6.2.1 Plots

6.2.1.1 All Probes

The CMC plots in Figure 6.1 show the FNIR of dermalog+0006 when searching FBI-Provided Solved Dataset #1 against enrollment database where a single mated identity for each search probe was present. The plots are faceted by the mated impression type and whether probe EFS data was provided. Tabular versions of FNIR at select ranks can be viewed in Table 6.3.

CMC when searching FBI-Provided Solved Dataset \#1 probes, faceted by the mated impression type and whether probe EFS data was provided.

Figure 6.1: CMC when searching FBI-Provided Solved Dataset #1 probes, faceted by the mated impression type and whether probe EFS data was provided.

6.2.1.2 Probes with EFS Data

Not all of the probes in the FBI-Provided Solved Dataset #1 dataset contain EFS data. The plot in Figure 6.2 shows the CMC over only the probes that contained EFS data. This plot also differs from Figure 6.1 with the inclusion of a line for probes where no image was provided when creating the probe template, meaning the only information available was EFS data. Only the region success location is shown.

CMC of region location when searching only the FBI-Provided Solved Dataset \#1 probes that contained EFS data.

Figure 6.2: CMC of region location when searching only the FBI-Provided Solved Dataset #1 probes that contained EFS data.

6.2.2 FNIR at Select Rank

6.2.2.1 All Probes

The values in Table 6.3 correspond to Figure 6.1.

Table 6.3: Region FNIR values from CMC plotted in Figure 6.1.
Probe Content Rank 1 Rank  ≤ 2 Rank  ≤ 5 Rank  ≤ 10 Rank  ≤ 50 Rank  ≤ 100
Image 0.2578 0.2442 0.2209 0.2112 0.1802 0.1609
Image + EFS 0.2771 0.2539 0.2151 0.2054 0.1647 0.1492

6.2.2.2 Probes with EFS Data

The values in Table 6.4 correspond to Figure 6.2.

Table 6.4: Region FNIR values from CMC plotted in Figure 6.2.
Probe Content Rank 1 Rank  ≤ 2 Rank  ≤ 5 Rank  ≤ 10 Rank  ≤ 50 Rank  ≤ 100
EFS 0.2982 0.2561 0.2000 0.1860 0.1579 0.1439
Image 0.2316 0.2246 0.2035 0.1895 0.1579 0.1298
Image + EFS 0.2667 0.2421 0.1930 0.1789 0.1298 0.1088

6.3 DET

6.3.1 Plots

6.3.1.1 All Probes

The DET plots in Figure 6.3 show the false positive and false negative identification rate tradeoffs of dermalog+0006 when searching FBI-Provided Solved Dataset #1 against enrollment database where a single mated identity for each search probe was present. The plots are faceted by the mated impression type and whether probe EFS data was provided. Tabular versions of FNIR at select FPIR can be viewed in Table 6.5. Annotated values indicate similarity scores from the Region line, which are tabulated in Table 6.7.

DET when searching FBI-Provided Solved Dataset \#1 probes, faceted by the mated impression type and whether probe EFS data was provided. Annotated values indicate similarity scores from the Region line.

Figure 6.3: DET when searching FBI-Provided Solved Dataset #1 probes, faceted by the mated impression type and whether probe EFS data was provided. Annotated values indicate similarity scores from the Region line.

6.3.1.2 Probes with EFS Data

Not all of the probes in the FBI-Provided Solved Dataset #1 dataset contain EFS data. The plot in Figure 6.4 shows the DET over only the probes that contained EFS data. This plot also differs from Figure 6.3 with the inclusion of a line for probes where no image was provided when creating the probe template, meaning the only information available was EFS data. Only the region success location is shown.

DET of region location when searching only the FBI-Provided Solved Dataset \#1 probes that contained EFS data.

Figure 6.4: DET of region location when searching only the FBI-Provided Solved Dataset #1 probes that contained EFS data.

6.3.2 FNIR at Select FPIR

6.3.2.1 All Probes

The values in Table 6.5 correspond to Figure 6.3.

Table 6.5: Region FNIR values corresponding to FPIR plotted in Figure 6.3.
Probe Content FPIR = 0.01 FPIR = 0.02 FPIR = 0.1
Image 0.4554 0.4380 0.3779
Image + EFS 0.5019 0.4884 0.4070

6.3.2.2 Probes with EFS Data

The values in Table 6.6 correspond to Figure 6.4.

Table 6.6: Region FNIR values corresponding to FPIR plotted in Figure 6.4.
Probe Content FPIR = 0.01 FPIR = 0.02 FPIR = 0.1
EFS 0.4807 0.4807 0.4281
Image 0.4554 0.4380 0.3779
Image + EFS 0.5019 0.4884 0.4070

6.3.3 Similarity Score Thresholds at Select FPIR

The values in Table 6.7 correspond to similarity score thresholds observed at the select FPIR values from Table 6.5.

Table 6.7: Similarity score thresholds corresponding to select FPIR values from Table 6.5.
Probe Content FPIR = 0.01 FPIR = 0.02 FPIR = 0.1
Image 114.23 110.76 95.15
Image + EFS 136.37 133.02 116.11