2 Timing Sample Dataset

A fixed sample of images randomly selected from the PFT III datasets are used to assess whether or not an implementation adheres to the minimum timing requirements set forth in the PFT III test plan. This sample is also used to provide an estimate on template size. The images and comparisons are identical to the “1K Sample Evaluation” from NIST’s PFT II evaluation, with the exception of the “IARPA N2N” dataset, which is new in PFT III. Table 2.1 shows information about the maximum dimensions and resolutions of the images in each of the timing sample datasets.

Table 2.1: Maximum dimensions in pixels and capture resolution in pixels per inch (PPI) for the images in each of the subsets comprising the timing sample dataset.
AZ LA County DHS 2 POE+BVA IARPA N2N
Max Dimensions (pixels) 800 x 800 412 x 1 000 368 x 368 500 x 500 1 600 x 1 500
Resolution (PPI) 500 500 500 500 1 000

2.1 Template Size

Figure 2.1 and Table 2.2 show the distribution of file sizes of templates. Failures of any kind reported during template generation result in NIST code writing 0 byte files. These files are excluded from the template size analysis in this section.

Box plots of template sizes in bytes of templates created from a fixed sample of data from the PFT III evaluation. An overall plot is shown, as well as individual plots per data origin. Tabular versions of this data are shown in Table 2.2.

Figure 2.1: Box plots of template sizes in bytes of templates created from a fixed sample of data from the PFT III evaluation. An overall plot is shown, as well as individual plots per data origin. Tabular versions of this data are shown in Table 2.2.

Table 2.2: Sizes in bytes of templates created from a fixed sample of data from the PFT III evaluation. The bottom row, Failures, shows the number of failures to create a template, which are not included in these statistics. Box plots of this data are shown in Figure 2.1.
Overall AZ LA County DHS 2 POE+BVA IARPA N2N
Min 8 819 10 476 10 308 12 847 8 819 11 553
25 % 13 106 15 577 14 331 13 085 12 285 13 655
Median 13 442 16 571 15 346 13 134 13 064 14 842
Mean 14 558 16 805 15 391 13 140 12 657 15 274
75 % 16 179 17 680 16 277 13 190 13 134 16 470
Max 23 379 23 379 20 960 13 547 13 442 22 761
Failures 0 0 0 0 0 0

2.2 Template Creation Time

Figure 2.2 and Table 2.3 show the distribution of durations of time consumed when creating templates. Failures of all kinds are incorporated into these statistics, since this time would be observed by the end user of a template creation algorithm.

Times are measured by running a single process on an isolated compute node equipped with an Intel Xeon Gold 6254 CPU.

Box plots of elapsed milliseconds when creating templates from a fixed sample of data from the PFT III evaluation. All times are used, even if a failure occurred. Tabular versions of this data are shown in Table 2.3.

Figure 2.2: Box plots of elapsed milliseconds when creating templates from a fixed sample of data from the PFT III evaluation. All times are used, even if a failure occurred. Tabular versions of this data are shown in Table 2.3.

Table 2.3: Elapsed milliseconds when creating templates from a fixed sample of data from the PFT III evaluation. All times are used, even if a failure occurred. The bottom row, Failures, shows the number of failures to generate a template. Failures are included in these statistics. Box plots of this data are shown in Figure 2.2.
Overall AZ LA County DHS 2 POE+BVA IARPA N2N
Min 216.8 260.7 247.6 351.8 216.8 331.9
25 % 362.2 523.7 451.1 360.0 337.9 457.8
Median 425.3 577.0 500.5 363.8 360.9 521.9
Mean 460.8 599.4 502.2 364.8 346.9 550.4
75 % 545.9 642.0 546.4 368.8 365.2 613.6
Max 2 354.6 2 236.6 2 181.4 1 646.2 1 718.1 2 354.6
Failures 0 0 0 0 0 0

2.3 Template Comparison Time

Figure 2.3 and Table 2.4 show the distribution of durations of time consumed when comparing templates. Failures of any kind are incorporated into these statistics, since this time would be observed by the end user of a template comparison algorithm.

Times are measured by running a single process on an isolated compute node equipped with an Intel Xeon Gold 6254 CPU.

Box plots of elapsed microseconds when comparing two templates from a fixed sample of data from the PFT III evaluation. All times are used, even if a failure occurred. Tabular versions of this data are shown in Table 2.4.

Figure 2.3: Box plots of elapsed microseconds when comparing two templates from a fixed sample of data from the PFT III evaluation. All times are used, even if a failure occurred. Tabular versions of this data are shown in Table 2.4.

Table 2.4: Elapsed microseconds when comparing two templates from a fixed sample of data from the PFT III evaluation. The bottom row, Failures, shows the number of failures to compare. Failures are included in these statistics. Box plots of this data are shown in Figure 2.4.
Overall AZ LA County DHS 2 POE+BVA IARPA N2N
Min 451.0 1 878.1 1 099.5 487.3 451.0 1 314.4
25 % 2 272.3 6 501.3 3 894.1 1 908.5 1 875.8 4 614.6
Median 4 107.4 8 073.2 4 700.8 2 191.8 2 115.7 5 884.2
Mean 4 999.1 8 769.0 5 238.3 2 306.2 2 294.0 6 410.6
75 % 6 636.3 10 095.6 5 836.6 2 502.5 2 450.1 7 444.4
Max 229 138.9 229 138.9 204 081.6 192 720.4 200 454.2 208 828.6
Failures 0 0 0 0 0 0