SRSE Speaker Recognition Sequestered Evaluation (SRSE) 2024

Participation Agreement | Validation [TBD] | Encryption [TBD]

Status

[2024-02-14] The Speaker Recognition Sequestered Evaluation (SRSE) 2024 activity is being conducted to assess the capability and performance level of speaker recognition system in purpose desgined scenarios. A draft evaluation plan [TBD] is now available for public comment until 2024-02-29.

Overview

Due to renewed interest in the U.S. Government Community, in 2024 NIST will conduct a Speaker Recognition Sequestered Evaluation (SRSE) pilot that will mark the beginning of a recurring series of SRSE evaluations. Due to its inaugural nature the participation in the SRSE 2024 pilot is by invitation only.

Subsequent evaluations in the series will be open to the broader community of participants and will use data new to the community and will proceed either annually or on a rolling basis. The participants shall provide their algorithms in a containerized form, that have potential to be deployed into biometric workflows for operational speaker recognition applications. A key difference of this new evaluation series is the use of sequestered data, which will remain inaccessible to participants during and after the evaluation. Accordingly, the participants are required to submit their containerized algorithms in a self-contained, containerized executable form (that does not rely on external cloud resources) along with all required state data, parameters, and dependencies needed for execution in a sequestered (air-gaped) environment. The participating systems must operate under a defined interface protocol that allows the NIST evaluators to control their initialization parameters, enrolment, target/non-target test trials, and the disposition of their outputs. Moreover, the executable system submissions must be compatible with the SRSE evaluation hardware and its software environment. The SRSE is structured so as to present the participants with speaker identification tasks and conditions that allow assessment of their algorithm capabilities and performance levels in various purpose designed scenarios.

How to Participate

To participate in this evaluation, developers need to:

Validation

Instructions on the download of the validation dataset will be provided in this section. All participants must run their containers against the validation set prior to submission to confirm proper function. NIST will run the submission containers against the same validation dataset to confirm proper execution on NIST infrastructure. The purpose of validation is to ensure consistent algorithm output between participant’s execution and NIST’s execution.

Instructions to download the validation data set – [TBD] [last update: 2024-02-14]

Submission

All containerized algorithm submissions must be submitted through the SRSE Submission Form, which requires encrypted files be provided as a download link from a generic http server (e.g., Google Drive). NIST cannot accept Dropbox links. NIST will not register, or establish any kind of membership, on the provided website. [last update: 2024-02-14]

Contact Information

Inquiries and comments may be submitted to srse@nist.gov.

Subscribe

Instructions on how to join our mailing list will be provided in this section once the mailing list is established.