NIST Pages Template

Usage of Test Method Tool

The tool is a command line interface (CLI) application, invoked using the command testcdf. It takes a Common Data Format (CDF) instance (i.e. a CDF file in JSON or XML) as an input, and produces a report as its primary output. There are additional flags that may or must be provided depending on the test.

The general invocation of the test method is of the form:

testcdf {pipeline to execute} [-silent] -input:source={path/uri to source document} [-output:report={path/uri to output report}] -option:schemaNS={namespace of CDF}

The required command line arguments for a particular CDF and serialization are given below:

CDF Input Type Pipeline Schema Namespace
Ballot Definition v1 JSON validate_json.xpl
Ballot Definition v1 XML validate_xml.xpl
Cast Vote Records v1 JSON validate_json.xpl
Cast Vote Records v1 XML validate_xml.xpl
Election Results Reporting v2 JSON validate_json.xpl
Election Results Reporting v2 XML validate_xml.xpl
Election Event Logging v1 JSON validate_eel_json.xpl
Election Event Logging v1 XML validate_eel_xml.xpl
Voter Records Interchange JSON validate_json.xpl
Voter Records Interchange XML validate_xml.xpl

Pipelines are located under the pipelines/ directory; they follow a standardized naming structure: validate_[cdf_abbreviation_cdf_version]_{xml|json}.xpl. All pipelines of this form can be directly invoked by the Test Method.

All pipelines take similar inputs, outputs and options.


Ports (where input or output may be placed) expect a path to a resource using OS-style path notation or a uri.


Other flags:



The Test Method can run in one of two modes, conformance mode and non-conformance mode. By default, the Test Method runs in conformance mode. Because non-conformance mode skips some internal checks, it may run somewhat faster, which may be useful for rapid testing by manufacturers during development.

Input and Output

By default, both STDERR and STDOUT are placed on the terminal output.

STDOUT contains the following:

STDERR contains the following:

If you wish to pipe the XVRL report to another program, e.g. for reporting, then use the -silent flag to suppress banners and other messages.

Return codes

The test method cannot change the return code of the underlying XProc processor. A 0 return code should not be interpreted as successful run per se, but running according to expected parameters of the test method. A nonzero return code represents a underlying error in the processor and should be carefully analyzed. If it appears to be a bug, please open a GitHub issue.

Handling Errors

As the TM is primarily a validation tool, it is important to distinguish between validation errors of files provided to the test method and runtime errors of the test method tool itself. The former will appear in an XVRL Report, the latter will appear using the c:error vocabulary. In rare cases, the TM may terminate abnormally (i.e. with a non-zero return code).

The kinds of runtime errors that may occur are hard to predict but usually involve factors of the test method’s execution environment. A runtime error might occur if the input provided was not in the correct format, the wrong pipeline was utilized for the given input, among others.

Example c:error:

<c:errors xmlns:c=""><c:error code="err:XD0038" name="json-validation" type="p:validate-with-json-schema" href="file:///C:/xproc-test-method/BUILD/pipelines/validate_eel_json.xpl" line="12" column="98" xmlns:p="" xmlns:err=""><message>Document with mediatype 'application/octet-stream' is not accepted by port 'source'.</message></c:error></c:errors>

In this case, an error message given as XML is sent to the STDOUT.