Testing and Curation of Federates

It is not sufficient to address solely the technical challenges required to enable federated experiments across testbeds. As experiments become more complex through integration of new federates, and as CPS applications become more reliant on experimental results to build assurance in their design and implementation, it becomes critical to characterize, validate and curate the individual federate models. It is likely that models developed for one project will be repurposed in another. The utilization of models created by others requires a means of model characterization that goes beyond the documentation produced solely for single test use. Methods of testing individual federates is necessary to gain confidence in their expected operation like unit testing in software systems engineering.

The topics of testing and curation require that a given federate be supported by life cycle management, specification of its features and capabilities, and control of version and function.

The life cycle for federate development consists of several major steps:

  1. design of the federate and its interfaces
  2. implementation of the federate in hardware or software,
  3. documentation of the federate to describe its scope and usage,
  4. development of an experimental test hardness to validate the federate behavior,
  5. archiving of the federate, its documentation, and its testing results,
  6. the evolution of the federate with respects to defect management, enhancement and extension.

We introduce here the concept of a test harness for a re-useable federate. The development of a federate must be kept synchronized with the development of its corresponding test harness. The test harness should be able to generate a pass-fail result to support the validation of federations on a continuous basis. This continuous integration is desirable so that regression tests over federates can be periodically invoked to ensure the integrity of the designs. The archived federate must utilize versioning so that as the federate is evolved, applications can rely on stable versions for their integration. Semantic versioning should be used.

A federate requires description so that a potential integrator of a federate can understand simply its capabilities and limitations. For example, a federate that acts as a source of weather that provides hourly data, cannot be used in a federation that demands minute by minute variations. The “federate descriptor” should provide this level of detail along with the interfaces that describe information received and emitted from the federate. The NIST CPS Framework describes a collection of “concerns” that a given CPS can satisfy. A federate descriptor based on the specification of these concerns addressed can also define the test boundaries of federate testing.

Together, these dimensions can provide for a robust organization of federate designs and testing that in turn can encourage re-use.