Reading and Writing Pipelines

Its

Google Colab Setup

Only uncomment and run the next cell if you are running this notebook in Google Colab or if don’t already have the AFL-agent package installed.

# !pip install git+https://github.com/usnistgov/AFL-agent.git

Writing A Pipeline

To begin, let’s load the necessary libraries and define a short pipeline

from AFL.double_agent import *

with Pipeline('MyPipeline') as my_important_pipeline:

        SavgolFilter(
            input_variable='measurement',
            output_variable='derivative',
            dim='x',
            derivative=1
            )

        Similarity(
            input_variable='derivative',
            output_variable='similarity',
            sample_dim='sample',
            params={'metric': 'laplacian','gamma':1e-4}
            )

        SpectralClustering(
            input_variable='similarity',
            output_variable='labels',
            dim='sample',
            params={'n_phases': 2}
            )

my_important_pipeline.print()

PipelineOp                               input_variable ---> output_variable
----------                               -----------------------------------
0  ) <SavgolFilter>                      measurement ---> derivative
1  ) <SimilarityMetric>                  derivative ---> similarity
2  ) <SpectralClustering>                similarity ---> labels

Input Variables
---------------
0) measurement

Output Variables
----------------
0) labels

Now, we can write the pipeline by simply calling the .write_json() method

my_important_pipeline.write_json('pipeline.json')

Pipeline successfully written to pipeline.json.

Excellent! Let’s take a look at the json file that was written. We can inspect it using the json module

import json

with open('pipeline.json','r') as f:
    display(json.load(f))

{'name': 'MyPipeline',
 'date': '03/04/25 19:59:19-576491',
 'ops': [{'class': 'AFL.double_agent.Preprocessor.SavgolFilter',
   'args': {'input_variable': 'measurement',
    'output_variable': 'derivative',
    'dim': 'x',
    'xlo': None,
    'xhi': None,
    'xlo_isel': None,
    'xhi_isel': None,
    'pedestal': None,
    'npts': 250,
    'derivative': 1,
    'window_length': 31,
    'polyorder': 2,
    'apply_log_scale': True,
    'name': 'SavgolFilter'}},
  {'class': 'AFL.double_agent.PairMetric.Similarity',
   'args': {'input_variable': 'derivative',
    'output_variable': 'similarity',
    'sample_dim': 'sample',
    'params': {'metric': 'laplacian', 'gamma': 0.0001},
    'constrain_same': [],
    'constrain_different': [],
    'name': 'SimilarityMetric'}},
  {'class': 'AFL.double_agent.Labeler.SpectralClustering',
   'args': {'input_variable': 'similarity',
    'output_variable': 'labels',
    'dim': 'sample',
    'params': {'n_phases': 2},
    'name': 'SpectralClustering',
    'use_silhouette': False}}]}

With this, we can see that all of the PipelineOps are stored in the ops keyword with the keyword arguments we specified above. Also included are any default arguments that we didn’t explicitly specify.

Reading a Pipeline

So the next and final step is to load the pipeline from disk.

loaded_pipeline = Pipeline.read_json('pipeline.json')
loaded_pipeline.print()

PipelineOp                               input_variable ---> output_variable
----------                               -----------------------------------
0  ) <SavgolFilter>                      measurement ---> derivative
1  ) <SimilarityMetric>                  derivative ---> similarity
2  ) <SpectralClustering>                similarity ---> labels

Input Variables
---------------
0) measurement

Output Variables
----------------
0) labels

Success!

Conclusion