14.1.3. Time-Series IO

14.1.3.1. mosaic.trajio.abfTrajIO module

digraph inheritancef6219e35d8 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "abfTrajIO.abfTrajIO" [URL="../doc/settingsFile.html#mosaic.trajio.abfTrajIO.abfTrajIO",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Read ABF1 and ABF2 file formats. Currently, only "]; "metaTrajIO.metaTrajIO" -> "abfTrajIO.abfTrajIO" [arrowsize=0.5,style="setlinewidth(0.5)"]; "metaTrajIO.metaTrajIO" [URL="../doc/settingsFile.html#mosaic.trajio.metaTrajIO.metaTrajIO",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip=".. warning:: |metaclass|"]; }

A TrajIO class that supports ABF1 and ABF2 file formats via abf/abf.py. Currently, only gap-free mode and single channel recordings are supported.

Created:

5/23/2013

Author:

Arvind Balijepalli <arvind.balijepalli@nist.gov>

License:

See LICENSE.TXT

ChangeLog:

03/20/23 AB Use pyabf to read ABF files.
9/13/15 AB Updated logging to use mosaicLogFormat class
3/28/15 AB Updated file read code to match new metaTrajIO API.
5/23/13 AB Initial version
class mosaic.trajio.abfTrajIO.abfTrajIO(**kwargs)[source]

Bases: metaTrajIO

Read ABF1 and ABF2 file formats. Currently, only gap-free mode and single channel recordings are supported.

A typical settings section to read ABF files is shown below.

"abfTrajIO" : {
"filter"                        : "*.abf",
"start"                         : 0.0,
"dcOffset"                      : 0.0,
"sweepNumber"                                   : 0,
"channel"                                               : 0
}
Parameters:
In addition to metaTrajIO args,

None

readdata(fname)[source]

Read one or more files and append their data to the data pipeline. Set a class attribute Fs with the sampling frequency in Hz.

Parameters:

  • fname : fileame to read

Returns:

  • An array object that holds raw (unscaled) data from fname

Errors:

  • SamplingRateChangedError : if the sampling rate for any data file differs from previous

14.1.3.2. mosaic.trajio.qdfTrajIO module

digraph inheritance6b7c8f8a97 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "metaTrajIO.metaTrajIO" [URL="../doc/settingsFile.html#mosaic.trajio.metaTrajIO.metaTrajIO",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip=".. warning:: |metaclass|"]; "qdfTrajIO.qdfTrajIO" [URL="../doc/settingsFile.html#mosaic.trajio.qdfTrajIO.qdfTrajIO",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Use the readqdf module from EBS to read individual QDF files."]; "metaTrajIO.metaTrajIO" -> "qdfTrajIO.qdfTrajIO" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

QDF implementation of metaTrajIO. Uses the readqdf module from EBS to read individual qdf files.

Created:

7/18/2012

Author:

Arvind Balijepalli <arvind.balijepalli@nist.gov>

License:

See LICENSE.TXT

ChangeLog:

9/13/15 AB Updated logging to use mosaicLogFormat class
3/28/15 AB Updated file read code to match new metaTrajIO API.
7/18/12 AB Initial version
2/11/14 AB Support qdf files that save the current in pA. This needs
format=’pA’ argument.
class mosaic.trajio.qdfTrajIO.qdfTrajIO(**kwargs)[source]

Bases: metaTrajIO

Use the readqdf module from EBS to read individual QDF files.

In addition to metaTrajIO args, check if the feedback resistance (Rfb) and feedback capacitance (Cfb) are defined to convert qdf binary data into pA.

A typical settings section to read QDF files is shown below. Note, that the values for Rfb and Cfb are specific to the amplifier used.

"qdfTrajIO": {
"Rfb"                           : 9.1e+12,
"Cfb"                           : 1.07e-12,
"dcOffset"                      : 0.0,
"filter"                        : "*.qdf",
"start"                         : 0.0
}
Parameters:
In addition to metaTrajIO.__init__ args,

  • Rfb : feedback resistance of amplifier

  • Cfb : feedback capacitance of amplifier

  • format : ‘V’ for voltage or ‘pA’ for current. Default is ‘V’

Returns:

None

Errors:

  • InsufficientArgumentsError : if the mandatory arguments Rfb and Cfb are not set.

readdata(fname)[source]

Read one or more files and append their data to the data pipeline. Set a class attribute Fs with the sampling frequency in Hz.

Parameters:

  • fname : list of data files to read

Returns:

None

Errors:

  • SamplingRateChangedError : if the sampling rate for any data file differs from previous

14.1.3.3. mosaic.trajio.binTrajIO module

digraph inheritance24256f9782 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "binTrajIO.InvalidDataColumnError" [URL="#mosaic.trajio.binTrajIO.InvalidDataColumnError",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "binTrajIO.binTrajIO" [URL="../doc/settingsFile.html#mosaic.trajio.binTrajIO.binTrajIO",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Read a file that contains interleaved binary data, ordered by column. Only a single "]; "metaTrajIO.metaTrajIO" -> "binTrajIO.binTrajIO" [arrowsize=0.5,style="setlinewidth(0.5)"]; "metaTrajIO.metaTrajIO" [URL="../doc/settingsFile.html#mosaic.trajio.metaTrajIO.metaTrajIO",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip=".. warning:: |metaclass|"]; }

Binary file implementation of metaTrajIO. Read raw binary files with specified record sizes

Created:

4/22/2013

Author:

Arvind Balijepalli <arvind.balijepalli@nist.gov>

License:

See LICENSE.TXT

ChangeLog:

9/13/15 AB Updated logging to use mosaicLogFormat class
3/28/15 AB Updated file read code to match new metaTrajIO API.
1/27/15 AB Memory map files on read.
1/26/15 AB Refactored code to read interleaved binary data.
7/27/14 AB Update interface to specify python PythonStructCode instead of
RecordSize. This will allow any binary file to be decoded
The AmplifierScale and AmplifierOffset are set to 1 and 0
respectively if PythonStructCode is an integer or short.
4/22/13 AB Initial version
exception mosaic.trajio.binTrajIO.InvalidDataColumnError[source]

Bases: Exception

class mosaic.trajio.binTrajIO.binTrajIO(**kwargs)[source]

Bases: metaTrajIO

Read a file that contains interleaved binary data, ordered by column. Only a single column that holds ionic current data is read. The current in pA is returned after scaling by the amplifier scale factor (AmplifierScale) and removing any offsets (AmplifierOffset) if provided.

Usage and Assumptions:

Binary data is interleaved by column. For three columns (a, b, and c) and N rows, binary data is assumed to be of the form:

[ a_1, b_1, c_1, a_2, b_2, c_2, … … …, a_N, b_N, c_N ]

The column layout is specified with the ColumnTypes parameter, which accepts a list of tuples. For the example above, if column a is the ionic current in a 64-bit floating point format, column b is the ionic current representation in 16-bit integer format and column c is an index in 16-bit integer format, the ColumnTypes paramter is a list with three tuples, one for each column, as shown below:

[(‘curr_pA’, ‘float64’), (‘AD_V’, ‘int16’), (‘index’, ‘int16’)]

The first element of each tuple is an arbitrary text label and the second element is a valid Numpy type.

Finally, the IonicCurrentColumn parameter holds the name (text label defined above) of the column that holds the ionic current time-series. Note that if an integer column is selected, the AmplifierScale and AmplifierOffset parameters can be used to convert the voltage from the A/D to a current.

Assuming that we use a floating point representation of the ionic current, and a sampling rate of 50 kHz, a settings section that will read the binary file format defined above is:

"binTrajIO": {
        "AmplifierScale" : "1",
        "AmplifierOffset" : "0",
        "SamplingFrequency" : "50000",
        "ColumnTypes" : "[('curr_pA', 'float64'), ('AD_V', 'int16'), ('index', 'int16')]",
        "IonicCurrentColumn" : "curr_pA",
        "dcOffset": "0.0", 
        "filter": "*.bin", 
        "start": "0.0",
        "HeaderOffset": 0 
}
Settings Examples:

Read 16-bit signed integers (big endian) with a 512 byte header offset. Set the amplifier scale to 400 pA, sampling rate to 200 kHz.

"binTrajIO": {
        "AmplifierOffset": "0.0", 
        "SamplingFrequency": 200000, 
        "AmplifierScale": "400./2**16", 
        "ColumnTypes": "[('curr_pA', '>i2')]", 
        "dcOffset": 0.0, 
        "filter": "*.dat", 
        "start": 0.0, 
        "HeaderOffset": 512, 
        "IonicCurrentColumn": "curr_pA"
}

Read a two-column file: 64-bit floating point and 64-bit integers, and no header offset. Set the amplifier scale to 1 and sampling rate to 200 kHz.

"binTrajIO": {
        "AmplifierOffset": "0.0", 
        "SamplingFrequency": 200000, 
        "AmplifierScale": "1.0", 
        "ColumnTypes" : "[('curr_pA', 'float64'), ('AD_V', 'int64')]",
        "dcOffset": 0.0, 
        "filter": "*.bin", 
        "start": 0.0, 
        "HeaderOffset": 0, 
        "IonicCurrentColumn": "curr_pA"
}
Parameters:
In addition to metaTrajIO args,

  • AmplifierScale : Full scale of amplifier (pA/2^nbits) that varies with the gain (default: 1.0).

  • AmplifierOffset : Current offset in the recorded data in pA (default: 0.0).

  • SamplingFrequency : Sampling rate of data in the file in Hz.

  • HeaderOffset : Ignore first n bytes of the file for header (default: 0 bytes).

  • ColumnTypes : A list of tuples with column names and types (see Numpy types). Note only integer and floating point numbers are supported.

  • IonicCurrentColumn : Column name that holds ionic current data.

Returns:

None

Errors:

None

readdata(fname)[source]

Return raw data from a single data file. Set a class attribute Fs with the sampling frequency in Hz.

Parameters:

  • fname : fileame to read

Returns:

  • An array object that holds raw (unscaled) data from fname

Errors:

None

scaleData(data)[source]

See mosaic.metaTrajIO.metaTrajIO.scaleData().

14.1.3.4. mosaic.trajio.chimeraTrajIO module

digraph inheritanceae9304eec5 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "chimeraTrajIO.InvalidDataColumnError" [URL="#mosaic.trajio.chimeraTrajIO.InvalidDataColumnError",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "chimeraTrajIO.chimeraTrajIO" [URL="../doc/settingsFile.html#mosaic.trajio.chimeraTrajIO.chimeraTrajIO",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Read a file generated by the Chimera VC100. The current in pA "]; "metaTrajIO.metaTrajIO" -> "chimeraTrajIO.chimeraTrajIO" [arrowsize=0.5,style="setlinewidth(0.5)"]; "metaTrajIO.metaTrajIO" [URL="../doc/settingsFile.html#mosaic.trajio.metaTrajIO.metaTrajIO",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip=".. warning:: |metaclass|"]; }

Chimera VC100 concatenated file format implementation of metaTrajIO. Read concatenated chimera files with specified amplifier settings.

Created:

7/11/2016

Author:

Kyle Briggs <kbrig035@uottawa.ca>

License:

See LICENSE.TXT

ChangeLog:

5/12/21 AB Update Chimera settings from MAT files generated during data collection.
7/29/16 KB Miscelleneous bugfixes
7/11/16 KB Initial version
exception mosaic.trajio.chimeraTrajIO.InvalidDataColumnError[source]

Bases: Exception

class mosaic.trajio.chimeraTrajIO.chimeraTrajIO(**kwargs)[source]

Bases: metaTrajIO

Read a file generated by the Chimera VC100. The current in pA is returned after scaling by the amplifier scale factors.

Usage and Assumptions:

Binary data is in a single column of unsigned 16 bit integers:

The column layout is specified with the ColumnTypes parameter, which accepts a list of tuples.

[(‘curr_pA’, ‘<u2’)]

The option is left in in case of future changes to the platform, but can be left alone in the settings file for now. The first element of each tuple is an arbitrary text label and the second element is a valid Numpy type.

Chimera gain settings are used to convert the integers stored by the ADC to current values. These values are automatically read in from matched MAT files generated by the Chimera software.

"chimeraTrajIO": {
        "filter": "*.log", 
        "start": "0.0",
        "HeaderOffset": "0"
}
Parameters:

In addition to metaTrajIO args,

  • HeaderOffset : Ignore first n bytes of the file for header (currently fixed at: 0 bytes).

Returns:

None

Errors:

None

readdata(fname)[source]

Return raw data from a single data file. Set a class attribute Fs with the sampling frequency in Hz.

Parameters:

  • fname : fileame to read

Returns:

  • An array object that holds raw (unscaled) data from fname

Errors:

None

14.1.3.5. mosaic.trajio.tsvTrajIO module

digraph inheritance3cb00b4a80 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "metaTrajIO.metaTrajIO" [URL="../doc/settingsFile.html#mosaic.trajio.metaTrajIO.metaTrajIO",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip=".. warning:: |metaclass|"]; "tsvTrajIO.tsvTrajIO" [URL="../doc/settingsFile.html#mosaic.trajio.tsvTrajIO.tsvTrajIO",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Read tab separated valued (TSV) files. "]; "metaTrajIO.metaTrajIO" -> "tsvTrajIO.tsvTrajIO" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

An implementation of metaTrajIO that reads tab separated valued (TSV) files

Created:

7/31/2012

Author:

Arvind Balijepalli <arvind.balijepalli@nist.gov>

License:

See LICENSE.TXT

ChangeLog:

11/30/15 AB Assumes timeCol is specified in seconds.
11/30/15 AB Added a new keyword scale to allow scaling TSV data.
3/28/15 AB Updated file read code to match new metaTrajIO API.
6/30/13 AB Added the ‘seprator’ kwarg to the class initializer to allow any delimited
files to be read. e.g. ‘”"t’ (default), ‘,’, etc.
7/31/12 AB Initial version
class mosaic.trajio.tsvTrajIO.tsvTrajIO(**kwargs)[source]

Bases: metaTrajIO

Read tab separated valued (TSV) files.

Parameters:
In addition to metaTrajIO args,

  • headers : If True, the first row is ignored (default: True)

  • separator : set the data separator (defualt: ‘”"t’)

  • scale : set the data scale (default: 1). For example to convert from to pA set scale=1e12.

Either:

  • Fs : Sampling frequency in Hz. If set, all other options are ignored and the first column in the file is assumed to be the current in pA.

Or:

  • nCols : number of columns in TSV file (default:2, first column is time in ms and second is current in pA)

  • timeCol : explicitly set the time column (default: 0, first col)

  • currCol : explicitly set the position of the current column (default: 1)

If neither Fs nor {nCols, timeCol, currCol} are set then the latter is assumed with the listed default values.

readdata(fname)[source]

Read a single TSV file and return raw (unscaled) data contained within it. Set/update a class attribute Fs with the sampling frequency in Hz.

Parameters:

  • fname : fileame to read

Returns:

  • An array object that holds raw (unscaled) data from fname

Errors:

  • SamplingRateChangedError : if the sampling rate for any data file differs from previous

scaleData(data)[source]

Important

Abstract method: This optional interface method can be overridden by a sub-class to modify functionality.

Scale the raw data loaded with readdata(). Note this function will not necessarily receive the entire data array loaded with readdata(). Transformations must be able to process partial data chunks.

Parameters:

  • data : partial chunk of raw data loaded using readdata().

Returns:

  • Array containing scaled data.

Default Behavior:

  • If not implemented by a sub-class, the default behavior is to return data to the calling function without modifications.

Example:

Assuming the amplifier scale and offset values are stored in the class variables AmplifierScale and AmplifierOffset, the raw data read using readdata() can be transformed by scaleData(). We can also use this function to change the array data type.

def scaleData(self, data):
        return np.array(data*self.AmplifierScale-self.AmplifierOffset, dtype='f8')