Reader Functions#

The following are functions which read various components of FCS files.

Dataset parsing#

The majority of the functions in this section are intended to read TEXT, possibly with accompanying data, possibly from multiple datasets in an FCS file.

These are summarized below:

Function

Parse Mode

Includes Data

Dataset Number

fcs_read_flat_text()

flat

no

one

fcs_read_std_text()

standard

no

one

fcs_read_flat_dataset()

flat

yes

one

fcs_read_std_dataset()

standard

yes

one

fcs_read_flat_texts()

flat

no

many

fcs_read_std_texts()

standard

no

many

fcs_read_flat_datasets()

flat

yes

many

fcs_read_std_datasets()

standard

yes

many

Each column denotes the category to which each function belongs and its intended purpose:

Parse Mode:

This refers to the method used to parse TEXT. “Flat” mode treats TEXT as a flat list of keywords and does not further processing. “Standard” mode attempts to collect this flat list into a well-defined data structure which in pyreflow is a version-specific python class (see CoreTEXT* and CoreDataset*).

“Standard” mode requires that TEXT first be parsed in “flat” mode, which implies the latter is more lenient with regard to deviations from the FCS standard.

Includes Data:

If “yes”, the function will include DATA, ANALYSIS, and OTHER segments in the returned object. Otherwise it will just include the TEXT segment.

Dataset Number:

This refers to the number of datasets in an FCS file that can be parsed by the function. If a function is “singular”, it can only parse the first dataset. Otherwise it can parse multiple datasets from a file, and returns these in a list rather than a single object.

The vast majority of FCS files only have one dataset, so the singular functions are simpler to use for many cases since they do not require any flags to be set to read one dataset.

Singular functions optionally take an dataset_offset argument which can be used to “jump” to any dataset in a file (assuming obviously one knows where it is).

Plural functions take skip and limit arguments. The former will skip the first n datasets when returning the final list (although the TEXT for all datasets will still be read to get $NEXTDATA). limit will stop the parser after n datasets have been parsed. The defaults for these are both None which will tell the parser to exhaustively read all datasets.

HEADER parsing#

fcs_read_header() merely reads the first HEADER in an FSC file.

There is no plural (multi-dataset) version of this function since reading multiple datasets requires TEXT to be parsed to obtain NEXTDATA

This function also takes a dataset_offset argument, so one can theoretically read any HEADER in the file if one knows its offset.

Summarization#

fcs_summarize() will provide a high-level summary of an FCS file.

Offline keyword repair#

fcs_read_flat_dataset_with_keywords() can be used to parse a flat list of keyword pairs into a dataset.

Sometimes, the flags provided by fcs_read_flat_dataset() are not enough to repair any issues in TEXT that might make a file unreadable.

In these cases, one can read TEXT in flat mode using fcs_read_flat_text(), repair the keywords and/or offsets out-of-band, and then feed these into fcs_read_flat_dataset_with_keywords().

This only applies to flat mode. For the standardized analogue, see the from_kws methods in CoreTEXT* and CoreDataset*.

All functions#

pyreflow.api.fcs_read_flat_text(path, text_correction=(0, 0), data_correction=(0, 0), analysis_correction=(0, 0), other_corrections=[], max_other=None, other_width=8, guess_other_width='none', squish_offsets=False, allow_pseudoempty=False, dataset_overflow_limit=0, overlap_correction_limit=0, version_override=None, supp_text_correction=(0, 0), nextdata_correction=0, allow_duplicated_supp_text='false', ignore_supp_text=False, delim_escape_mode='escaped', allow_non_ascii_delim='false', allow_nonunique='false', allow_even_delims='false', allow_odd_tokens='false', allow_empty_keys='false', allow_delim_at_boundary='false', use_encoding='utf8', allow_non_ascii_keys='false', allow_non_utf8_values='false', allow_missing_supp_text='false', allow_supp_text_own_delim='false', allow_missing_nextdata='false', trim_value_whitespace='notrim', warnings_are_errors=False, hide_warnings=False, dataset_offset=0)#

Read HEADER and TEXT from first dataset in FCS file.

Parameters:

  • path (Path) – Path to be read.

  • text_correction (OffsetCorrection) – Corrections for Primary TEXT offsets in HEADER. Defaults to (0, 0).

  • data_correction (OffsetCorrection) – Corrections for DATA offsets in HEADER. Defaults to (0, 0).

  • analysis_correction (OffsetCorrection) – Corrections for ANALYSIS offsets in HEADER. Defaults to (0, 0).

  • other_corrections (list[OffsetCorrection]) – Corrections for OTHER offsets if they exist. Each correction will be applied in order. If an offset does not need to be corrected, use (0,0). This will not affect the number of OTHER segments that are read; this is controlled by max_other. Defaults to [].

  • max_other (int | None) – Maximum number of OTHER segments that can be parsed. None means limitless. Defaults to None.

  • other_width (int) – Width (in bytes) to use when parsing OTHER offsets. Defaults to 8.

  • guess_other_width (GuessOtherWidth) – Guess the width of OTHER segments. Non-fatal failure to guess width will fall back to 8 or whatever was given in other_width Defaults to "none".

  • squish_offsets (bool) – If True and a segment’s ending offset is zero, treat entire offset as empty. This might happen if the ending offset is longer than 8 digits, in which case it must be written in TEXT. If this happens, the standards mandate that both offsets be written to TEXT and that the HEADER offsets be set to 0,0, so only writing one is an error unless this flag is set. This should only happen in FCS 3.0 files and above. Defaults to False.

  • allow_pseudoempty (bool) – If True, allow offsets like X,X-1. Some files will denote an “empty” offset as 0,-1 or 1000,999, which is logically correct since the last offset points to the last byte, thus 0,0 is actually 1 byte long. If this flat is set, such offsets will be treated as if they were 0,0. Defaults to False.

  • dataset_overflow_limit (int) – Limit by which offsets can be truncated if they exceed end of file or $NEXTDATA. Defaults to 0.

  • overlap_correction_limit (int) – Limit by which ending segment offset can be truncated if they overlap another offset. Defaults to 0.

  • version_override (VersionOverride | None) – Override the FCS version as seen in HEADER. Defaults to None.

  • supp_text_correction (OffsetCorrection) – Corrections for Supplemental TEXT offsets in TEXT. Defaults to (0, 0).

  • nextdata_correction (int) – Correction for $NEXTDATA. Defaults to 0.

  • allow_duplicated_supp_text (TriFlag) – Choose what happens if supplemental TEXT offsets overlap the primary TEXT offsets from HEADER or HEADER. The offsets will not be used if an overlap is found. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • ignore_supp_text (bool) – If True, ignore supplemental TEXT entirely. Defaults to False.

  • delim_escape_mode (DelimEscapeMode) – Determine how to escape delims in TEXT. Defaults to "escaped".

  • allow_non_ascii_delim (TriFlag) – Choose how to handle non-ASCII delimiters (outside 1-126). If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_nonunique (TriFlag) – Choose how to handle non-unique keys in TEXT. In such cases, only the first will be used regardless of this setting. If "false", raise ParseKeyError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_even_delims (TriFlag) – Choose what happens if TEXT has an even number of delimiters. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_odd_tokens (TriFlag) – Choose what happens if TEXT contains an odd number of tokens. The last ‘dangling’ token will be dropped regardless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_empty_keys (TriFlag) – Choose what happens if any keys are blank. Only relevant if if delimiters are unescaped. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_delim_at_boundary (TriFlag) – Choose what happens if there are delimiters at token boundaries. The FCS standard forbids this because it is impossible to tell if such delimiters belong to the previous or the next token. Consequently, delimiters at boundaries will be dropped regardless of this flag. Only relevant if delimiters are escaped. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • use_encoding (UseEncoding) – Choose how to interpret characters in TEXT. Defaults to "utf8".

  • allow_non_ascii_keys (TriFlag) – Choose how to handle non-ASCII keys. This only applies to non-standard keywords, as all standardized keywords may only contain letters, numbers, and start with "$". Regardless, all compliant keys must only have ASCII. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_non_utf8_values (TriFlag) – Choose what happens if non-UTF8 characters are in TEXT. Tokens with such characters will be dropped regardless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_missing_supp_text (TriFlag) – Choose how to handle supplemental missing TEXT offsets in primary TEXT. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_supp_text_own_delim (TriFlag) – Choose what happens if supplemental TEXT has a different delimiter compared to primary TEXT. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_missing_nextdata (TriFlag) – Choose how to handle missing $NEXTDATA. This is a required keyword in all versions. However, most files only have one dataset in which case this keyword is meaningless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • trim_value_whitespace (TrimValueWhitespace) – Trim whitespace from beginning and end of all values. This may create blank values if the starting string is entirely whitespace. Defaults to "notrim".

  • warnings_are_errors (bool) – If True all warnings will be regarded as errors. Defaults to False.

  • hide_warnings (bool) – If True hide all warnings. Defaults to False.

  • dataset_offset (int) – Starting position in the file of the dataset to be read. Defaults to 0.

Return type:

FlatTEXTOutput

Raises:
pyreflow.api.fcs_read_std_text(path, text_correction=(0, 0), data_correction=(0, 0), analysis_correction=(0, 0), other_corrections=[], max_other=None, other_width=8, guess_other_width='none', squish_offsets=False, allow_pseudoempty=False, dataset_overflow_limit=0, overlap_correction_limit=0, version_override=None, supp_text_correction=(0, 0), nextdata_correction=0, allow_duplicated_supp_text='false', ignore_supp_text=False, delim_escape_mode='escaped', allow_non_ascii_delim='false', allow_nonunique='false', allow_even_delims='false', allow_odd_tokens='false', allow_empty_keys='false', allow_delim_at_boundary='false', use_encoding='utf8', allow_non_ascii_keys='false', allow_non_utf8_values='false', allow_missing_supp_text='false', allow_supp_text_own_delim='false', allow_missing_nextdata='false', trim_value_whitespace='notrim', dedup_measurement_names=False, trim_intra_value_whitespace=False, time_meas_pattern='^(TIME|Time)$', allow_missing_time='false', force_linear_scale='none', ignore_optical_only_keys=[], process_optical_only_keys='demote_warn', date_pattern=None, time_pattern=None, datetime_pattern=None, last_modified_pattern=None, allow_other_feature=False, process_pseudostandard='error', process_hyper_par='error', process_other_version='error', process_extra_timestep='error', fix_log_scale_offsets=False, add_missing_timestep=None, spillover_measurement_mode='named', disallow_localtime=False, ignore_standard_keys=[], promote_to_standard=[], demote_from_standard=[], rename_standard_keys={}, replace_standard_key_values={}, append_standard_keywords={}, substitute_standard_key_values={}, allow_repair_non_unique='false', text_data_correction=(0, 0), text_analysis_correction=(0, 0), ignore_text_data_offsets=False, ignore_text_analysis_offsets=False, allow_header_text_offset_mismatch='error', allow_missing_required_offsets='false', process_optional_failure='error', int_width_override='never', byteord_override='none', disallow_range_truncation='false', warnings_are_errors=False, hide_warnings=False, dataset_offset=0)#

Read standardized TEXT from first dataset in FCS file.

Parameters:

  • path (Path) – Path to be read.

  • text_correction (OffsetCorrection) – Corrections for Primary TEXT offsets in HEADER. Defaults to (0, 0).

  • data_correction (OffsetCorrection) – Corrections for DATA offsets in HEADER. Defaults to (0, 0).

  • analysis_correction (OffsetCorrection) – Corrections for ANALYSIS offsets in HEADER. Defaults to (0, 0).

  • other_corrections (list[OffsetCorrection]) – Corrections for OTHER offsets if they exist. Each correction will be applied in order. If an offset does not need to be corrected, use (0,0). This will not affect the number of OTHER segments that are read; this is controlled by max_other. Defaults to [].

  • max_other (int | None) – Maximum number of OTHER segments that can be parsed. None means limitless. Defaults to None.

  • other_width (int) – Width (in bytes) to use when parsing OTHER offsets. Defaults to 8.

  • guess_other_width (GuessOtherWidth) – Guess the width of OTHER segments. Non-fatal failure to guess width will fall back to 8 or whatever was given in other_width Defaults to "none".

  • squish_offsets (bool) – If True and a segment’s ending offset is zero, treat entire offset as empty. This might happen if the ending offset is longer than 8 digits, in which case it must be written in TEXT. If this happens, the standards mandate that both offsets be written to TEXT and that the HEADER offsets be set to 0,0, so only writing one is an error unless this flag is set. This should only happen in FCS 3.0 files and above. Defaults to False.

  • allow_pseudoempty (bool) – If True, allow offsets like X,X-1. Some files will denote an “empty” offset as 0,-1 or 1000,999, which is logically correct since the last offset points to the last byte, thus 0,0 is actually 1 byte long. If this flat is set, such offsets will be treated as if they were 0,0. Defaults to False.

  • dataset_overflow_limit (int) – Limit by which offsets can be truncated if they exceed end of file or $NEXTDATA. Defaults to 0.

  • overlap_correction_limit (int) – Limit by which ending segment offset can be truncated if they overlap another offset. Defaults to 0.

  • version_override (VersionOverride | None) – Override the FCS version as seen in HEADER. Defaults to None.

  • supp_text_correction (OffsetCorrection) – Corrections for Supplemental TEXT offsets in TEXT. Defaults to (0, 0).

  • nextdata_correction (int) – Correction for $NEXTDATA. Defaults to 0.

  • allow_duplicated_supp_text (TriFlag) – Choose what happens if supplemental TEXT offsets overlap the primary TEXT offsets from HEADER or HEADER. The offsets will not be used if an overlap is found. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • ignore_supp_text (bool) – If True, ignore supplemental TEXT entirely. Defaults to False.

  • delim_escape_mode (DelimEscapeMode) – Determine how to escape delims in TEXT. Defaults to "escaped".

  • allow_non_ascii_delim (TriFlag) – Choose how to handle non-ASCII delimiters (outside 1-126). If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_nonunique (TriFlag) – Choose how to handle non-unique keys in TEXT. In such cases, only the first will be used regardless of this setting. If "false", raise ParseKeyError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_even_delims (TriFlag) – Choose what happens if TEXT has an even number of delimiters. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_odd_tokens (TriFlag) – Choose what happens if TEXT contains an odd number of tokens. The last ‘dangling’ token will be dropped regardless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_empty_keys (TriFlag) – Choose what happens if any keys are blank. Only relevant if if delimiters are unescaped. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_delim_at_boundary (TriFlag) – Choose what happens if there are delimiters at token boundaries. The FCS standard forbids this because it is impossible to tell if such delimiters belong to the previous or the next token. Consequently, delimiters at boundaries will be dropped regardless of this flag. Only relevant if delimiters are escaped. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • use_encoding (UseEncoding) – Choose how to interpret characters in TEXT. Defaults to "utf8".

  • allow_non_ascii_keys (TriFlag) – Choose how to handle non-ASCII keys. This only applies to non-standard keywords, as all standardized keywords may only contain letters, numbers, and start with "$". Regardless, all compliant keys must only have ASCII. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_non_utf8_values (TriFlag) – Choose what happens if non-UTF8 characters are in TEXT. Tokens with such characters will be dropped regardless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_missing_supp_text (TriFlag) – Choose how to handle supplemental missing TEXT offsets in primary TEXT. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_supp_text_own_delim (TriFlag) – Choose what happens if supplemental TEXT has a different delimiter compared to primary TEXT. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_missing_nextdata (TriFlag) – Choose how to handle missing $NEXTDATA. This is a required keyword in all versions. However, most files only have one dataset in which case this keyword is meaningless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • trim_value_whitespace (TrimValueWhitespace) – Trim whitespace from beginning and end of all values. This may create blank values if the starting string is entirely whitespace. Defaults to "notrim".

  • dedup_measurement_names (bool) – If True, force all $PnN to be unique by appending "~X" to each duplicate and incrementing X starting at 0. Defaults to False.

  • trim_intra_value_whitespace (bool) – If True, trim whitespace between delimiters such as "," and ";" within keyword value strings. Defaults to False.

  • time_meas_pattern (Selector[str | None]) – A pattern to match the $PnN of the time measurement. If "NoTime", do not try to find a time measurement. Defaults to "^(TIME|Time)$".

  • allow_missing_time (TriFlag) – Choose what to do when time measurement is be missing. If "false", raise RelationalError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • force_linear_scale (ForceLinearScale) – Force $PnE to be linear for certain measurements. Affected measurements will never fail. Defaults to "none".

  • ignore_optical_only_keys (list[OpticalOnlyKey]) – Ignore optical keys in temporal measurement. These keys are $PnG which is explicitly forbidden by the standard but allowed in this library to be set to 1.0 (noop), or others which are nonsensical for time measurements but are not explicitly forbidden in the the standard (such as $PnL). Provided keys are the string after the "Pn" in the "PnX" keywords. Defaults to [].

  • process_optical_only_keys (ProcessOpticalOnlyKeys) – Choose how to handle optical keys found in temporal measurements. Does nothing unless keys are specified in ignore_optical_only_keys. Defaults to "demote_warn".

  • date_pattern (Selector[str | None]) – If supplied, will be used as an alternative pattern when parsing $DATE. If not supplied, $DATE will be parsed according to the standard pattern which is "%d-%b-%Y". Defaults to None.

  • time_pattern (Selector[str | None]) – If supplied, will be used as an alternative pattern when parsing $BTIM and $ETIM. The values "%!" or "%@" may be used to match 1/60 seconds or centiseconds respectively. If not supplied, $BTIM and $ETIM will be parsed according to the standard pattern which is "%H:%M:%S" for 2.0, "%H:%M:%S:%!" for 3.0 and "%H:%M:%S.%@" for 3.1 and up. Defaults to None.

  • datetime_pattern (Selector[str | None]) – If supplied, will be used as an alternative pattern when parsing $BEGINDATETIME and $ENDDATETIME. The pattern must follow the format outlined in chrono. If not supplied, these will be parsed as ISO timestamps with optional timezone. Defaults to None.

  • last_modified_pattern (Selector[str | None]) – If supplied, will be used as an alternative pattern when parsing $LAST_MODIFIED. The pattern must follow the format outlined in chrono. If not supplied, these will be parsed according to the default pattern which is "%d-%b-%Y %H:%M:%S" possibly with centiseconds after. Defaults to None.

  • allow_other_feature (bool) – If True, allow $PnFEATURE to be a value other than "Area", "Width", or "Height". Defaults to False.

  • process_pseudostandard (ProcessKeywordFailure) – Process non-standard keywords with a leading "$". The presence of such keywords often means the version in HEADER is incorrect. Defaults to "error".

  • process_hyper_par (ProcessKeywordFailure) – Process measurement keywords whose index is greater than $PAR. Defaults to "error".

  • process_other_version (ProcessKeywordFailure) – Process standard keywords from different FCS versions. Defaults to "error".

  • process_extra_timestep (ProcessKeywordFailure) – Process $TIMESTEP to be present which may indicate a time measurement is present but not identified. Defaults to "error".

  • fix_log_scale_offsets (bool) – If True fix log-scale $PnE and keywords which have zero offset (ie <X>,0.0 where X is non-zero). Defaults to False.

  • add_missing_timestep (Timestep | None) – Set $TIMESTEP if it is not present and required. This will do nothing on FCS2.0 files since this version does not specify $TIMESTEP. Defaults to None.

  • spillover_measurement_mode (SpilloverMeasurementMode) – Choose how to interpret measurement strings in $SPILLOVER. Defaults to "named".

  • disallow_localtime (bool) – If True, require that $BEGINDATETIME and $ENDDATETIME have a timezone if provided. This is not required by the standard, but not having a timezone is ambiguous since the absolute value of the timestamp is dependent on localtime and therefore is location-dependent. Only affects FCS 3.2. Defaults to False.

  • ignore_standard_keys (AppendableSelector[KeyPatterns]) – Remove standard keys from TEXT. The leading "$" is implied so do not include it. Defaults to [].

  • promote_to_standard (AppendableSelector[KeyPatterns]) – Promote nonstandard keys to standard keys in TEXT. Defaults to [].

  • demote_from_standard (AppendableSelector[KeyPatterns]) – Demote nonstandard keys from standard keys in TEXT. Defaults to [].

  • rename_standard_keys (AppendableSelector[KeyStringPairs]) – Rename standard keys in TEXT. Keys matching the first part of the pair will be replaced by the second. Comparisons are case insensitive. The leading "$" is implied so do not include it. Defaults to {}.

  • replace_standard_key_values (AppendableSelector[KeyStringValues]) – Replace values for standard keys in TEXT. Comparisons are case insensitive. The leading "$" is implied so do not include it. Defaults to {}.

  • append_standard_keywords (AppendableSelector[KeyStringValues]) – Append standard key/value pairs to TEXT. All keys and values will be included as they appear here. The leading "$" is implied so do not include it. Defaults to {}.

  • substitute_standard_key_values (AppendableSelector[SubPatterns]) – Apply sed-like substitution operation on matching standard keys. The leading "$" is implied when matching keys. Defaults to {}.

  • allow_repair_non_unique (TriFlag) – Choose how to handle key collisions when repairing keywords. Non-unique keywords will not be kept in the final FCS file since each list of standard and non-standard keywords must be unique. If "false", raise ConfigError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • text_data_correction (OffsetCorrection) – Corrections for DATA offsets in TEXT. Defaults to (0, 0).

  • text_analysis_correction (OffsetCorrection) – Corrections for ANALYSIS offsets in TEXT. Defaults to (0, 0).

  • ignore_text_data_offsets (bool) – If True ignore DATA offsets in TEXT Defaults to False.

  • ignore_text_analysis_offsets (bool) – If True ignore ANALYSIS offsets in TEXT Defaults to False.

  • allow_header_text_offset_mismatch (AllowHeaderTextOffsetMismatch) – Choose what to do if HEADER and TEXT offsets are different. Exception will be FileLayoutError if emitted. Defaults to "error".

  • allow_missing_required_offsets (TriFlag) – Choose what happens when required DATA and ANALYSIS (3.1 or lower) offsets in TEXT are be missing. If missing, fall back to offsets from HEADER. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • process_optional_failure (ProcessKeywordFailure) – Process optional keys which cause an error. Defaults to "error".

  • int_width_override (IntWidthOverride) – Override $PnB. Only affects integer layouts in FCS 2.0/3.0. Defaults to "never".

  • byteord_override (ByteordOverride) – Override $BYTEORD. Only affects integer layouts in FCS 2.0/3.0. Defaults to "none".

  • disallow_range_truncation (TriFlag) – Choose how to handle $PnR values that need to be truncated to match the number of bytes specified by $PnB and $DATATYPE. If "false", throw warning. If "true", raise RelationalError. If "silent", do nothing. Defaults to "false".

  • warnings_are_errors (bool) – If True all warnings will be regarded as errors. Defaults to False.

  • hide_warnings (bool) – If True hide all warnings. Defaults to False.

  • dataset_offset (int) – Starting position in the file of the dataset to be read. Defaults to 0.

Return type:

tuple[CoreTEXT2_0 | CoreTEXT3_0 | CoreTEXT3_1 | CoreTEXT3_2, StdTEXTOutput]

Raises:
pyreflow.api.fcs_read_flat_dataset(path, text_correction=(0, 0), data_correction=(0, 0), analysis_correction=(0, 0), other_corrections=[], max_other=None, other_width=8, guess_other_width='none', squish_offsets=False, allow_pseudoempty=False, dataset_overflow_limit=0, overlap_correction_limit=0, version_override=None, supp_text_correction=(0, 0), nextdata_correction=0, allow_duplicated_supp_text='false', ignore_supp_text=False, delim_escape_mode='escaped', allow_non_ascii_delim='false', allow_nonunique='false', allow_even_delims='false', allow_odd_tokens='false', allow_empty_keys='false', allow_delim_at_boundary='false', use_encoding='utf8', allow_non_ascii_keys='false', allow_non_utf8_values='false', allow_missing_supp_text='false', allow_supp_text_own_delim='false', allow_missing_nextdata='false', trim_value_whitespace='notrim', ignore_standard_keys=[], promote_to_standard=[], demote_from_standard=[], rename_standard_keys={}, replace_standard_key_values={}, append_standard_keywords={}, substitute_standard_key_values={}, allow_repair_non_unique='false', text_data_correction=(0, 0), text_analysis_correction=(0, 0), ignore_text_data_offsets=False, ignore_text_analysis_offsets=False, allow_header_text_offset_mismatch='error', allow_missing_required_offsets='false', process_optional_failure='error', int_width_override='never', byteord_override='none', disallow_range_truncation='false', data_remainder_limit=0, allow_uneven_event_width='false', allow_tot_mismatch='false', over_bitmask_action='trunc_warn', over_range_action='warn', allow_missing_crc='false', allow_mismatch_crc='false', compute_crc='never', read_intra_segment_dark_bytes=False, read_post_dataset_dark_bytes=False, row_buffer_size=28000, warnings_are_errors=False, hide_warnings=False, dataset_offset=0, scan=False)#

Read one dataset from FCS file in flat mode.

Parameters:

  • path (Path) – Path to be read.

  • text_correction (OffsetCorrection) – Corrections for Primary TEXT offsets in HEADER. Defaults to (0, 0).

  • data_correction (OffsetCorrection) – Corrections for DATA offsets in HEADER. Defaults to (0, 0).

  • analysis_correction (OffsetCorrection) – Corrections for ANALYSIS offsets in HEADER. Defaults to (0, 0).

  • other_corrections (list[OffsetCorrection]) – Corrections for OTHER offsets if they exist. Each correction will be applied in order. If an offset does not need to be corrected, use (0,0). This will not affect the number of OTHER segments that are read; this is controlled by max_other. Defaults to [].

  • max_other (int | None) – Maximum number of OTHER segments that can be parsed. None means limitless. Defaults to None.

  • other_width (int) – Width (in bytes) to use when parsing OTHER offsets. Defaults to 8.

  • guess_other_width (GuessOtherWidth) – Guess the width of OTHER segments. Non-fatal failure to guess width will fall back to 8 or whatever was given in other_width Defaults to "none".

  • squish_offsets (bool) – If True and a segment’s ending offset is zero, treat entire offset as empty. This might happen if the ending offset is longer than 8 digits, in which case it must be written in TEXT. If this happens, the standards mandate that both offsets be written to TEXT and that the HEADER offsets be set to 0,0, so only writing one is an error unless this flag is set. This should only happen in FCS 3.0 files and above. Defaults to False.

  • allow_pseudoempty (bool) – If True, allow offsets like X,X-1. Some files will denote an “empty” offset as 0,-1 or 1000,999, which is logically correct since the last offset points to the last byte, thus 0,0 is actually 1 byte long. If this flat is set, such offsets will be treated as if they were 0,0. Defaults to False.

  • dataset_overflow_limit (int) – Limit by which offsets can be truncated if they exceed end of file or $NEXTDATA. Defaults to 0.

  • overlap_correction_limit (int) – Limit by which ending segment offset can be truncated if they overlap another offset. Defaults to 0.

  • version_override (VersionOverride | None) – Override the FCS version as seen in HEADER. Defaults to None.

  • supp_text_correction (OffsetCorrection) – Corrections for Supplemental TEXT offsets in TEXT. Defaults to (0, 0).

  • nextdata_correction (int) – Correction for $NEXTDATA. Defaults to 0.

  • allow_duplicated_supp_text (TriFlag) – Choose what happens if supplemental TEXT offsets overlap the primary TEXT offsets from HEADER or HEADER. The offsets will not be used if an overlap is found. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • ignore_supp_text (bool) – If True, ignore supplemental TEXT entirely. Defaults to False.

  • delim_escape_mode (DelimEscapeMode) – Determine how to escape delims in TEXT. Defaults to "escaped".

  • allow_non_ascii_delim (TriFlag) – Choose how to handle non-ASCII delimiters (outside 1-126). If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_nonunique (TriFlag) – Choose how to handle non-unique keys in TEXT. In such cases, only the first will be used regardless of this setting. If "false", raise ParseKeyError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_even_delims (TriFlag) – Choose what happens if TEXT has an even number of delimiters. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_odd_tokens (TriFlag) – Choose what happens if TEXT contains an odd number of tokens. The last ‘dangling’ token will be dropped regardless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_empty_keys (TriFlag) – Choose what happens if any keys are blank. Only relevant if if delimiters are unescaped. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_delim_at_boundary (TriFlag) – Choose what happens if there are delimiters at token boundaries. The FCS standard forbids this because it is impossible to tell if such delimiters belong to the previous or the next token. Consequently, delimiters at boundaries will be dropped regardless of this flag. Only relevant if delimiters are escaped. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • use_encoding (UseEncoding) – Choose how to interpret characters in TEXT. Defaults to "utf8".

  • allow_non_ascii_keys (TriFlag) – Choose how to handle non-ASCII keys. This only applies to non-standard keywords, as all standardized keywords may only contain letters, numbers, and start with "$". Regardless, all compliant keys must only have ASCII. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_non_utf8_values (TriFlag) – Choose what happens if non-UTF8 characters are in TEXT. Tokens with such characters will be dropped regardless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_missing_supp_text (TriFlag) – Choose how to handle supplemental missing TEXT offsets in primary TEXT. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_supp_text_own_delim (TriFlag) – Choose what happens if supplemental TEXT has a different delimiter compared to primary TEXT. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_missing_nextdata (TriFlag) – Choose how to handle missing $NEXTDATA. This is a required keyword in all versions. However, most files only have one dataset in which case this keyword is meaningless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • trim_value_whitespace (TrimValueWhitespace) – Trim whitespace from beginning and end of all values. This may create blank values if the starting string is entirely whitespace. Defaults to "notrim".

  • ignore_standard_keys (AppendableSelector[KeyPatterns]) – Remove standard keys from TEXT. The leading "$" is implied so do not include it. Defaults to [].

  • promote_to_standard (AppendableSelector[KeyPatterns]) – Promote nonstandard keys to standard keys in TEXT. Defaults to [].

  • demote_from_standard (AppendableSelector[KeyPatterns]) – Demote nonstandard keys from standard keys in TEXT. Defaults to [].

  • rename_standard_keys (AppendableSelector[KeyStringPairs]) – Rename standard keys in TEXT. Keys matching the first part of the pair will be replaced by the second. Comparisons are case insensitive. The leading "$" is implied so do not include it. Defaults to {}.

  • replace_standard_key_values (AppendableSelector[KeyStringValues]) – Replace values for standard keys in TEXT. Comparisons are case insensitive. The leading "$" is implied so do not include it. Defaults to {}.

  • append_standard_keywords (AppendableSelector[KeyStringValues]) – Append standard key/value pairs to TEXT. All keys and values will be included as they appear here. The leading "$" is implied so do not include it. Defaults to {}.

  • substitute_standard_key_values (AppendableSelector[SubPatterns]) – Apply sed-like substitution operation on matching standard keys. The leading "$" is implied when matching keys. Defaults to {}.

  • allow_repair_non_unique (TriFlag) – Choose how to handle key collisions when repairing keywords. Non-unique keywords will not be kept in the final FCS file since each list of standard and non-standard keywords must be unique. If "false", raise ConfigError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • text_data_correction (OffsetCorrection) – Corrections for DATA offsets in TEXT. Defaults to (0, 0).

  • text_analysis_correction (OffsetCorrection) – Corrections for ANALYSIS offsets in TEXT. Defaults to (0, 0).

  • ignore_text_data_offsets (bool) – If True ignore DATA offsets in TEXT Defaults to False.

  • ignore_text_analysis_offsets (bool) – If True ignore ANALYSIS offsets in TEXT Defaults to False.

  • allow_header_text_offset_mismatch (AllowHeaderTextOffsetMismatch) – Choose what to do if HEADER and TEXT offsets are different. Exception will be FileLayoutError if emitted. Defaults to "error".

  • allow_missing_required_offsets (TriFlag) – Choose what happens when required DATA and ANALYSIS (3.1 or lower) offsets in TEXT are be missing. If missing, fall back to offsets from HEADER. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • process_optional_failure (ProcessKeywordFailure) – Process optional keys which cause an error. Defaults to "error".

  • int_width_override (IntWidthOverride) – Override $PnB. Only affects integer layouts in FCS 2.0/3.0. Defaults to "never".

  • byteord_override (ByteordOverride) – Override $BYTEORD. Only affects integer layouts in FCS 2.0/3.0. Defaults to "none".

  • disallow_range_truncation (TriFlag) – Choose how to handle $PnR values that need to be truncated to match the number of bytes specified by $PnB and $DATATYPE. If "false", throw warning. If "true", raise RelationalError. If "silent", do nothing. Defaults to "false".

  • data_remainder_limit (int) – Limit by which ending DATA offset can be truncated if its length modulo event width produces a remainder. Defaults to 0.

  • allow_uneven_event_width (TriFlag) – Choose what to do when event width does not perfectly divide length of DATA. Does not apply to delimited ASCII data schema. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_tot_mismatch (TriFlag) – Choose what happens when $TOT does not match number of events as computed by the event width and length of DATA. Does not apply to delimited ASCII data schema. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • over_bitmask_action (OverLimitAction) – Choose what to do with event integer values in DATA which exceed bitmask. Defaults to "trunc_warn".

  • over_range_action (OverLimitAction) – Choose what to do with event values in DATA which exceed $PnR. Defaults to "warn".

  • allow_missing_crc (TriFlag) – Choose what to do when CRC is missing from the end of a dataset. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_mismatch_crc (TriFlag) – Choose what to do when computed CRC and CRC at the end of a dataset do not match. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • compute_crc (ComputeCRC) – Choose when to compute the CRC for a dataset. Defaults to "never".

  • read_intra_segment_dark_bytes (bool) – If True read bytes which are between segments. Defaults to False.

  • read_post_dataset_dark_bytes (bool) – If True read bytes between the end of the current dataset and the next. Defaults to False.

  • row_buffer_size (int) – Set the size in bytes for the internal buffer used to read DATA. This is a performance parameter that balances read syscalls (too low) and cache misses (too high). It should generally be 90% of the CPU’s L1D cache size. Defaults to 28000.

  • warnings_are_errors (bool) – If True all warnings will be regarded as errors. Defaults to False.

  • hide_warnings (bool) – If True hide all warnings. Defaults to False.

  • dataset_offset (int) – Starting position in the file of the dataset to be read. Defaults to 0.

  • scan (bool) – If True, scan through file to find next dataset based on version tags rather than relying on $NEXTDATA. Defaults to False.

Return type:

FlatDatasetOutput

Raises:
pyreflow.api.fcs_read_std_dataset(path, text_correction=(0, 0), data_correction=(0, 0), analysis_correction=(0, 0), other_corrections=[], max_other=None, other_width=8, guess_other_width='none', squish_offsets=False, allow_pseudoempty=False, dataset_overflow_limit=0, overlap_correction_limit=0, version_override=None, supp_text_correction=(0, 0), nextdata_correction=0, allow_duplicated_supp_text='false', ignore_supp_text=False, delim_escape_mode='escaped', allow_non_ascii_delim='false', allow_nonunique='false', allow_even_delims='false', allow_odd_tokens='false', allow_empty_keys='false', allow_delim_at_boundary='false', use_encoding='utf8', allow_non_ascii_keys='false', allow_non_utf8_values='false', allow_missing_supp_text='false', allow_supp_text_own_delim='false', allow_missing_nextdata='false', trim_value_whitespace='notrim', dedup_measurement_names=False, trim_intra_value_whitespace=False, time_meas_pattern='^(TIME|Time)$', allow_missing_time='false', force_linear_scale='none', ignore_optical_only_keys=[], process_optical_only_keys='demote_warn', date_pattern=None, time_pattern=None, datetime_pattern=None, last_modified_pattern=None, allow_other_feature=False, process_pseudostandard='error', process_hyper_par='error', process_other_version='error', process_extra_timestep='error', fix_log_scale_offsets=False, add_missing_timestep=None, spillover_measurement_mode='named', disallow_localtime=False, ignore_standard_keys=[], promote_to_standard=[], demote_from_standard=[], rename_standard_keys={}, replace_standard_key_values={}, append_standard_keywords={}, substitute_standard_key_values={}, allow_repair_non_unique='false', text_data_correction=(0, 0), text_analysis_correction=(0, 0), ignore_text_data_offsets=False, ignore_text_analysis_offsets=False, allow_header_text_offset_mismatch='error', allow_missing_required_offsets='false', process_optional_failure='error', int_width_override='never', byteord_override='none', disallow_range_truncation='false', data_remainder_limit=0, allow_uneven_event_width='false', allow_tot_mismatch='false', over_bitmask_action='trunc_warn', over_range_action='warn', allow_missing_crc='false', allow_mismatch_crc='false', compute_crc='never', read_intra_segment_dark_bytes=False, read_post_dataset_dark_bytes=False, row_buffer_size=28000, warnings_are_errors=False, hide_warnings=False, dataset_offset=0, scan=False)#

Read one standardized dataset from FCS file.

Parameters:

  • path (Path) – Path to be read.

  • text_correction (OffsetCorrection) – Corrections for Primary TEXT offsets in HEADER. Defaults to (0, 0).

  • data_correction (OffsetCorrection) – Corrections for DATA offsets in HEADER. Defaults to (0, 0).

  • analysis_correction (OffsetCorrection) – Corrections for ANALYSIS offsets in HEADER. Defaults to (0, 0).

  • other_corrections (list[OffsetCorrection]) – Corrections for OTHER offsets if they exist. Each correction will be applied in order. If an offset does not need to be corrected, use (0,0). This will not affect the number of OTHER segments that are read; this is controlled by max_other. Defaults to [].

  • max_other (int | None) – Maximum number of OTHER segments that can be parsed. None means limitless. Defaults to None.

  • other_width (int) – Width (in bytes) to use when parsing OTHER offsets. Defaults to 8.

  • guess_other_width (GuessOtherWidth) – Guess the width of OTHER segments. Non-fatal failure to guess width will fall back to 8 or whatever was given in other_width Defaults to "none".

  • squish_offsets (bool) – If True and a segment’s ending offset is zero, treat entire offset as empty. This might happen if the ending offset is longer than 8 digits, in which case it must be written in TEXT. If this happens, the standards mandate that both offsets be written to TEXT and that the HEADER offsets be set to 0,0, so only writing one is an error unless this flag is set. This should only happen in FCS 3.0 files and above. Defaults to False.

  • allow_pseudoempty (bool) – If True, allow offsets like X,X-1. Some files will denote an “empty” offset as 0,-1 or 1000,999, which is logically correct since the last offset points to the last byte, thus 0,0 is actually 1 byte long. If this flat is set, such offsets will be treated as if they were 0,0. Defaults to False.

  • dataset_overflow_limit (int) – Limit by which offsets can be truncated if they exceed end of file or $NEXTDATA. Defaults to 0.

  • overlap_correction_limit (int) – Limit by which ending segment offset can be truncated if they overlap another offset. Defaults to 0.

  • version_override (VersionOverride | None) – Override the FCS version as seen in HEADER. Defaults to None.

  • supp_text_correction (OffsetCorrection) – Corrections for Supplemental TEXT offsets in TEXT. Defaults to (0, 0).

  • nextdata_correction (int) – Correction for $NEXTDATA. Defaults to 0.

  • allow_duplicated_supp_text (TriFlag) – Choose what happens if supplemental TEXT offsets overlap the primary TEXT offsets from HEADER or HEADER. The offsets will not be used if an overlap is found. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • ignore_supp_text (bool) – If True, ignore supplemental TEXT entirely. Defaults to False.

  • delim_escape_mode (DelimEscapeMode) – Determine how to escape delims in TEXT. Defaults to "escaped".

  • allow_non_ascii_delim (TriFlag) – Choose how to handle non-ASCII delimiters (outside 1-126). If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_nonunique (TriFlag) – Choose how to handle non-unique keys in TEXT. In such cases, only the first will be used regardless of this setting. If "false", raise ParseKeyError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_even_delims (TriFlag) – Choose what happens if TEXT has an even number of delimiters. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_odd_tokens (TriFlag) – Choose what happens if TEXT contains an odd number of tokens. The last ‘dangling’ token will be dropped regardless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_empty_keys (TriFlag) – Choose what happens if any keys are blank. Only relevant if if delimiters are unescaped. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_delim_at_boundary (TriFlag) – Choose what happens if there are delimiters at token boundaries. The FCS standard forbids this because it is impossible to tell if such delimiters belong to the previous or the next token. Consequently, delimiters at boundaries will be dropped regardless of this flag. Only relevant if delimiters are escaped. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • use_encoding (UseEncoding) – Choose how to interpret characters in TEXT. Defaults to "utf8".

  • allow_non_ascii_keys (TriFlag) – Choose how to handle non-ASCII keys. This only applies to non-standard keywords, as all standardized keywords may only contain letters, numbers, and start with "$". Regardless, all compliant keys must only have ASCII. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_non_utf8_values (TriFlag) – Choose what happens if non-UTF8 characters are in TEXT. Tokens with such characters will be dropped regardless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_missing_supp_text (TriFlag) – Choose how to handle supplemental missing TEXT offsets in primary TEXT. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_supp_text_own_delim (TriFlag) – Choose what happens if supplemental TEXT has a different delimiter compared to primary TEXT. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_missing_nextdata (TriFlag) – Choose how to handle missing $NEXTDATA. This is a required keyword in all versions. However, most files only have one dataset in which case this keyword is meaningless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • trim_value_whitespace (TrimValueWhitespace) – Trim whitespace from beginning and end of all values. This may create blank values if the starting string is entirely whitespace. Defaults to "notrim".

  • dedup_measurement_names (bool) – If True, force all $PnN to be unique by appending "~X" to each duplicate and incrementing X starting at 0. Defaults to False.

  • trim_intra_value_whitespace (bool) – If True, trim whitespace between delimiters such as "," and ";" within keyword value strings. Defaults to False.

  • time_meas_pattern (Selector[str | None]) – A pattern to match the $PnN of the time measurement. If "NoTime", do not try to find a time measurement. Defaults to "^(TIME|Time)$".

  • allow_missing_time (TriFlag) – Choose what to do when time measurement is be missing. If "false", raise RelationalError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • force_linear_scale (ForceLinearScale) – Force $PnE to be linear for certain measurements. Affected measurements will never fail. Defaults to "none".

  • ignore_optical_only_keys (list[OpticalOnlyKey]) – Ignore optical keys in temporal measurement. These keys are $PnG which is explicitly forbidden by the standard but allowed in this library to be set to 1.0 (noop), or others which are nonsensical for time measurements but are not explicitly forbidden in the the standard (such as $PnL). Provided keys are the string after the "Pn" in the "PnX" keywords. Defaults to [].

  • process_optical_only_keys (ProcessOpticalOnlyKeys) – Choose how to handle optical keys found in temporal measurements. Does nothing unless keys are specified in ignore_optical_only_keys. Defaults to "demote_warn".

  • date_pattern (Selector[str | None]) – If supplied, will be used as an alternative pattern when parsing $DATE. If not supplied, $DATE will be parsed according to the standard pattern which is "%d-%b-%Y". Defaults to None.

  • time_pattern (Selector[str | None]) – If supplied, will be used as an alternative pattern when parsing $BTIM and $ETIM. The values "%!" or "%@" may be used to match 1/60 seconds or centiseconds respectively. If not supplied, $BTIM and $ETIM will be parsed according to the standard pattern which is "%H:%M:%S" for 2.0, "%H:%M:%S:%!" for 3.0 and "%H:%M:%S.%@" for 3.1 and up. Defaults to None.

  • datetime_pattern (Selector[str | None]) – If supplied, will be used as an alternative pattern when parsing $BEGINDATETIME and $ENDDATETIME. The pattern must follow the format outlined in chrono. If not supplied, these will be parsed as ISO timestamps with optional timezone. Defaults to None.

  • last_modified_pattern (Selector[str | None]) – If supplied, will be used as an alternative pattern when parsing $LAST_MODIFIED. The pattern must follow the format outlined in chrono. If not supplied, these will be parsed according to the default pattern which is "%d-%b-%Y %H:%M:%S" possibly with centiseconds after. Defaults to None.

  • allow_other_feature (bool) – If True, allow $PnFEATURE to be a value other than "Area", "Width", or "Height". Defaults to False.

  • process_pseudostandard (ProcessKeywordFailure) – Process non-standard keywords with a leading "$". The presence of such keywords often means the version in HEADER is incorrect. Defaults to "error".

  • process_hyper_par (ProcessKeywordFailure) – Process measurement keywords whose index is greater than $PAR. Defaults to "error".

  • process_other_version (ProcessKeywordFailure) – Process standard keywords from different FCS versions. Defaults to "error".

  • process_extra_timestep (ProcessKeywordFailure) – Process $TIMESTEP to be present which may indicate a time measurement is present but not identified. Defaults to "error".

  • fix_log_scale_offsets (bool) – If True fix log-scale $PnE and keywords which have zero offset (ie <X>,0.0 where X is non-zero). Defaults to False.

  • add_missing_timestep (Timestep | None) – Set $TIMESTEP if it is not present and required. This will do nothing on FCS2.0 files since this version does not specify $TIMESTEP. Defaults to None.

  • spillover_measurement_mode (SpilloverMeasurementMode) – Choose how to interpret measurement strings in $SPILLOVER. Defaults to "named".

  • disallow_localtime (bool) – If True, require that $BEGINDATETIME and $ENDDATETIME have a timezone if provided. This is not required by the standard, but not having a timezone is ambiguous since the absolute value of the timestamp is dependent on localtime and therefore is location-dependent. Only affects FCS 3.2. Defaults to False.

  • ignore_standard_keys (AppendableSelector[KeyPatterns]) – Remove standard keys from TEXT. The leading "$" is implied so do not include it. Defaults to [].

  • promote_to_standard (AppendableSelector[KeyPatterns]) – Promote nonstandard keys to standard keys in TEXT. Defaults to [].

  • demote_from_standard (AppendableSelector[KeyPatterns]) – Demote nonstandard keys from standard keys in TEXT. Defaults to [].

  • rename_standard_keys (AppendableSelector[KeyStringPairs]) – Rename standard keys in TEXT. Keys matching the first part of the pair will be replaced by the second. Comparisons are case insensitive. The leading "$" is implied so do not include it. Defaults to {}.

  • replace_standard_key_values (AppendableSelector[KeyStringValues]) – Replace values for standard keys in TEXT. Comparisons are case insensitive. The leading "$" is implied so do not include it. Defaults to {}.

  • append_standard_keywords (AppendableSelector[KeyStringValues]) – Append standard key/value pairs to TEXT. All keys and values will be included as they appear here. The leading "$" is implied so do not include it. Defaults to {}.

  • substitute_standard_key_values (AppendableSelector[SubPatterns]) – Apply sed-like substitution operation on matching standard keys. The leading "$" is implied when matching keys. Defaults to {}.

  • allow_repair_non_unique (TriFlag) – Choose how to handle key collisions when repairing keywords. Non-unique keywords will not be kept in the final FCS file since each list of standard and non-standard keywords must be unique. If "false", raise ConfigError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • text_data_correction (OffsetCorrection) – Corrections for DATA offsets in TEXT. Defaults to (0, 0).

  • text_analysis_correction (OffsetCorrection) – Corrections for ANALYSIS offsets in TEXT. Defaults to (0, 0).

  • ignore_text_data_offsets (bool) – If True ignore DATA offsets in TEXT Defaults to False.

  • ignore_text_analysis_offsets (bool) – If True ignore ANALYSIS offsets in TEXT Defaults to False.

  • allow_header_text_offset_mismatch (AllowHeaderTextOffsetMismatch) – Choose what to do if HEADER and TEXT offsets are different. Exception will be FileLayoutError if emitted. Defaults to "error".

  • allow_missing_required_offsets (TriFlag) – Choose what happens when required DATA and ANALYSIS (3.1 or lower) offsets in TEXT are be missing. If missing, fall back to offsets from HEADER. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • process_optional_failure (ProcessKeywordFailure) – Process optional keys which cause an error. Defaults to "error".

  • int_width_override (IntWidthOverride) – Override $PnB. Only affects integer layouts in FCS 2.0/3.0. Defaults to "never".

  • byteord_override (ByteordOverride) – Override $BYTEORD. Only affects integer layouts in FCS 2.0/3.0. Defaults to "none".

  • disallow_range_truncation (TriFlag) – Choose how to handle $PnR values that need to be truncated to match the number of bytes specified by $PnB and $DATATYPE. If "false", throw warning. If "true", raise RelationalError. If "silent", do nothing. Defaults to "false".

  • data_remainder_limit (int) – Limit by which ending DATA offset can be truncated if its length modulo event width produces a remainder. Defaults to 0.

  • allow_uneven_event_width (TriFlag) – Choose what to do when event width does not perfectly divide length of DATA. Does not apply to delimited ASCII data schema. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_tot_mismatch (TriFlag) – Choose what happens when $TOT does not match number of events as computed by the event width and length of DATA. Does not apply to delimited ASCII data schema. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • over_bitmask_action (OverLimitAction) – Choose what to do with event integer values in DATA which exceed bitmask. Defaults to "trunc_warn".

  • over_range_action (OverLimitAction) – Choose what to do with event values in DATA which exceed $PnR. Defaults to "warn".

  • allow_missing_crc (TriFlag) – Choose what to do when CRC is missing from the end of a dataset. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_mismatch_crc (TriFlag) – Choose what to do when computed CRC and CRC at the end of a dataset do not match. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • compute_crc (ComputeCRC) – Choose when to compute the CRC for a dataset. Defaults to "never".

  • read_intra_segment_dark_bytes (bool) – If True read bytes which are between segments. Defaults to False.

  • read_post_dataset_dark_bytes (bool) – If True read bytes between the end of the current dataset and the next. Defaults to False.

  • row_buffer_size (int) – Set the size in bytes for the internal buffer used to read DATA. This is a performance parameter that balances read syscalls (too low) and cache misses (too high). It should generally be 90% of the CPU’s L1D cache size. Defaults to 28000.

  • warnings_are_errors (bool) – If True all warnings will be regarded as errors. Defaults to False.

  • hide_warnings (bool) – If True hide all warnings. Defaults to False.

  • dataset_offset (int) – Starting position in the file of the dataset to be read. Defaults to 0.

  • scan (bool) – If True, scan through file to find next dataset based on version tags rather than relying on $NEXTDATA. Defaults to False.

Return type:

tuple[CoreDataset2_0 | CoreDataset3_0 | CoreDataset3_1 | CoreDataset3_2, StdDatasetOutput]

Raises:

  • ConfigError – if other_width is less than 8 and greater than 20

  • FileLayoutError – If HEADER, TEXT, or DATA are unparsable

  • ParseKeywordValueError – If any keyword values could not be read from their string encoding

  • RelationalError – If keywords are incompatible with indicated layout of DATA or if keywords that are referenced by other keywords do not exist

  • EventDataError – If values in DATA cannot be read

  • ExtraKeywordError – If any standard keys are unused and not dropped by some other option

pyreflow.api.fcs_read_flat_texts(path, skip=None, limit=None, scan=False, text_correction=(0, 0), data_correction=(0, 0), analysis_correction=(0, 0), other_corrections=[], max_other=None, other_width=8, guess_other_width='none', squish_offsets=False, allow_pseudoempty=False, dataset_overflow_limit=0, overlap_correction_limit=0, version_override=None, supp_text_correction=(0, 0), nextdata_correction=0, allow_duplicated_supp_text='false', ignore_supp_text=False, delim_escape_mode='escaped', allow_non_ascii_delim='false', allow_nonunique='false', allow_even_delims='false', allow_odd_tokens='false', allow_empty_keys='false', allow_delim_at_boundary='false', use_encoding='utf8', allow_non_ascii_keys='false', allow_non_utf8_values='false', allow_missing_supp_text='false', allow_supp_text_own_delim='false', allow_missing_nextdata='false', trim_value_whitespace='notrim', warnings_are_errors=False, hide_warnings=False)#

Read HEADER and TEXT from multiple datasets in FCS file.

Parameters:

  • path (Path) – Path to be read.

  • skip (int | None) – Number of datasets to skip Defaults to None.

  • limit (int | None) – Parse up to this many datasets Defaults to None.

  • scan (bool) – If True, scan through file to find next dataset based on version tags rather than relying on $NEXTDATA. Defaults to False.

  • text_correction (OffsetCorrection) – Corrections for Primary TEXT offsets in HEADER. Defaults to (0, 0).

  • data_correction (OffsetCorrection) – Corrections for DATA offsets in HEADER. Defaults to (0, 0).

  • analysis_correction (OffsetCorrection) – Corrections for ANALYSIS offsets in HEADER. Defaults to (0, 0).

  • other_corrections (list[OffsetCorrection]) – Corrections for OTHER offsets if they exist. Each correction will be applied in order. If an offset does not need to be corrected, use (0,0). This will not affect the number of OTHER segments that are read; this is controlled by max_other. Defaults to [].

  • max_other (int | None) – Maximum number of OTHER segments that can be parsed. None means limitless. Defaults to None.

  • other_width (int) – Width (in bytes) to use when parsing OTHER offsets. Defaults to 8.

  • guess_other_width (GuessOtherWidth) – Guess the width of OTHER segments. Non-fatal failure to guess width will fall back to 8 or whatever was given in other_width Defaults to "none".

  • squish_offsets (bool) – If True and a segment’s ending offset is zero, treat entire offset as empty. This might happen if the ending offset is longer than 8 digits, in which case it must be written in TEXT. If this happens, the standards mandate that both offsets be written to TEXT and that the HEADER offsets be set to 0,0, so only writing one is an error unless this flag is set. This should only happen in FCS 3.0 files and above. Defaults to False.

  • allow_pseudoempty (bool) – If True, allow offsets like X,X-1. Some files will denote an “empty” offset as 0,-1 or 1000,999, which is logically correct since the last offset points to the last byte, thus 0,0 is actually 1 byte long. If this flat is set, such offsets will be treated as if they were 0,0. Defaults to False.

  • dataset_overflow_limit (int) – Limit by which offsets can be truncated if they exceed end of file or $NEXTDATA. Defaults to 0.

  • overlap_correction_limit (int) – Limit by which ending segment offset can be truncated if they overlap another offset. Defaults to 0.

  • version_override (VersionOverride | None) – Override the FCS version as seen in HEADER. Defaults to None.

  • supp_text_correction (OffsetCorrection) – Corrections for Supplemental TEXT offsets in TEXT. Defaults to (0, 0).

  • nextdata_correction (int) – Correction for $NEXTDATA. Defaults to 0.

  • allow_duplicated_supp_text (TriFlag) – Choose what happens if supplemental TEXT offsets overlap the primary TEXT offsets from HEADER or HEADER. The offsets will not be used if an overlap is found. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • ignore_supp_text (bool) – If True, ignore supplemental TEXT entirely. Defaults to False.

  • delim_escape_mode (DelimEscapeMode) – Determine how to escape delims in TEXT. Defaults to "escaped".

  • allow_non_ascii_delim (TriFlag) – Choose how to handle non-ASCII delimiters (outside 1-126). If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_nonunique (TriFlag) – Choose how to handle non-unique keys in TEXT. In such cases, only the first will be used regardless of this setting. If "false", raise ParseKeyError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_even_delims (TriFlag) – Choose what happens if TEXT has an even number of delimiters. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_odd_tokens (TriFlag) – Choose what happens if TEXT contains an odd number of tokens. The last ‘dangling’ token will be dropped regardless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_empty_keys (TriFlag) – Choose what happens if any keys are blank. Only relevant if if delimiters are unescaped. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_delim_at_boundary (TriFlag) – Choose what happens if there are delimiters at token boundaries. The FCS standard forbids this because it is impossible to tell if such delimiters belong to the previous or the next token. Consequently, delimiters at boundaries will be dropped regardless of this flag. Only relevant if delimiters are escaped. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • use_encoding (UseEncoding) – Choose how to interpret characters in TEXT. Defaults to "utf8".

  • allow_non_ascii_keys (TriFlag) – Choose how to handle non-ASCII keys. This only applies to non-standard keywords, as all standardized keywords may only contain letters, numbers, and start with "$". Regardless, all compliant keys must only have ASCII. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_non_utf8_values (TriFlag) – Choose what happens if non-UTF8 characters are in TEXT. Tokens with such characters will be dropped regardless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_missing_supp_text (TriFlag) – Choose how to handle supplemental missing TEXT offsets in primary TEXT. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_supp_text_own_delim (TriFlag) – Choose what happens if supplemental TEXT has a different delimiter compared to primary TEXT. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_missing_nextdata (TriFlag) – Choose how to handle missing $NEXTDATA. This is a required keyword in all versions. However, most files only have one dataset in which case this keyword is meaningless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • trim_value_whitespace (TrimValueWhitespace) – Trim whitespace from beginning and end of all values. This may create blank values if the starting string is entirely whitespace. Defaults to "notrim".

  • warnings_are_errors (bool) – If True all warnings will be regarded as errors. Defaults to False.

  • hide_warnings (bool) – If True hide all warnings. Defaults to False.

Return type:

list[FlatTEXTOutput]

Raises:
pyreflow.api.fcs_read_std_texts(path, skip=None, limit=None, scan=False, text_correction=(0, 0), data_correction=(0, 0), analysis_correction=(0, 0), other_corrections=[], max_other=None, other_width=8, guess_other_width='none', squish_offsets=False, allow_pseudoempty=False, dataset_overflow_limit=0, overlap_correction_limit=0, version_override=None, supp_text_correction=(0, 0), nextdata_correction=0, allow_duplicated_supp_text='false', ignore_supp_text=False, delim_escape_mode='escaped', allow_non_ascii_delim='false', allow_nonunique='false', allow_even_delims='false', allow_odd_tokens='false', allow_empty_keys='false', allow_delim_at_boundary='false', use_encoding='utf8', allow_non_ascii_keys='false', allow_non_utf8_values='false', allow_missing_supp_text='false', allow_supp_text_own_delim='false', allow_missing_nextdata='false', trim_value_whitespace='notrim', dedup_measurement_names=False, trim_intra_value_whitespace=False, time_meas_pattern='^(TIME|Time)$', allow_missing_time='false', force_linear_scale='none', ignore_optical_only_keys=[], process_optical_only_keys='demote_warn', date_pattern=None, time_pattern=None, datetime_pattern=None, last_modified_pattern=None, allow_other_feature=False, process_pseudostandard='error', process_hyper_par='error', process_other_version='error', process_extra_timestep='error', fix_log_scale_offsets=False, add_missing_timestep=None, spillover_measurement_mode='named', disallow_localtime=False, ignore_standard_keys=[], promote_to_standard=[], demote_from_standard=[], rename_standard_keys={}, replace_standard_key_values={}, append_standard_keywords={}, substitute_standard_key_values={}, allow_repair_non_unique='false', text_data_correction=(0, 0), text_analysis_correction=(0, 0), ignore_text_data_offsets=False, ignore_text_analysis_offsets=False, allow_header_text_offset_mismatch='error', allow_missing_required_offsets='false', process_optional_failure='error', int_width_override='never', byteord_override='none', disallow_range_truncation='false', warnings_are_errors=False, hide_warnings=False)#

Read standardized TEXT from multiple datasets in FCS file.

Parameters:

  • path (Path) – Path to be read.

  • skip (int | None) – Number of datasets to skip. The HEADER and TEXT from skipped datasets will still be read to obtain $NEXTDATA for the next dataset in the file. Defaults to None.

  • limit (int | None) – Parse up to this many datasets Defaults to None.

  • scan (bool) – If True, scan through file to find next dataset based on version tags rather than relying on $NEXTDATA. Defaults to False.

  • text_correction (OffsetCorrection) – Corrections for Primary TEXT offsets in HEADER. Defaults to (0, 0).

  • data_correction (OffsetCorrection) – Corrections for DATA offsets in HEADER. Defaults to (0, 0).

  • analysis_correction (OffsetCorrection) – Corrections for ANALYSIS offsets in HEADER. Defaults to (0, 0).

  • other_corrections (list[OffsetCorrection]) – Corrections for OTHER offsets if they exist. Each correction will be applied in order. If an offset does not need to be corrected, use (0,0). This will not affect the number of OTHER segments that are read; this is controlled by max_other. Defaults to [].

  • max_other (int | None) – Maximum number of OTHER segments that can be parsed. None means limitless. Defaults to None.

  • other_width (int) – Width (in bytes) to use when parsing OTHER offsets. Defaults to 8.

  • guess_other_width (GuessOtherWidth) – Guess the width of OTHER segments. Non-fatal failure to guess width will fall back to 8 or whatever was given in other_width Defaults to "none".

  • squish_offsets (bool) – If True and a segment’s ending offset is zero, treat entire offset as empty. This might happen if the ending offset is longer than 8 digits, in which case it must be written in TEXT. If this happens, the standards mandate that both offsets be written to TEXT and that the HEADER offsets be set to 0,0, so only writing one is an error unless this flag is set. This should only happen in FCS 3.0 files and above. Defaults to False.

  • allow_pseudoempty (bool) – If True, allow offsets like X,X-1. Some files will denote an “empty” offset as 0,-1 or 1000,999, which is logically correct since the last offset points to the last byte, thus 0,0 is actually 1 byte long. If this flat is set, such offsets will be treated as if they were 0,0. Defaults to False.

  • dataset_overflow_limit (int) – Limit by which offsets can be truncated if they exceed end of file or $NEXTDATA. Defaults to 0.

  • overlap_correction_limit (int) – Limit by which ending segment offset can be truncated if they overlap another offset. Defaults to 0.

  • version_override (VersionOverride | None) – Override the FCS version as seen in HEADER. Defaults to None.

  • supp_text_correction (OffsetCorrection) – Corrections for Supplemental TEXT offsets in TEXT. Defaults to (0, 0).

  • nextdata_correction (int) – Correction for $NEXTDATA. Defaults to 0.

  • allow_duplicated_supp_text (TriFlag) – Choose what happens if supplemental TEXT offsets overlap the primary TEXT offsets from HEADER or HEADER. The offsets will not be used if an overlap is found. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • ignore_supp_text (bool) – If True, ignore supplemental TEXT entirely. Defaults to False.

  • delim_escape_mode (DelimEscapeMode) – Determine how to escape delims in TEXT. Defaults to "escaped".

  • allow_non_ascii_delim (TriFlag) – Choose how to handle non-ASCII delimiters (outside 1-126). If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_nonunique (TriFlag) – Choose how to handle non-unique keys in TEXT. In such cases, only the first will be used regardless of this setting. If "false", raise ParseKeyError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_even_delims (TriFlag) – Choose what happens if TEXT has an even number of delimiters. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_odd_tokens (TriFlag) – Choose what happens if TEXT contains an odd number of tokens. The last ‘dangling’ token will be dropped regardless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_empty_keys (TriFlag) – Choose what happens if any keys are blank. Only relevant if if delimiters are unescaped. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_delim_at_boundary (TriFlag) – Choose what happens if there are delimiters at token boundaries. The FCS standard forbids this because it is impossible to tell if such delimiters belong to the previous or the next token. Consequently, delimiters at boundaries will be dropped regardless of this flag. Only relevant if delimiters are escaped. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • use_encoding (UseEncoding) – Choose how to interpret characters in TEXT. Defaults to "utf8".

  • allow_non_ascii_keys (TriFlag) – Choose how to handle non-ASCII keys. This only applies to non-standard keywords, as all standardized keywords may only contain letters, numbers, and start with "$". Regardless, all compliant keys must only have ASCII. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_non_utf8_values (TriFlag) – Choose what happens if non-UTF8 characters are in TEXT. Tokens with such characters will be dropped regardless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_missing_supp_text (TriFlag) – Choose how to handle supplemental missing TEXT offsets in primary TEXT. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_supp_text_own_delim (TriFlag) – Choose what happens if supplemental TEXT has a different delimiter compared to primary TEXT. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_missing_nextdata (TriFlag) – Choose how to handle missing $NEXTDATA. This is a required keyword in all versions. However, most files only have one dataset in which case this keyword is meaningless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • trim_value_whitespace (TrimValueWhitespace) – Trim whitespace from beginning and end of all values. This may create blank values if the starting string is entirely whitespace. Defaults to "notrim".

  • dedup_measurement_names (bool) – If True, force all $PnN to be unique by appending "~X" to each duplicate and incrementing X starting at 0. Defaults to False.

  • trim_intra_value_whitespace (bool) – If True, trim whitespace between delimiters such as "," and ";" within keyword value strings. Defaults to False.

  • time_meas_pattern (Selector[str | None]) – A pattern to match the $PnN of the time measurement. If "NoTime", do not try to find a time measurement. Defaults to "^(TIME|Time)$".

  • allow_missing_time (TriFlag) – Choose what to do when time measurement is be missing. If "false", raise RelationalError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • force_linear_scale (ForceLinearScale) – Force $PnE to be linear for certain measurements. Affected measurements will never fail. Defaults to "none".

  • ignore_optical_only_keys (list[OpticalOnlyKey]) – Ignore optical keys in temporal measurement. These keys are $PnG which is explicitly forbidden by the standard but allowed in this library to be set to 1.0 (noop), or others which are nonsensical for time measurements but are not explicitly forbidden in the the standard (such as $PnL). Provided keys are the string after the "Pn" in the "PnX" keywords. Defaults to [].

  • process_optical_only_keys (ProcessOpticalOnlyKeys) – Choose how to handle optical keys found in temporal measurements. Does nothing unless keys are specified in ignore_optical_only_keys. Defaults to "demote_warn".

  • date_pattern (Selector[str | None]) – If supplied, will be used as an alternative pattern when parsing $DATE. If not supplied, $DATE will be parsed according to the standard pattern which is "%d-%b-%Y". Defaults to None.

  • time_pattern (Selector[str | None]) – If supplied, will be used as an alternative pattern when parsing $BTIM and $ETIM. The values "%!" or "%@" may be used to match 1/60 seconds or centiseconds respectively. If not supplied, $BTIM and $ETIM will be parsed according to the standard pattern which is "%H:%M:%S" for 2.0, "%H:%M:%S:%!" for 3.0 and "%H:%M:%S.%@" for 3.1 and up. Defaults to None.

  • datetime_pattern (Selector[str | None]) – If supplied, will be used as an alternative pattern when parsing $BEGINDATETIME and $ENDDATETIME. The pattern must follow the format outlined in chrono. If not supplied, these will be parsed as ISO timestamps with optional timezone. Defaults to None.

  • last_modified_pattern (Selector[str | None]) – If supplied, will be used as an alternative pattern when parsing $LAST_MODIFIED. The pattern must follow the format outlined in chrono. If not supplied, these will be parsed according to the default pattern which is "%d-%b-%Y %H:%M:%S" possibly with centiseconds after. Defaults to None.

  • allow_other_feature (bool) – If True, allow $PnFEATURE to be a value other than "Area", "Width", or "Height". Defaults to False.

  • process_pseudostandard (ProcessKeywordFailure) – Process non-standard keywords with a leading "$". The presence of such keywords often means the version in HEADER is incorrect. Defaults to "error".

  • process_hyper_par (ProcessKeywordFailure) – Process measurement keywords whose index is greater than $PAR. Defaults to "error".

  • process_other_version (ProcessKeywordFailure) – Process standard keywords from different FCS versions. Defaults to "error".

  • process_extra_timestep (ProcessKeywordFailure) – Process $TIMESTEP to be present which may indicate a time measurement is present but not identified. Defaults to "error".

  • fix_log_scale_offsets (bool) – If True fix log-scale $PnE and keywords which have zero offset (ie <X>,0.0 where X is non-zero). Defaults to False.

  • add_missing_timestep (Timestep | None) – Set $TIMESTEP if it is not present and required. This will do nothing on FCS2.0 files since this version does not specify $TIMESTEP. Defaults to None.

  • spillover_measurement_mode (SpilloverMeasurementMode) – Choose how to interpret measurement strings in $SPILLOVER. Defaults to "named".

  • disallow_localtime (bool) – If True, require that $BEGINDATETIME and $ENDDATETIME have a timezone if provided. This is not required by the standard, but not having a timezone is ambiguous since the absolute value of the timestamp is dependent on localtime and therefore is location-dependent. Only affects FCS 3.2. Defaults to False.

  • ignore_standard_keys (AppendableSelector[KeyPatterns]) – Remove standard keys from TEXT. The leading "$" is implied so do not include it. Defaults to [].

  • promote_to_standard (AppendableSelector[KeyPatterns]) – Promote nonstandard keys to standard keys in TEXT. Defaults to [].

  • demote_from_standard (AppendableSelector[KeyPatterns]) – Demote nonstandard keys from standard keys in TEXT. Defaults to [].

  • rename_standard_keys (AppendableSelector[KeyStringPairs]) – Rename standard keys in TEXT. Keys matching the first part of the pair will be replaced by the second. Comparisons are case insensitive. The leading "$" is implied so do not include it. Defaults to {}.

  • replace_standard_key_values (AppendableSelector[KeyStringValues]) – Replace values for standard keys in TEXT. Comparisons are case insensitive. The leading "$" is implied so do not include it. Defaults to {}.

  • append_standard_keywords (AppendableSelector[KeyStringValues]) – Append standard key/value pairs to TEXT. All keys and values will be included as they appear here. The leading "$" is implied so do not include it. Defaults to {}.

  • substitute_standard_key_values (AppendableSelector[SubPatterns]) – Apply sed-like substitution operation on matching standard keys. The leading "$" is implied when matching keys. Defaults to {}.

  • allow_repair_non_unique (TriFlag) – Choose how to handle key collisions when repairing keywords. Non-unique keywords will not be kept in the final FCS file since each list of standard and non-standard keywords must be unique. If "false", raise ConfigError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • text_data_correction (OffsetCorrection) – Corrections for DATA offsets in TEXT. Defaults to (0, 0).

  • text_analysis_correction (OffsetCorrection) – Corrections for ANALYSIS offsets in TEXT. Defaults to (0, 0).

  • ignore_text_data_offsets (bool) – If True ignore DATA offsets in TEXT Defaults to False.

  • ignore_text_analysis_offsets (bool) – If True ignore ANALYSIS offsets in TEXT Defaults to False.

  • allow_header_text_offset_mismatch (AllowHeaderTextOffsetMismatch) – Choose what to do if HEADER and TEXT offsets are different. Exception will be FileLayoutError if emitted. Defaults to "error".

  • allow_missing_required_offsets (TriFlag) – Choose what happens when required DATA and ANALYSIS (3.1 or lower) offsets in TEXT are be missing. If missing, fall back to offsets from HEADER. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • process_optional_failure (ProcessKeywordFailure) – Process optional keys which cause an error. Defaults to "error".

  • int_width_override (IntWidthOverride) – Override $PnB. Only affects integer layouts in FCS 2.0/3.0. Defaults to "never".

  • byteord_override (ByteordOverride) – Override $BYTEORD. Only affects integer layouts in FCS 2.0/3.0. Defaults to "none".

  • disallow_range_truncation (TriFlag) – Choose how to handle $PnR values that need to be truncated to match the number of bytes specified by $PnB and $DATATYPE. If "false", throw warning. If "true", raise RelationalError. If "silent", do nothing. Defaults to "false".

  • warnings_are_errors (bool) – If True all warnings will be regarded as errors. Defaults to False.

  • hide_warnings (bool) – If True hide all warnings. Defaults to False.

Return type:

list[tuple[CoreTEXT2_0 | CoreTEXT3_0 | CoreTEXT3_1 | CoreTEXT3_2, StdTEXTOutput]]

Raises:
pyreflow.api.fcs_read_flat_datasets(path, skip=None, limit=None, scan=False, text_correction=(0, 0), data_correction=(0, 0), analysis_correction=(0, 0), other_corrections=[], max_other=None, other_width=8, guess_other_width='none', squish_offsets=False, allow_pseudoempty=False, dataset_overflow_limit=0, overlap_correction_limit=0, version_override=None, supp_text_correction=(0, 0), nextdata_correction=0, allow_duplicated_supp_text='false', ignore_supp_text=False, delim_escape_mode='escaped', allow_non_ascii_delim='false', allow_nonunique='false', allow_even_delims='false', allow_odd_tokens='false', allow_empty_keys='false', allow_delim_at_boundary='false', use_encoding='utf8', allow_non_ascii_keys='false', allow_non_utf8_values='false', allow_missing_supp_text='false', allow_supp_text_own_delim='false', allow_missing_nextdata='false', trim_value_whitespace='notrim', ignore_standard_keys=[], promote_to_standard=[], demote_from_standard=[], rename_standard_keys={}, replace_standard_key_values={}, append_standard_keywords={}, substitute_standard_key_values={}, allow_repair_non_unique='false', text_data_correction=(0, 0), text_analysis_correction=(0, 0), ignore_text_data_offsets=False, ignore_text_analysis_offsets=False, allow_header_text_offset_mismatch='error', allow_missing_required_offsets='false', process_optional_failure='error', int_width_override='never', byteord_override='none', disallow_range_truncation='false', data_remainder_limit=0, allow_uneven_event_width='false', allow_tot_mismatch='false', over_bitmask_action='trunc_warn', over_range_action='warn', allow_missing_crc='false', allow_mismatch_crc='false', compute_crc='never', read_intra_segment_dark_bytes=False, read_post_dataset_dark_bytes=False, row_buffer_size=28000, warnings_are_errors=False, hide_warnings=False)#

Read multiple datasets from FCS file in flat mode.

Parameters:

  • path (Path) – Path to be read.

  • skip (int | None) – Number of datasets to skip. The HEADER and TEXT from skipped datasets will still be read to obtain $NEXTDATA for the next dataset in the file. Defaults to None.

  • limit (int | None) – Parse up to this many datasets Defaults to None.

  • scan (bool) – If True, scan through file to find next dataset based on version tags rather than relying on $NEXTDATA. Defaults to False.

  • text_correction (OffsetCorrection) – Corrections for Primary TEXT offsets in HEADER. Defaults to (0, 0).

  • data_correction (OffsetCorrection) – Corrections for DATA offsets in HEADER. Defaults to (0, 0).

  • analysis_correction (OffsetCorrection) – Corrections for ANALYSIS offsets in HEADER. Defaults to (0, 0).

  • other_corrections (list[OffsetCorrection]) – Corrections for OTHER offsets if they exist. Each correction will be applied in order. If an offset does not need to be corrected, use (0,0). This will not affect the number of OTHER segments that are read; this is controlled by max_other. Defaults to [].

  • max_other (int | None) – Maximum number of OTHER segments that can be parsed. None means limitless. Defaults to None.

  • other_width (int) – Width (in bytes) to use when parsing OTHER offsets. Defaults to 8.

  • guess_other_width (GuessOtherWidth) – Guess the width of OTHER segments. Non-fatal failure to guess width will fall back to 8 or whatever was given in other_width Defaults to "none".

  • squish_offsets (bool) – If True and a segment’s ending offset is zero, treat entire offset as empty. This might happen if the ending offset is longer than 8 digits, in which case it must be written in TEXT. If this happens, the standards mandate that both offsets be written to TEXT and that the HEADER offsets be set to 0,0, so only writing one is an error unless this flag is set. This should only happen in FCS 3.0 files and above. Defaults to False.

  • allow_pseudoempty (bool) – If True, allow offsets like X,X-1. Some files will denote an “empty” offset as 0,-1 or 1000,999, which is logically correct since the last offset points to the last byte, thus 0,0 is actually 1 byte long. If this flat is set, such offsets will be treated as if they were 0,0. Defaults to False.

  • dataset_overflow_limit (int) – Limit by which offsets can be truncated if they exceed end of file or $NEXTDATA. Defaults to 0.

  • overlap_correction_limit (int) – Limit by which ending segment offset can be truncated if they overlap another offset. Defaults to 0.

  • version_override (VersionOverride | None) – Override the FCS version as seen in HEADER. Defaults to None.

  • supp_text_correction (OffsetCorrection) – Corrections for Supplemental TEXT offsets in TEXT. Defaults to (0, 0).

  • nextdata_correction (int) – Correction for $NEXTDATA. Defaults to 0.

  • allow_duplicated_supp_text (TriFlag) – Choose what happens if supplemental TEXT offsets overlap the primary TEXT offsets from HEADER or HEADER. The offsets will not be used if an overlap is found. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • ignore_supp_text (bool) – If True, ignore supplemental TEXT entirely. Defaults to False.

  • delim_escape_mode (DelimEscapeMode) – Determine how to escape delims in TEXT. Defaults to "escaped".

  • allow_non_ascii_delim (TriFlag) – Choose how to handle non-ASCII delimiters (outside 1-126). If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_nonunique (TriFlag) – Choose how to handle non-unique keys in TEXT. In such cases, only the first will be used regardless of this setting. If "false", raise ParseKeyError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_even_delims (TriFlag) – Choose what happens if TEXT has an even number of delimiters. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_odd_tokens (TriFlag) – Choose what happens if TEXT contains an odd number of tokens. The last ‘dangling’ token will be dropped regardless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_empty_keys (TriFlag) – Choose what happens if any keys are blank. Only relevant if if delimiters are unescaped. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_delim_at_boundary (TriFlag) – Choose what happens if there are delimiters at token boundaries. The FCS standard forbids this because it is impossible to tell if such delimiters belong to the previous or the next token. Consequently, delimiters at boundaries will be dropped regardless of this flag. Only relevant if delimiters are escaped. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • use_encoding (UseEncoding) – Choose how to interpret characters in TEXT. Defaults to "utf8".

  • allow_non_ascii_keys (TriFlag) – Choose how to handle non-ASCII keys. This only applies to non-standard keywords, as all standardized keywords may only contain letters, numbers, and start with "$". Regardless, all compliant keys must only have ASCII. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_non_utf8_values (TriFlag) – Choose what happens if non-UTF8 characters are in TEXT. Tokens with such characters will be dropped regardless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_missing_supp_text (TriFlag) – Choose how to handle supplemental missing TEXT offsets in primary TEXT. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_supp_text_own_delim (TriFlag) – Choose what happens if supplemental TEXT has a different delimiter compared to primary TEXT. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_missing_nextdata (TriFlag) – Choose how to handle missing $NEXTDATA. This is a required keyword in all versions. However, most files only have one dataset in which case this keyword is meaningless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • trim_value_whitespace (TrimValueWhitespace) – Trim whitespace from beginning and end of all values. This may create blank values if the starting string is entirely whitespace. Defaults to "notrim".

  • ignore_standard_keys (AppendableSelector[KeyPatterns]) – Remove standard keys from TEXT. The leading "$" is implied so do not include it. Defaults to [].

  • promote_to_standard (AppendableSelector[KeyPatterns]) – Promote nonstandard keys to standard keys in TEXT. Defaults to [].

  • demote_from_standard (AppendableSelector[KeyPatterns]) – Demote nonstandard keys from standard keys in TEXT. Defaults to [].

  • rename_standard_keys (AppendableSelector[KeyStringPairs]) – Rename standard keys in TEXT. Keys matching the first part of the pair will be replaced by the second. Comparisons are case insensitive. The leading "$" is implied so do not include it. Defaults to {}.

  • replace_standard_key_values (AppendableSelector[KeyStringValues]) – Replace values for standard keys in TEXT. Comparisons are case insensitive. The leading "$" is implied so do not include it. Defaults to {}.

  • append_standard_keywords (AppendableSelector[KeyStringValues]) – Append standard key/value pairs to TEXT. All keys and values will be included as they appear here. The leading "$" is implied so do not include it. Defaults to {}.

  • substitute_standard_key_values (AppendableSelector[SubPatterns]) – Apply sed-like substitution operation on matching standard keys. The leading "$" is implied when matching keys. Defaults to {}.

  • allow_repair_non_unique (TriFlag) – Choose how to handle key collisions when repairing keywords. Non-unique keywords will not be kept in the final FCS file since each list of standard and non-standard keywords must be unique. If "false", raise ConfigError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • text_data_correction (OffsetCorrection) – Corrections for DATA offsets in TEXT. Defaults to (0, 0).

  • text_analysis_correction (OffsetCorrection) – Corrections for ANALYSIS offsets in TEXT. Defaults to (0, 0).

  • ignore_text_data_offsets (bool) – If True ignore DATA offsets in TEXT Defaults to False.

  • ignore_text_analysis_offsets (bool) – If True ignore ANALYSIS offsets in TEXT Defaults to False.

  • allow_header_text_offset_mismatch (AllowHeaderTextOffsetMismatch) – Choose what to do if HEADER and TEXT offsets are different. Exception will be FileLayoutError if emitted. Defaults to "error".

  • allow_missing_required_offsets (TriFlag) – Choose what happens when required DATA and ANALYSIS (3.1 or lower) offsets in TEXT are be missing. If missing, fall back to offsets from HEADER. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • process_optional_failure (ProcessKeywordFailure) – Process optional keys which cause an error. Defaults to "error".

  • int_width_override (IntWidthOverride) – Override $PnB. Only affects integer layouts in FCS 2.0/3.0. Defaults to "never".

  • byteord_override (ByteordOverride) – Override $BYTEORD. Only affects integer layouts in FCS 2.0/3.0. Defaults to "none".

  • disallow_range_truncation (TriFlag) – Choose how to handle $PnR values that need to be truncated to match the number of bytes specified by $PnB and $DATATYPE. If "false", throw warning. If "true", raise RelationalError. If "silent", do nothing. Defaults to "false".

  • data_remainder_limit (int) – Limit by which ending DATA offset can be truncated if its length modulo event width produces a remainder. Defaults to 0.

  • allow_uneven_event_width (TriFlag) – Choose what to do when event width does not perfectly divide length of DATA. Does not apply to delimited ASCII data schema. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_tot_mismatch (TriFlag) – Choose what happens when $TOT does not match number of events as computed by the event width and length of DATA. Does not apply to delimited ASCII data schema. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • over_bitmask_action (OverLimitAction) – Choose what to do with event integer values in DATA which exceed bitmask. Defaults to "trunc_warn".

  • over_range_action (OverLimitAction) – Choose what to do with event values in DATA which exceed $PnR. Defaults to "warn".

  • allow_missing_crc (TriFlag) – Choose what to do when CRC is missing from the end of a dataset. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_mismatch_crc (TriFlag) – Choose what to do when computed CRC and CRC at the end of a dataset do not match. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • compute_crc (ComputeCRC) – Choose when to compute the CRC for a dataset. Defaults to "never".

  • read_intra_segment_dark_bytes (bool) – If True read bytes which are between segments. Defaults to False.

  • read_post_dataset_dark_bytes (bool) – If True read bytes between the end of the current dataset and the next. Defaults to False.

  • row_buffer_size (int) – Set the size in bytes for the internal buffer used to read DATA. This is a performance parameter that balances read syscalls (too low) and cache misses (too high). It should generally be 90% of the CPU’s L1D cache size. Defaults to 28000.

  • warnings_are_errors (bool) – If True all warnings will be regarded as errors. Defaults to False.

  • hide_warnings (bool) – If True hide all warnings. Defaults to False.

Return type:

list[FlatDatasetOutput]

Raises:
pyreflow.api.fcs_read_std_datasets(path, skip=None, limit=None, scan=False, text_correction=(0, 0), data_correction=(0, 0), analysis_correction=(0, 0), other_corrections=[], max_other=None, other_width=8, guess_other_width='none', squish_offsets=False, allow_pseudoempty=False, dataset_overflow_limit=0, overlap_correction_limit=0, version_override=None, supp_text_correction=(0, 0), nextdata_correction=0, allow_duplicated_supp_text='false', ignore_supp_text=False, delim_escape_mode='escaped', allow_non_ascii_delim='false', allow_nonunique='false', allow_even_delims='false', allow_odd_tokens='false', allow_empty_keys='false', allow_delim_at_boundary='false', use_encoding='utf8', allow_non_ascii_keys='false', allow_non_utf8_values='false', allow_missing_supp_text='false', allow_supp_text_own_delim='false', allow_missing_nextdata='false', trim_value_whitespace='notrim', dedup_measurement_names=False, trim_intra_value_whitespace=False, time_meas_pattern='^(TIME|Time)$', allow_missing_time='false', force_linear_scale='none', ignore_optical_only_keys=[], process_optical_only_keys='demote_warn', date_pattern=None, time_pattern=None, datetime_pattern=None, last_modified_pattern=None, allow_other_feature=False, process_pseudostandard='error', process_hyper_par='error', process_other_version='error', process_extra_timestep='error', fix_log_scale_offsets=False, add_missing_timestep=None, spillover_measurement_mode='named', disallow_localtime=False, ignore_standard_keys=[], promote_to_standard=[], demote_from_standard=[], rename_standard_keys={}, replace_standard_key_values={}, append_standard_keywords={}, substitute_standard_key_values={}, allow_repair_non_unique='false', text_data_correction=(0, 0), text_analysis_correction=(0, 0), ignore_text_data_offsets=False, ignore_text_analysis_offsets=False, allow_header_text_offset_mismatch='error', allow_missing_required_offsets='false', process_optional_failure='error', int_width_override='never', byteord_override='none', disallow_range_truncation='false', data_remainder_limit=0, allow_uneven_event_width='false', allow_tot_mismatch='false', over_bitmask_action='trunc_warn', over_range_action='warn', allow_missing_crc='false', allow_mismatch_crc='false', compute_crc='never', read_intra_segment_dark_bytes=False, read_post_dataset_dark_bytes=False, row_buffer_size=28000, warnings_are_errors=False, hide_warnings=False)#

Read multiple standardized datasets from FCS file.

Parameters:

  • path (Path) – Path to be read.

  • skip (int | None) – Number of datasets to skip. The HEADER and TEXT from skipped datasets will still be read to obtain $NEXTDATA for the next dataset in the file. Defaults to None.

  • limit (int | None) – Parse up to this many datasets Defaults to None.

  • scan (bool) – If True, scan through file to find next dataset based on version tags rather than relying on $NEXTDATA. Defaults to False.

  • text_correction (OffsetCorrection) – Corrections for Primary TEXT offsets in HEADER. Defaults to (0, 0).

  • data_correction (OffsetCorrection) – Corrections for DATA offsets in HEADER. Defaults to (0, 0).

  • analysis_correction (OffsetCorrection) – Corrections for ANALYSIS offsets in HEADER. Defaults to (0, 0).

  • other_corrections (list[OffsetCorrection]) – Corrections for OTHER offsets if they exist. Each correction will be applied in order. If an offset does not need to be corrected, use (0,0). This will not affect the number of OTHER segments that are read; this is controlled by max_other. Defaults to [].

  • max_other (int | None) – Maximum number of OTHER segments that can be parsed. None means limitless. Defaults to None.

  • other_width (int) – Width (in bytes) to use when parsing OTHER offsets. Defaults to 8.

  • guess_other_width (GuessOtherWidth) – Guess the width of OTHER segments. Non-fatal failure to guess width will fall back to 8 or whatever was given in other_width Defaults to "none".

  • squish_offsets (bool) – If True and a segment’s ending offset is zero, treat entire offset as empty. This might happen if the ending offset is longer than 8 digits, in which case it must be written in TEXT. If this happens, the standards mandate that both offsets be written to TEXT and that the HEADER offsets be set to 0,0, so only writing one is an error unless this flag is set. This should only happen in FCS 3.0 files and above. Defaults to False.

  • allow_pseudoempty (bool) – If True, allow offsets like X,X-1. Some files will denote an “empty” offset as 0,-1 or 1000,999, which is logically correct since the last offset points to the last byte, thus 0,0 is actually 1 byte long. If this flat is set, such offsets will be treated as if they were 0,0. Defaults to False.

  • dataset_overflow_limit (int) – Limit by which offsets can be truncated if they exceed end of file or $NEXTDATA. Defaults to 0.

  • overlap_correction_limit (int) – Limit by which ending segment offset can be truncated if they overlap another offset. Defaults to 0.

  • version_override (VersionOverride | None) – Override the FCS version as seen in HEADER. Defaults to None.

  • supp_text_correction (OffsetCorrection) – Corrections for Supplemental TEXT offsets in TEXT. Defaults to (0, 0).

  • nextdata_correction (int) – Correction for $NEXTDATA. Defaults to 0.

  • allow_duplicated_supp_text (TriFlag) – Choose what happens if supplemental TEXT offsets overlap the primary TEXT offsets from HEADER or HEADER. The offsets will not be used if an overlap is found. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • ignore_supp_text (bool) – If True, ignore supplemental TEXT entirely. Defaults to False.

  • delim_escape_mode (DelimEscapeMode) – Determine how to escape delims in TEXT. Defaults to "escaped".

  • allow_non_ascii_delim (TriFlag) – Choose how to handle non-ASCII delimiters (outside 1-126). If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_nonunique (TriFlag) – Choose how to handle non-unique keys in TEXT. In such cases, only the first will be used regardless of this setting. If "false", raise ParseKeyError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_even_delims (TriFlag) – Choose what happens if TEXT has an even number of delimiters. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_odd_tokens (TriFlag) – Choose what happens if TEXT contains an odd number of tokens. The last ‘dangling’ token will be dropped regardless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_empty_keys (TriFlag) – Choose what happens if any keys are blank. Only relevant if if delimiters are unescaped. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_delim_at_boundary (TriFlag) – Choose what happens if there are delimiters at token boundaries. The FCS standard forbids this because it is impossible to tell if such delimiters belong to the previous or the next token. Consequently, delimiters at boundaries will be dropped regardless of this flag. Only relevant if delimiters are escaped. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • use_encoding (UseEncoding) – Choose how to interpret characters in TEXT. Defaults to "utf8".

  • allow_non_ascii_keys (TriFlag) – Choose how to handle non-ASCII keys. This only applies to non-standard keywords, as all standardized keywords may only contain letters, numbers, and start with "$". Regardless, all compliant keys must only have ASCII. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_non_utf8_values (TriFlag) – Choose what happens if non-UTF8 characters are in TEXT. Tokens with such characters will be dropped regardless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_missing_supp_text (TriFlag) – Choose how to handle supplemental missing TEXT offsets in primary TEXT. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_supp_text_own_delim (TriFlag) – Choose what happens if supplemental TEXT has a different delimiter compared to primary TEXT. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_missing_nextdata (TriFlag) – Choose how to handle missing $NEXTDATA. This is a required keyword in all versions. However, most files only have one dataset in which case this keyword is meaningless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • trim_value_whitespace (TrimValueWhitespace) – Trim whitespace from beginning and end of all values. This may create blank values if the starting string is entirely whitespace. Defaults to "notrim".

  • dedup_measurement_names (bool) – If True, force all $PnN to be unique by appending "~X" to each duplicate and incrementing X starting at 0. Defaults to False.

  • trim_intra_value_whitespace (bool) – If True, trim whitespace between delimiters such as "," and ";" within keyword value strings. Defaults to False.

  • time_meas_pattern (Selector[str | None]) – A pattern to match the $PnN of the time measurement. If "NoTime", do not try to find a time measurement. Defaults to "^(TIME|Time)$".

  • allow_missing_time (TriFlag) – Choose what to do when time measurement is be missing. If "false", raise RelationalError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • force_linear_scale (ForceLinearScale) – Force $PnE to be linear for certain measurements. Affected measurements will never fail. Defaults to "none".

  • ignore_optical_only_keys (list[OpticalOnlyKey]) – Ignore optical keys in temporal measurement. These keys are $PnG which is explicitly forbidden by the standard but allowed in this library to be set to 1.0 (noop), or others which are nonsensical for time measurements but are not explicitly forbidden in the the standard (such as $PnL). Provided keys are the string after the "Pn" in the "PnX" keywords. Defaults to [].

  • process_optical_only_keys (ProcessOpticalOnlyKeys) – Choose how to handle optical keys found in temporal measurements. Does nothing unless keys are specified in ignore_optical_only_keys. Defaults to "demote_warn".

  • date_pattern (Selector[str | None]) – If supplied, will be used as an alternative pattern when parsing $DATE. If not supplied, $DATE will be parsed according to the standard pattern which is "%d-%b-%Y". Defaults to None.

  • time_pattern (Selector[str | None]) – If supplied, will be used as an alternative pattern when parsing $BTIM and $ETIM. The values "%!" or "%@" may be used to match 1/60 seconds or centiseconds respectively. If not supplied, $BTIM and $ETIM will be parsed according to the standard pattern which is "%H:%M:%S" for 2.0, "%H:%M:%S:%!" for 3.0 and "%H:%M:%S.%@" for 3.1 and up. Defaults to None.

  • datetime_pattern (Selector[str | None]) – If supplied, will be used as an alternative pattern when parsing $BEGINDATETIME and $ENDDATETIME. The pattern must follow the format outlined in chrono. If not supplied, these will be parsed as ISO timestamps with optional timezone. Defaults to None.

  • last_modified_pattern (Selector[str | None]) – If supplied, will be used as an alternative pattern when parsing $LAST_MODIFIED. The pattern must follow the format outlined in chrono. If not supplied, these will be parsed according to the default pattern which is "%d-%b-%Y %H:%M:%S" possibly with centiseconds after. Defaults to None.

  • allow_other_feature (bool) – If True, allow $PnFEATURE to be a value other than "Area", "Width", or "Height". Defaults to False.

  • process_pseudostandard (ProcessKeywordFailure) – Process non-standard keywords with a leading "$". The presence of such keywords often means the version in HEADER is incorrect. Defaults to "error".

  • process_hyper_par (ProcessKeywordFailure) – Process measurement keywords whose index is greater than $PAR. Defaults to "error".

  • process_other_version (ProcessKeywordFailure) – Process standard keywords from different FCS versions. Defaults to "error".

  • process_extra_timestep (ProcessKeywordFailure) – Process $TIMESTEP to be present which may indicate a time measurement is present but not identified. Defaults to "error".

  • fix_log_scale_offsets (bool) – If True fix log-scale $PnE and keywords which have zero offset (ie <X>,0.0 where X is non-zero). Defaults to False.

  • add_missing_timestep (Timestep | None) – Set $TIMESTEP if it is not present and required. This will do nothing on FCS2.0 files since this version does not specify $TIMESTEP. Defaults to None.

  • spillover_measurement_mode (SpilloverMeasurementMode) – Choose how to interpret measurement strings in $SPILLOVER. Defaults to "named".

  • disallow_localtime (bool) – If True, require that $BEGINDATETIME and $ENDDATETIME have a timezone if provided. This is not required by the standard, but not having a timezone is ambiguous since the absolute value of the timestamp is dependent on localtime and therefore is location-dependent. Only affects FCS 3.2. Defaults to False.

  • ignore_standard_keys (AppendableSelector[KeyPatterns]) – Remove standard keys from TEXT. The leading "$" is implied so do not include it. Defaults to [].

  • promote_to_standard (AppendableSelector[KeyPatterns]) – Promote nonstandard keys to standard keys in TEXT. Defaults to [].

  • demote_from_standard (AppendableSelector[KeyPatterns]) – Demote nonstandard keys from standard keys in TEXT. Defaults to [].

  • rename_standard_keys (AppendableSelector[KeyStringPairs]) – Rename standard keys in TEXT. Keys matching the first part of the pair will be replaced by the second. Comparisons are case insensitive. The leading "$" is implied so do not include it. Defaults to {}.

  • replace_standard_key_values (AppendableSelector[KeyStringValues]) – Replace values for standard keys in TEXT. Comparisons are case insensitive. The leading "$" is implied so do not include it. Defaults to {}.

  • append_standard_keywords (AppendableSelector[KeyStringValues]) – Append standard key/value pairs to TEXT. All keys and values will be included as they appear here. The leading "$" is implied so do not include it. Defaults to {}.

  • substitute_standard_key_values (AppendableSelector[SubPatterns]) – Apply sed-like substitution operation on matching standard keys. The leading "$" is implied when matching keys. Defaults to {}.

  • allow_repair_non_unique (TriFlag) – Choose how to handle key collisions when repairing keywords. Non-unique keywords will not be kept in the final FCS file since each list of standard and non-standard keywords must be unique. If "false", raise ConfigError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • text_data_correction (OffsetCorrection) – Corrections for DATA offsets in TEXT. Defaults to (0, 0).

  • text_analysis_correction (OffsetCorrection) – Corrections for ANALYSIS offsets in TEXT. Defaults to (0, 0).

  • ignore_text_data_offsets (bool) – If True ignore DATA offsets in TEXT Defaults to False.

  • ignore_text_analysis_offsets (bool) – If True ignore ANALYSIS offsets in TEXT Defaults to False.

  • allow_header_text_offset_mismatch (AllowHeaderTextOffsetMismatch) – Choose what to do if HEADER and TEXT offsets are different. Exception will be FileLayoutError if emitted. Defaults to "error".

  • allow_missing_required_offsets (TriFlag) – Choose what happens when required DATA and ANALYSIS (3.1 or lower) offsets in TEXT are be missing. If missing, fall back to offsets from HEADER. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • process_optional_failure (ProcessKeywordFailure) – Process optional keys which cause an error. Defaults to "error".

  • int_width_override (IntWidthOverride) – Override $PnB. Only affects integer layouts in FCS 2.0/3.0. Defaults to "never".

  • byteord_override (ByteordOverride) – Override $BYTEORD. Only affects integer layouts in FCS 2.0/3.0. Defaults to "none".

  • disallow_range_truncation (TriFlag) – Choose how to handle $PnR values that need to be truncated to match the number of bytes specified by $PnB and $DATATYPE. If "false", throw warning. If "true", raise RelationalError. If "silent", do nothing. Defaults to "false".

  • data_remainder_limit (int) – Limit by which ending DATA offset can be truncated if its length modulo event width produces a remainder. Defaults to 0.

  • allow_uneven_event_width (TriFlag) – Choose what to do when event width does not perfectly divide length of DATA. Does not apply to delimited ASCII data schema. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_tot_mismatch (TriFlag) – Choose what happens when $TOT does not match number of events as computed by the event width and length of DATA. Does not apply to delimited ASCII data schema. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • over_bitmask_action (OverLimitAction) – Choose what to do with event integer values in DATA which exceed bitmask. Defaults to "trunc_warn".

  • over_range_action (OverLimitAction) – Choose what to do with event values in DATA which exceed $PnR. Defaults to "warn".

  • allow_missing_crc (TriFlag) – Choose what to do when CRC is missing from the end of a dataset. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_mismatch_crc (TriFlag) – Choose what to do when computed CRC and CRC at the end of a dataset do not match. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • compute_crc (ComputeCRC) – Choose when to compute the CRC for a dataset. Defaults to "never".

  • read_intra_segment_dark_bytes (bool) – If True read bytes which are between segments. Defaults to False.

  • read_post_dataset_dark_bytes (bool) – If True read bytes between the end of the current dataset and the next. Defaults to False.

  • row_buffer_size (int) – Set the size in bytes for the internal buffer used to read DATA. This is a performance parameter that balances read syscalls (too low) and cache misses (too high). It should generally be 90% of the CPU’s L1D cache size. Defaults to 28000.

  • warnings_are_errors (bool) – If True all warnings will be regarded as errors. Defaults to False.

  • hide_warnings (bool) – If True hide all warnings. Defaults to False.

Return type:

list[tuple[CoreDataset2_0 | CoreDataset3_0 | CoreDataset3_1 | CoreDataset3_2, StdDatasetOutput]]

Raises:

  • ConfigError – if other_width is less than 8 and greater than 20

  • FileLayoutError – If HEADER, TEXT, or DATA are unparsable

  • ParseKeywordValueError – If any keyword values could not be read from their string encoding

  • RelationalError – If keywords are incompatible with indicated layout of DATA or if keywords that are referenced by other keywords do not exist

  • EventDataError – If values in DATA cannot be read

  • ExtraKeywordError – If any standard keys are unused and not dropped by some other option

pyreflow.api.fcs_read_header(path, text_correction=(0, 0), data_correction=(0, 0), analysis_correction=(0, 0), other_corrections=[], max_other=None, other_width=8, guess_other_width='none', squish_offsets=False, allow_pseudoempty=False, dataset_overflow_limit=0, overlap_correction_limit=0, dataset_offset=0)#

Read the HEADER of an FCS file.

Parameters:

  • path (Path) – Path to be read.

  • text_correction (OffsetCorrection) – Corrections for Primary TEXT offsets in HEADER. Defaults to (0, 0).

  • data_correction (OffsetCorrection) – Corrections for DATA offsets in HEADER. Defaults to (0, 0).

  • analysis_correction (OffsetCorrection) – Corrections for ANALYSIS offsets in HEADER. Defaults to (0, 0).

  • other_corrections (list[OffsetCorrection]) – Corrections for OTHER offsets if they exist. Each correction will be applied in order. If an offset does not need to be corrected, use (0,0). This will not affect the number of OTHER segments that are read; this is controlled by max_other. Defaults to [].

  • max_other (int | None) – Maximum number of OTHER segments that can be parsed. None means limitless. Defaults to None.

  • other_width (int) – Width (in bytes) to use when parsing OTHER offsets. Defaults to 8.

  • guess_other_width (GuessOtherWidth) – Guess the width of OTHER segments. Non-fatal failure to guess width will fall back to 8 or whatever was given in other_width Defaults to "none".

  • squish_offsets (bool) – If True and a segment’s ending offset is zero, treat entire offset as empty. This might happen if the ending offset is longer than 8 digits, in which case it must be written in TEXT. If this happens, the standards mandate that both offsets be written to TEXT and that the HEADER offsets be set to 0,0, so only writing one is an error unless this flag is set. This should only happen in FCS 3.0 files and above. Defaults to False.

  • allow_pseudoempty (bool) – If True, allow offsets like X,X-1. Some files will denote an “empty” offset as 0,-1 or 1000,999, which is logically correct since the last offset points to the last byte, thus 0,0 is actually 1 byte long. If this flat is set, such offsets will be treated as if they were 0,0. Defaults to False.

  • dataset_overflow_limit (int) – Limit by which offsets can be truncated if they exceed end of file or $NEXTDATA. Defaults to 0.

  • overlap_correction_limit (int) – Limit by which ending segment offset can be truncated if they overlap another offset. Defaults to 0.

  • dataset_offset (int) – Starting position in the file of the dataset to be read. Defaults to 0.

Return type:

Header

Raises:
pyreflow.api.fcs_read_flat_dataset_with_keywords(path, header, kws, allow_pseudoempty=False, dataset_overflow_limit=0, overlap_correction_limit=0, ignore_standard_keys=[], promote_to_standard=[], demote_from_standard=[], rename_standard_keys={}, replace_standard_key_values={}, append_standard_keywords={}, substitute_standard_key_values={}, allow_repair_non_unique='false', text_data_correction=(0, 0), text_analysis_correction=(0, 0), ignore_text_data_offsets=False, ignore_text_analysis_offsets=False, allow_header_text_offset_mismatch='error', allow_missing_required_offsets='false', process_optional_failure='error', int_width_override='never', byteord_override='none', disallow_range_truncation='false', data_remainder_limit=0, allow_uneven_event_width='false', allow_tot_mismatch='false', over_bitmask_action='trunc_warn', over_range_action='warn', allow_missing_crc='false', allow_mismatch_crc='false', compute_crc='never', read_intra_segment_dark_bytes=False, read_post_dataset_dark_bytes=False, row_buffer_size=28000, warnings_are_errors=False, hide_warnings=False, dataset_offset=0, dataset_len=None)#

Read dataset from FCS file from keywords in flat mode.

Parameters:

  • path (Path) – Path to be read.

  • header (HeaderAndSuppOffsets) – The HEADER and supplemental TEXT offsets from parsed file

  • kws (ValidKeywords) – Standard and non-standard keywords.

  • allow_pseudoempty (bool) – If True, allow offsets like X,X-1. Some files will denote an “empty” offset as 0,-1 or 1000,999, which is logically correct since the last offset points to the last byte, thus 0,0 is actually 1 byte long. If this flat is set, such offsets will be treated as if they were 0,0. Defaults to False.

  • dataset_overflow_limit (int) – Limit by which offsets can be truncated if they exceed end of file or $NEXTDATA. Defaults to 0.

  • overlap_correction_limit (int) – Limit by which ending segment offset can be truncated if they overlap another offset. Defaults to 0.

  • ignore_standard_keys (AppendableSelector[KeyPatterns]) – Remove standard keys from TEXT. The leading "$" is implied so do not include it. Defaults to [].

  • promote_to_standard (AppendableSelector[KeyPatterns]) – Promote nonstandard keys to standard keys in TEXT. Defaults to [].

  • demote_from_standard (AppendableSelector[KeyPatterns]) – Demote nonstandard keys from standard keys in TEXT. Defaults to [].

  • rename_standard_keys (AppendableSelector[KeyStringPairs]) – Rename standard keys in TEXT. Keys matching the first part of the pair will be replaced by the second. Comparisons are case insensitive. The leading "$" is implied so do not include it. Defaults to {}.

  • replace_standard_key_values (AppendableSelector[KeyStringValues]) – Replace values for standard keys in TEXT. Comparisons are case insensitive. The leading "$" is implied so do not include it. Defaults to {}.

  • append_standard_keywords (AppendableSelector[KeyStringValues]) – Append standard key/value pairs to TEXT. All keys and values will be included as they appear here. The leading "$" is implied so do not include it. Defaults to {}.

  • substitute_standard_key_values (AppendableSelector[SubPatterns]) – Apply sed-like substitution operation on matching standard keys. The leading "$" is implied when matching keys. Defaults to {}.

  • allow_repair_non_unique (TriFlag) – Choose how to handle key collisions when repairing keywords. Non-unique keywords will not be kept in the final FCS file since each list of standard and non-standard keywords must be unique. If "false", raise ConfigError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • text_data_correction (OffsetCorrection) – Corrections for DATA offsets in TEXT. Defaults to (0, 0).

  • text_analysis_correction (OffsetCorrection) – Corrections for ANALYSIS offsets in TEXT. Defaults to (0, 0).

  • ignore_text_data_offsets (bool) – If True ignore DATA offsets in TEXT Defaults to False.

  • ignore_text_analysis_offsets (bool) – If True ignore ANALYSIS offsets in TEXT Defaults to False.

  • allow_header_text_offset_mismatch (AllowHeaderTextOffsetMismatch) – Choose what to do if HEADER and TEXT offsets are different. Exception will be FileLayoutError if emitted. Defaults to "error".

  • allow_missing_required_offsets (TriFlag) – Choose what happens when required DATA and ANALYSIS (3.1 or lower) offsets in TEXT are be missing. If missing, fall back to offsets from HEADER. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • process_optional_failure (ProcessKeywordFailure) – Process optional keys which cause an error. Defaults to "error".

  • int_width_override (IntWidthOverride) – Override $PnB. Only affects integer layouts in FCS 2.0/3.0. Defaults to "never".

  • byteord_override (ByteordOverride) – Override $BYTEORD. Only affects integer layouts in FCS 2.0/3.0. Defaults to "none".

  • disallow_range_truncation (TriFlag) – Choose how to handle $PnR values that need to be truncated to match the number of bytes specified by $PnB and $DATATYPE. If "false", throw warning. If "true", raise RelationalError. If "silent", do nothing. Defaults to "false".

  • data_remainder_limit (int) – Limit by which ending DATA offset can be truncated if its length modulo event width produces a remainder. Defaults to 0.

  • allow_uneven_event_width (TriFlag) – Choose what to do when event width does not perfectly divide length of DATA. Does not apply to delimited ASCII data schema. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_tot_mismatch (TriFlag) – Choose what happens when $TOT does not match number of events as computed by the event width and length of DATA. Does not apply to delimited ASCII data schema. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • over_bitmask_action (OverLimitAction) – Choose what to do with event integer values in DATA which exceed bitmask. Defaults to "trunc_warn".

  • over_range_action (OverLimitAction) – Choose what to do with event values in DATA which exceed $PnR. Defaults to "warn".

  • allow_missing_crc (TriFlag) – Choose what to do when CRC is missing from the end of a dataset. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_mismatch_crc (TriFlag) – Choose what to do when computed CRC and CRC at the end of a dataset do not match. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • compute_crc (ComputeCRC) – Choose when to compute the CRC for a dataset. Defaults to "never".

  • read_intra_segment_dark_bytes (bool) – If True read bytes which are between segments. Defaults to False.

  • read_post_dataset_dark_bytes (bool) – If True read bytes between the end of the current dataset and the next. Defaults to False.

  • row_buffer_size (int) – Set the size in bytes for the internal buffer used to read DATA. This is a performance parameter that balances read syscalls (too low) and cache misses (too high). It should generally be 90% of the CPU’s L1D cache size. Defaults to 28000.

  • warnings_are_errors (bool) – If True all warnings will be regarded as errors. Defaults to False.

  • hide_warnings (bool) – If True hide all warnings. Defaults to False.

  • dataset_offset (int) – Starting position in the file of the dataset to be read. Defaults to 0.

  • dataset_len (int | None) – The length of the dataset to be read; should correspond to $NEXTDATA or end of file (whichever is lesser) minus the starting offset of the dataset. Defaults to None.

Return type:

NewFlatDatasetFromKwsOutput

Raises:
pyreflow.api.fcs_summarize(path, skip=None, limit=None, scan=False, text_correction=(0, 0), data_correction=(0, 0), analysis_correction=(0, 0), other_corrections=[], max_other=None, other_width=8, guess_other_width='none', squish_offsets=False, allow_pseudoempty=False, dataset_overflow_limit=0, overlap_correction_limit=0, version_override=None, supp_text_correction=(0, 0), nextdata_correction=0, allow_duplicated_supp_text='false', ignore_supp_text=False, delim_escape_mode='escaped', allow_non_ascii_delim='false', allow_nonunique='false', allow_even_delims='false', allow_odd_tokens='false', allow_empty_keys='false', allow_delim_at_boundary='false', use_encoding='utf8', allow_non_ascii_keys='false', allow_non_utf8_values='false', allow_missing_supp_text='false', allow_supp_text_own_delim='false', allow_missing_nextdata='false', trim_value_whitespace='notrim', ignore_standard_keys=[], promote_to_standard=[], demote_from_standard=[], rename_standard_keys={}, replace_standard_key_values={}, append_standard_keywords={}, substitute_standard_key_values={}, allow_repair_non_unique='false', text_data_correction=(0, 0), text_analysis_correction=(0, 0), ignore_text_data_offsets=False, ignore_text_analysis_offsets=False, allow_header_text_offset_mismatch='error', allow_missing_required_offsets='false', process_optional_failure='error', int_width_override='never', byteord_override='none', disallow_range_truncation='false', data_remainder_limit=0, allow_uneven_event_width='false', allow_tot_mismatch='false', over_bitmask_action='trunc_warn', over_range_action='warn', allow_missing_crc='false', allow_mismatch_crc='false', compute_crc='never', read_intra_segment_dark_bytes=False, read_post_dataset_dark_bytes=False, row_buffer_size=28000, warnings_are_errors=False, hide_warnings=False)#

Summarize datasets in FCS file.

Parameters:

  • path (Path) – Path to be read.

  • skip (int | None) – Number of datasets to skip. The HEADER and TEXT from skipped datasets will still be read to obtain $NEXTDATA for the next dataset in the file. Defaults to None.

  • limit (int | None) – Parse up to this many datasets Defaults to None.

  • scan (bool) – If True, scan through file to find next dataset based on version tags rather than relying on $NEXTDATA. Defaults to False.

  • text_correction (OffsetCorrection) – Corrections for Primary TEXT offsets in HEADER. Defaults to (0, 0).

  • data_correction (OffsetCorrection) – Corrections for DATA offsets in HEADER. Defaults to (0, 0).

  • analysis_correction (OffsetCorrection) – Corrections for ANALYSIS offsets in HEADER. Defaults to (0, 0).

  • other_corrections (list[OffsetCorrection]) – Corrections for OTHER offsets if they exist. Each correction will be applied in order. If an offset does not need to be corrected, use (0,0). This will not affect the number of OTHER segments that are read; this is controlled by max_other. Defaults to [].

  • max_other (int | None) – Maximum number of OTHER segments that can be parsed. None means limitless. Defaults to None.

  • other_width (int) – Width (in bytes) to use when parsing OTHER offsets. Defaults to 8.

  • guess_other_width (GuessOtherWidth) – Guess the width of OTHER segments. Non-fatal failure to guess width will fall back to 8 or whatever was given in other_width Defaults to "none".

  • squish_offsets (bool) – If True and a segment’s ending offset is zero, treat entire offset as empty. This might happen if the ending offset is longer than 8 digits, in which case it must be written in TEXT. If this happens, the standards mandate that both offsets be written to TEXT and that the HEADER offsets be set to 0,0, so only writing one is an error unless this flag is set. This should only happen in FCS 3.0 files and above. Defaults to False.

  • allow_pseudoempty (bool) – If True, allow offsets like X,X-1. Some files will denote an “empty” offset as 0,-1 or 1000,999, which is logically correct since the last offset points to the last byte, thus 0,0 is actually 1 byte long. If this flat is set, such offsets will be treated as if they were 0,0. Defaults to False.

  • dataset_overflow_limit (int) – Limit by which offsets can be truncated if they exceed end of file or $NEXTDATA. Defaults to 0.

  • overlap_correction_limit (int) – Limit by which ending segment offset can be truncated if they overlap another offset. Defaults to 0.

  • version_override (VersionOverride | None) – Override the FCS version as seen in HEADER. Defaults to None.

  • supp_text_correction (OffsetCorrection) – Corrections for Supplemental TEXT offsets in TEXT. Defaults to (0, 0).

  • nextdata_correction (int) – Correction for $NEXTDATA. Defaults to 0.

  • allow_duplicated_supp_text (TriFlag) – Choose what happens if supplemental TEXT offsets overlap the primary TEXT offsets from HEADER or HEADER. The offsets will not be used if an overlap is found. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • ignore_supp_text (bool) – If True, ignore supplemental TEXT entirely. Defaults to False.

  • delim_escape_mode (DelimEscapeMode) – Determine how to escape delims in TEXT. Defaults to "escaped".

  • allow_non_ascii_delim (TriFlag) – Choose how to handle non-ASCII delimiters (outside 1-126). If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_nonunique (TriFlag) – Choose how to handle non-unique keys in TEXT. In such cases, only the first will be used regardless of this setting. If "false", raise ParseKeyError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_even_delims (TriFlag) – Choose what happens if TEXT has an even number of delimiters. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_odd_tokens (TriFlag) – Choose what happens if TEXT contains an odd number of tokens. The last ‘dangling’ token will be dropped regardless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_empty_keys (TriFlag) – Choose what happens if any keys are blank. Only relevant if if delimiters are unescaped. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_delim_at_boundary (TriFlag) – Choose what happens if there are delimiters at token boundaries. The FCS standard forbids this because it is impossible to tell if such delimiters belong to the previous or the next token. Consequently, delimiters at boundaries will be dropped regardless of this flag. Only relevant if delimiters are escaped. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • use_encoding (UseEncoding) – Choose how to interpret characters in TEXT. Defaults to "utf8".

  • allow_non_ascii_keys (TriFlag) – Choose how to handle non-ASCII keys. This only applies to non-standard keywords, as all standardized keywords may only contain letters, numbers, and start with "$". Regardless, all compliant keys must only have ASCII. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_non_utf8_values (TriFlag) – Choose what happens if non-UTF8 characters are in TEXT. Tokens with such characters will be dropped regardless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_missing_supp_text (TriFlag) – Choose how to handle supplemental missing TEXT offsets in primary TEXT. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_supp_text_own_delim (TriFlag) – Choose what happens if supplemental TEXT has a different delimiter compared to primary TEXT. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_missing_nextdata (TriFlag) – Choose how to handle missing $NEXTDATA. This is a required keyword in all versions. However, most files only have one dataset in which case this keyword is meaningless. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • trim_value_whitespace (TrimValueWhitespace) – Trim whitespace from beginning and end of all values. This may create blank values if the starting string is entirely whitespace. Defaults to "notrim".

  • ignore_standard_keys (AppendableSelector[KeyPatterns]) – Remove standard keys from TEXT. The leading "$" is implied so do not include it. Defaults to [].

  • promote_to_standard (AppendableSelector[KeyPatterns]) – Promote nonstandard keys to standard keys in TEXT. Defaults to [].

  • demote_from_standard (AppendableSelector[KeyPatterns]) – Demote nonstandard keys from standard keys in TEXT. Defaults to [].

  • rename_standard_keys (AppendableSelector[KeyStringPairs]) – Rename standard keys in TEXT. Keys matching the first part of the pair will be replaced by the second. Comparisons are case insensitive. The leading "$" is implied so do not include it. Defaults to {}.

  • replace_standard_key_values (AppendableSelector[KeyStringValues]) – Replace values for standard keys in TEXT. Comparisons are case insensitive. The leading "$" is implied so do not include it. Defaults to {}.

  • append_standard_keywords (AppendableSelector[KeyStringValues]) – Append standard key/value pairs to TEXT. All keys and values will be included as they appear here. The leading "$" is implied so do not include it. Defaults to {}.

  • substitute_standard_key_values (AppendableSelector[SubPatterns]) – Apply sed-like substitution operation on matching standard keys. The leading "$" is implied when matching keys. Defaults to {}.

  • allow_repair_non_unique (TriFlag) – Choose how to handle key collisions when repairing keywords. Non-unique keywords will not be kept in the final FCS file since each list of standard and non-standard keywords must be unique. If "false", raise ConfigError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • text_data_correction (OffsetCorrection) – Corrections for DATA offsets in TEXT. Defaults to (0, 0).

  • text_analysis_correction (OffsetCorrection) – Corrections for ANALYSIS offsets in TEXT. Defaults to (0, 0).

  • ignore_text_data_offsets (bool) – If True ignore DATA offsets in TEXT Defaults to False.

  • ignore_text_analysis_offsets (bool) – If True ignore ANALYSIS offsets in TEXT Defaults to False.

  • allow_header_text_offset_mismatch (AllowHeaderTextOffsetMismatch) – Choose what to do if HEADER and TEXT offsets are different. Exception will be FileLayoutError if emitted. Defaults to "error".

  • allow_missing_required_offsets (TriFlag) – Choose what happens when required DATA and ANALYSIS (3.1 or lower) offsets in TEXT are be missing. If missing, fall back to offsets from HEADER. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • process_optional_failure (ProcessKeywordFailure) – Process optional keys which cause an error. Defaults to "error".

  • int_width_override (IntWidthOverride) – Override $PnB. Only affects integer layouts in FCS 2.0/3.0. Defaults to "never".

  • byteord_override (ByteordOverride) – Override $BYTEORD. Only affects integer layouts in FCS 2.0/3.0. Defaults to "none".

  • disallow_range_truncation (TriFlag) – Choose how to handle $PnR values that need to be truncated to match the number of bytes specified by $PnB and $DATATYPE. If "false", throw warning. If "true", raise RelationalError. If "silent", do nothing. Defaults to "false".

  • data_remainder_limit (int) – Limit by which ending DATA offset can be truncated if its length modulo event width produces a remainder. Defaults to 0.

  • allow_uneven_event_width (TriFlag) – Choose what to do when event width does not perfectly divide length of DATA. Does not apply to delimited ASCII data schema. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_tot_mismatch (TriFlag) – Choose what happens when $TOT does not match number of events as computed by the event width and length of DATA. Does not apply to delimited ASCII data schema. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • over_bitmask_action (OverLimitAction) – Choose what to do with event integer values in DATA which exceed bitmask. Defaults to "trunc_warn".

  • over_range_action (OverLimitAction) – Choose what to do with event values in DATA which exceed $PnR. Defaults to "warn".

  • allow_missing_crc (TriFlag) – Choose what to do when CRC is missing from the end of a dataset. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • allow_mismatch_crc (TriFlag) – Choose what to do when computed CRC and CRC at the end of a dataset do not match. If "false", raise FileLayoutError. If "true", throw warning. If "silent", do nothing. Defaults to "false".

  • compute_crc (ComputeCRC) – Choose when to compute the CRC for a dataset. Defaults to "never".

  • read_intra_segment_dark_bytes (bool) – If True read bytes which are between segments. Defaults to False.

  • read_post_dataset_dark_bytes (bool) – If True read bytes between the end of the current dataset and the next. Defaults to False.

  • row_buffer_size (int) – Set the size in bytes for the internal buffer used to read DATA. This is a performance parameter that balances read syscalls (too low) and cache misses (too high). It should generally be 90% of the CPU’s L1D cache size. Defaults to 28000.

  • warnings_are_errors (bool) – If True all warnings will be regarded as errors. Defaults to False.

  • hide_warnings (bool) – If True hide all warnings. Defaults to False.

Return type:

list[DatasetSummary]

Raises:

Outputs#

These are neatly bundled classes of data returned by each of the functions above.

class pyreflow.api.Header(dataset_offset, version, final_offsets, original_offsets, overlaps, dark_bytes, read_header_ns)#

The HEADER segment from an FCS dataset.

Variables:

  • dataset_offset (int) – (read-only) Starting position in the file of the dataset to be read.

  • version (FCSVersion) – (read-only) The FCS version.

  • final_offsets (FinalHeaderOffsets) – (read-only) The offsets from HEADER after corrections were applied.

  • original_offsets (OriginalHeaderOffsets) – (read-only) The original offset pairs from HEADER as written in the file.

  • overlaps (list[HeaderToHeaderOffsetsOverlap]) – (read-only) Overlaps between HEADER offset pairs.

  • dark_bytes (DarkBytes | None) – (read-only) Bytes between the end of the HEADER and the first segment.

  • read_header_ns (int) – (read-only) Number of nanoseconds spent reading HEADER

class pyreflow.api.FlatTEXTOutput(keywords, flat_diagnostics)#

Parsed HEADER and TEXT.

Variables:

  • keywords (ValidKeywords) – (read-only) Standard and non-standard keywords.

  • flat_diagnostics (FlatTEXTDiagnostics) – (read-only) Diagnostic data obtained when parsing TEXT.

class pyreflow.api.StdTEXTOutput(tot, dataset_offsets, repair_diagnostics, std_diagnostics, flat_diagnostics, version_scores)#

Miscellaneous data when standardizing TEXT.

Variables:

  • tot (int | None) – (read-only) Value of $TOT from TEXT.

  • dataset_offsets (DatasetOffsets) – (read-only) Offsets used to parse DATA and ANALYSIS.

  • repair_diagnostics (RepairDiagnostics) – (read-only) Diagnostic output from repairing keyword list

  • std_diagnostics (StdTEXTDiagnostics) – (read-only) Diagnostic output from TEXT standardization

  • flat_diagnostics (FlatTEXTDiagnostics) – (read-only) Diagnostic data obtained when parsing TEXT.

  • version_scores (KeywordVersionScores | None) – (read-only) Scores generated if version was guessed.

class pyreflow.api.FlatDatasetOutput(keywords, flat_diagnostics, dataset, version_scores)#

Dataset from FCS file parsed with flat mode.

Variables:
class pyreflow.api.StdDatasetOutput(dataset, flat_diagnostics, version_scores)#

Miscellaneous data when standardizing TEXT.

Variables:
class pyreflow.api.NewFlatDatasetFromKwsOutput(dataset, header)#

Output of using keywords to crate new standardized TEXT and DATA.

Variables:
class pyreflow.api.NewStdDatasetFromKwsOutput(dataset, header)#

Output of using keywords to crate new standardized TEXT and DATA.

Variables:
class pyreflow.api.DatasetSummary(version, text_len, data_len, analysis_len, n_events, n_measurements, n_other, others_len, datatype, dataset_offset, file_crc, computed_crc, read_header_ns, read_text_ns, read_schema_ns, read_data_ns, check_range_ns, read_other_analysis_ns, read_crc_ns, read_dark_bytes_ns, scan_next_ns)#

High-level data describing an FCS dataset

Variables:

  • version (FCSVersion) – (read-only) The FCS version.

  • text_len (int) – (read-only) Length of TEXT (in bytes)

  • data_len (int) – (read-only) Length of DATA (in bytes)

  • analysis_len (int) – (read-only) Length of ANALYSIS (in bytes)

  • n_events (int) – (read-only) Number of events ($TOT)

  • n_measurements (int) – (read-only) Number of measurements ($PAR)

  • n_other (int) – (read-only) Number of OTHER segments

  • others_len (int) – (read-only) Total length of OTHER segments (in bytes)

  • datatype (Datatype | None) – (read-only) The value of $DATATYPE

  • dataset_offset (int) – (read-only) Starting position in the file of the dataset to be read.

  • file_crc (CRCOutput) – (read-only) The value of the cyclic redundancy check (CRC) value. FCS 2.0 will always return None.

  • computed_crc (int | None) – (read-only) The value of the cyclic redundancy check (CRC) value as computed based on file contents. Will be None for 2.0 or if the the user chose not to compute the CRC.

  • read_header_ns (int) – (read-only) Number of nanoseconds spent reading HEADER.

  • read_text_ns (int) – (read-only) Number of nanoseconds spent reading TEXT.

  • read_schema_ns (int) – (read-only) Number of nanoseconds spent reading the schema for DATA.

  • read_data_ns (int) – (read-only) Number of nanoseconds spent reading DATA.

  • check_range_ns (int) – (read-only) Number of nanoseconds spent checking DATA against $PnR.

  • read_other_analysis_ns (int) – (read-only) Number of nanoseconds spent reading OTHER and/or ANALYSIS.

  • read_crc_ns (int) – (read-only) Number of nanoseconds spent computing the CRC.

  • read_dark_bytes_ns (int) – (read-only) Number of nanoseconds spent reading dark bytes.

  • scan_next_ns (int) – (read-only) Number of nanoseconds spent scanning for the next dataset.

Diagnostic outputs#

These are clusters of output data specifically meant to “diagnose” problems that were identified with FCS files (example, keywords that were dropped because they could not be parsed). Use these to reverse-engineer what problems an FCS file may have.

class pyreflow.api.FlatTEXTDiagnostics(header_supp, primary_text_overflow, header_overflows, byte_pairs, non_unique_std_keywords, non_unique_nonstd_keywords, keys_with_empty_trimmed_values, keys_with_trimmed_values, read_text_ns, primary_split, supp_split)#

Diagnostic data from parsing TEXT.

Variables:

  • header_supp (HeaderAndSuppOffsets) – (read-only) HEADER data and supplemental TEXT offsets.

  • primary_text_overflow (int) – (read-only) The amount by which primary TEXT exceeds end of file.

  • header_overflows (list[HeaderOffsetsOverflow]) – (read-only) Offsets from HEADER which exceed $NEXTDATA or end of file.

  • byte_pairs (list[tuple[KeyOrBytes, NEStrOrBytes]]) – (read-only) Keywords with keys that are not ASCII or values that are not UTF-8.

  • non_unique_std_keywords (list[tuple[StdKey, NEStr]]) – (read-only) Standard keys which already appeared in TEXT previously.

  • non_unique_nonstd_keywords (list[tuple[NonStdKey, NEStr]]) – (read-only) Nonstandard keys which already appeared in TEXT previously.

  • keys_with_empty_trimmed_values (list[KeyOrBytes]) – (read-only) Keys with empty values as a result of trimming whitespace.

  • keys_with_trimmed_values (list[tuple[KeyOrBytes, NEStr]]) – (read-only) Keys with values that are not empty after whitespace was trimmed off.

  • read_text_ns (int) – (read-only) Number of nanoseconds spent reading TEXT.

  • primary_split (SplitTEXTDiagnostics) – (read-only) Additional parsing diagnostics for primary TEXT.

  • supp_split (SplitTEXTDiagnostics | None) – (read-only) Additional parsing diagnostics for supplemental TEXT.

class pyreflow.api.SplitTEXTDiagnostics(delimiter, escaped, keys_with_blank_values, values_with_blank_keys, skipped_pairs, tokens_with_boundary_delims, last_odd_token, has_even_delims, extra_leading_delims, multibyte_encoded)#

Diagnostic data when parsing a specific TEXT segment.

Variables:

  • delimiter (int) – (read-only) Delimiter used to parse TEXT.

  • escaped (bool) – (read-only) True if delimiters were escaped.

  • keys_with_blank_values (list[NEStrOrBytes]) – (read-only) Keys which have blank values (relatively common).

  • values_with_blank_keys (list[NEStrOrBytes]) – (read-only) Values which have blank keys (relatively rare).

  • skipped_pairs (int) – (read-only) Number of key/value pairs that were skipped because both were blank.

  • tokens_with_boundary_delims (list[NEStrOrBytes]) – (read-only) Tokens (keys or values) which have delimiters at their boundary.

  • last_odd_token (str | bytes) – (read-only) Last token if the number of tokens is odd (empty if not present).

  • has_even_delims (bool) – (read-only) True if TEXT has an even number of delimiters.

  • extra_leading_delims (int) – (read-only) The number of delimiters at the front of TEXT (excluding the first).

  • multibyte_encoded (bool) – (read-only) True if TEXT was encoded with UTF-8, False if encoded with Latin-1.

class pyreflow.api.StdTEXTDiagnostics(optional, pseudostandard, hyper_par, hyper_gate, other_version, timestep, dedup_names, scale, gate_scale, trimmed, temporal_optical_pairs, timestep_added, spillover_was_indexed, btim_pattern, etim_pattern, date_pattern, begindatetime_pattern, enddatetime_pattern, begindatetime_used_localtime, enddatetime_used_localtime, last_modified_pattern, read_std_ns, schema_diagnostics)#

Diagnostic output from TEXT standardization.

Variables:

  • optional (StdKeywords) – (read-only) Optional standard keywords which failed parsing and were dropped.

  • pseudostandard (StdKeywords) – (read-only) Keywords which start with "$" but are not part of the standard.

  • hyper_par (StdKeywords) – (read-only) Measurement keywords which are part of the standard but have an index outside $PAR.

  • hyper_gate (StdKeywords) – (read-only) Gating keywords which are part of the standard but have an index outside $GATE.

  • other_version (StdKeywords) – (read-only) Keywords which are from a different FCS version.

  • timestep (NEStr | None) – (read-only) Unused $TIMESTEP keyword

  • dedup_names (list[Shortname | None]) – (read-only) Original $PnN if they were renamed.

  • scale (list[MeasScaleDiagnostic]) – (read-only) Diagnostic data from parsing $PnE keywords.

  • gate_scale (list[GateScaleDiagnostic]) – (read-only) Diagnostic data from parsing $GmE keywords.

  • trimmed (list[tuple[StdKey, NEStr]]) – (read-only) Keywords which had whitespace between commas trimmed.

  • temporal_optical_pairs (list[tuple[StdKey, NEStr]]) – (read-only) Optical keys that were found in the temporal measurement.

  • timestep_added (bool) – (read-only) True if $TIMESTEP was missing and added via configuration.

  • spillover_was_indexed (bool | None) – (read-only) True if $SPILLOVER used indices rather than names.

  • btim_pattern (str | None) – (read-only) Alternative pattern used to parse $BTIM.

  • etim_pattern (str | None) – (read-only) Alternative pattern used to parse $ETIM.

  • date_pattern (str | None) – (read-only) Alternative pattern used to parse $DATE.

  • begindatetime_pattern (str | None) – (read-only) Alternative pattern used to parse $BEGINDATETIME.

  • enddatetime_pattern (str | None) – (read-only) Alternative pattern used to parse $ENDDATETIME.

  • begindatetime_used_localtime (bool | None) – (read-only) True if localtime was used when parsing $BEGINDATETIME.

  • enddatetime_used_localtime (bool | None) – (read-only) True if localtime was used when parsing $ENDDATETIME.

  • last_modified_pattern (str | None) – (read-only) Alternative pattern used to parse $LAST_MODIFIED.

  • read_std_ns (int) – (read-only) Number of nanoseconds spent standardizing TEXT.

  • schema_diagnostics (DataSchemaDiagnostics) – (read-only) Diagnostic output from parsing the data schema

Raises:

  • ConfigError – if btim_pattern or etim_pattern does not have specifiers for hours, minutes, seconds, and optionally sub-seconds (where "%!" and "%@" correspond to 1/60 seconds and centiseconds respectively) as outlined in chrono

  • ConfigError – if date_pattern does not have year, month, and day specifiers as outlined in chrono

  • ParseKeywordValueError – if any in dedup_names is "" or contains commas

class pyreflow.api.RepairDiagnostics(non_unique_std, non_unique_nonstd, demoted, promoted, subbed, replaced, renamed, ignored, removed)#

Diagnostic output from repairing the keyword list.

Variables:

  • non_unique_std (list[tuple[StdKey, NEStr]]) – (read-only) Standard keys which where promoted and collided with existing keys.

  • non_unique_nonstd (list[tuple[NonStdKey, NEStr]]) – (read-only) Non-standard keys which where demoted and collided with existing keys.

  • demoted (list[StdKey]) – (read-only) Standard keys which were demoted.

  • promoted (list[NonStdKey]) – (read-only) Non-standard keys which were promoted.

  • subbed (list[tuple[StdKey, NEStr]]) – (read-only) Standard keys which had values that were edited by substitution.

  • replaced (list[tuple[StdKey, NEStr]]) – (read-only) Standard keys which had values that were replaced.

  • renamed (list[tuple[StdKey, StdKey]]) – (read-only) Standard keys which were renamed. The first element of the pair is the original name.

  • ignored (list[tuple[StdKey, NEStr]]) – (read-only) Standard keys which were ignored.

  • removed (list[tuple[StdKey, NEStr]]) – (read-only) Standard keys which were removed.

class pyreflow.api.DataSchemaDiagnostics(truncated_columns, original_int_width, original_byteord, read_schema_ns)#

Diagnostic output from creating data schema.

Variables:

  • truncated_columns (list[Decimal | None]) – (read-only) Original values of $PnR if they were truncated. Length of list will be equal to $PAR.

  • original_int_width (int | None) – (read-only) Original $PnB value if was changed. Only applies to integer schemas for 2.0/3.0. Will be None if $PnB was explicitly overridden.

  • original_byteord (list[int] | None) – (read-only) Original $BYTEORD value if it was changed. Only applies to integer schemas for 2.0/3.0.

  • read_schema_ns (int) – (read-only) Number of nanoseconds spent reading schema for DATA.

class pyreflow.api.DatasetDiagnostics(event_width, event_data_remainder, tot_event_mismatch, overrange_columns, intra_segment_dark_bytes, post_dataset_dark_bytes, file_crc, computed_crc, dataset_len, next_dataset_offset, next_dataset_manually_scanned, read_data_ns, check_range_ns, read_other_analysis_ns, read_crc_ns, read_dark_bytes_ns, scan_next_ns)#

Diagnostic output from reading DATA segment.

Variables:

  • event_width (int | None) – (read-only) The width of one event in bytes (if not ASCII delimited).

  • event_data_remainder (int | None) – (read-only) The remainder after dividing length of DATA by event width.

  • tot_event_mismatch (bool | None) – (read-only) True if $TOT does not match the number of events computed via event width.

  • overrange_columns (list[tuple[int, bool] | None]) – (read-only) Columns for which at least one event was out of range via $PnR. Each index corresponds to a column in DATA. Elements will be None if not overrange at all, otherwise the first integer is the first row that is overrange and the second boolean is True if the value was truncated.

  • intra_segment_dark_bytes (list[IntraSegmentDarkBytes]) – (read-only) Unparsed bytes between segments.

  • post_dataset_dark_bytes (DarkBytes | None) – (read-only) Unparsed bytes between the end of this dataset and the beginning of the next.

  • file_crc (CRCOutput) – (read-only) The value of the cyclic redundancy check (CRC) value. FCS 2.0 will always return None.

  • computed_crc (int | None) – (read-only) The value of the cyclic redundancy check (CRC) value as computed based on file contents. Will be None for 2.0 or if the the user chose not to compute the CRC.

  • dataset_len (int) – (read-only) The length of the entire dataset in bytes.

  • next_dataset_offset (int | None) – (read-only) The offset of the next dataset if it exists. This can be obtained either from $NEXTDATA or by manually scanning the file for the next dataset. Will be None if this is the last dataset in the FCS file.

  • next_dataset_manually_scanned (bool) – (read-only) True if the value of next_dataset_offset was found by manually scanning the file.

  • read_data_ns (int) – (read-only) Number of nanoseconds spent reading DATA.

  • check_range_ns (int) – (read-only) Number of nanoseconds spent checking DATA against $PnR.

  • read_other_analysis_ns (int) – (read-only) Number of nanoseconds spent reading OTHER and/or ANALYSIS.

  • read_crc_ns (int) – (read-only) Number of nanoseconds spent computing the CRC.

  • read_dark_bytes_ns (int) – (read-only) Number of nanoseconds spent reading dark bytes.

  • scan_next_ns (int) – (read-only) Number of nanoseconds spent scanning for the next dataset.

Offset Outputs#

Data corresponding to the offsets that were used to parse an FCS file.

class pyreflow.api.OriginalHeaderOffsets(text, data, analysis, others)#

The uncorrected segments from HEADER

Variables:

  • text (OriginalOffsets) – (read-only) The uncorrected Primary TEXT offsets from HEADER.

  • data (OriginalOffsets) – (read-only) The uncorrected DATA offsets from HEADER.

  • analysis (OriginalOffsets) – (read-only) The uncorrected ANALYSIS offsets from HEADER.

  • others (list[OriginalOffsets]) – (read-only) The uncorrected OTHER segments from HEADER.

class pyreflow.api.FinalHeaderOffsets(text, data, analysis, others)#

The offsets from HEADER.

Variables:

  • text (FinalOffsets) – (read-only) The primary TEXT offsets from HEADER.

  • data (FinalOffsets) – (read-only) The DATA offsets from HEADER.

  • analysis (FinalOffsets) – (read-only) The ANALYSIS offsets from HEADER.

  • others (FinalOtherOffsets | None) – (read-only) The OTHER offsets from HEADER.

class pyreflow.api.HeaderAndSuppOffsets(header, supp_text, nextdata)#

HEADER data and supplemental TEXT offsets.

Variables:

  • header (Header) – (read-only) The HEADER from parsed file

  • supp_text (SuppTEXTOffsetsOutput) – (read-only) The supplemental TEXT offsets and associated parse data.

  • nextdata (int | None) – (read-only) The value of $NEXTDATA.

class pyreflow.api.DatasetOffsets(final_data_offsets, final_analysis_offsets, data_origin, analysis_origin, data_analysis_overlap)#

Segments used to parse DATA and ANALYSIS

Variables:

  • final_data_offsets (FinalOffsets) – (read-only) The DATA offsets from HEADER or TEXT.

  • final_analysis_offsets (FinalOffsets) – (read-only) The ANALYSIS offsets from HEADER or TEXT.

  • data_origin (TEXTOffsetsOrigin) – (read-only) The origin of the offsets that were actually used to read DATA.

  • analysis_origin (TEXTOffsetsOrigin) – (read-only) The origin of the offsets that were actually used to read ANALYSIS.

  • data_analysis_overlap (int | None) – (read-only) The overlap between DATA and ANALYSIS if applicable.

class pyreflow.api.SuppTEXTOffsetsOutput(origin_type, final_offsets, original_offsets, other_index, overlaps, overflow)#

Output from reading supplemental TEXT offsets from file.

The meaning of each configuration and possible values for other attributes is determined by origin_type:

  • "empty": offsets are empty. Nothing was done. All other attributes are null. 2.0 files will always return this since they do not have supplemental TEXT at all.

  • "unparsed": offsets were required based on version but could not be parsed, either because they were missing entirely or the keywords were not valid numbers. All other attributes are null.

  • "malformed": offsets were required based on version, were parsed to digits, but did not form valid offsets. original_offsets will have the original offsets.

  • "dup_ptext": offsets were parsed and were exact duplicates of primary TEXT and were thus ignored. All other attributes are null since this situation adds no more new information.

  • "dup_analysis": offsets were parsed and were exact duplicates of ANALYSIS and were thus ignored. All other attributes are null since this situation adds no more new information.

  • "ignored": offsets were parsed but ignored by user request. If keywords were valid and parsed to digits, they will be returned via original_offsets.

  • "dup_other": offsets were parsed and were exact duplicates of an OTHER segment. In this case, the segment was assumed to be a real supplemental TEXT segment and thus was read while the OTHER offsets were ignored. The index of the matching OTHER offsets is recorded in other_index. The final and original offsets are returned in final_offsets and original_offsets. Overlaps are recorded in overlaps and overflow if applicable.

  • "valid": offsets were parsed and valid; they did not overlap anything else. other_index will be null. The final and original offsets are returned in final_offsets and original_offsets. Overlaps are recorded in overlaps and overflow if applicable.

Variables:

  • origin_type (SuppTEXTOffsetsOriginType) – (read-only) The discrete configuration of the supplemental TEXT offsets that produced this configuration.

  • final_offsets (FinalOffsets | None) – (read-only) The final offsets used to read supplemental TEXT.

  • original_offsets (OriginalOffsets | None) – (read-only) The original supplemental TEXT offsets as written in the file.

  • other_index (int | None) – (read-only) The index of the OTHER offsets which duplicate supplemental TEXT if applicable.

  • overlaps (list[SuppToHeaderOffsetsOverlap]) – (read-only) Overlaps between supplemental TEXT and HEADER offsets.

  • overflow (SuppOffsetsOverflow | None) – (read-only) Amount by which supp TEXT exceeded $NEXTDATA or end of file.

class pyreflow.api.TEXTOffsetsOrigin(origin_type, original_offsets, overlaps, overflow)#

Output from reading DATA or ANALYSIS offsets from TEXT in file.

The meaning of each configuration and possible values for other attributes is determined by origin_type:

  • "empty_text": offsets are empty and those from HEADER were used. All other attributes are null. 2.0 files will always return this since they do not have TEXT offset keywords at all.

  • "ignored": offsets were parsed but ignored by user request. If keywords were valid and parsed to digits, they will be returned via original_offsets.

  • "unparsed": offsets were required based on version but could not be parsed, either because they were missing entirely or the keywords were not valid numbers. All other attributes are null.

  • "malformed": offsets were required based on version, were parsed to digits, but did not form valid offsets. original_offsets will have the original offsets.

  • "match": offsets exactly match those in HEADER. All attributes are null since this situation provides no new information.

  • "mismatch_header": offsets mismatch those in HEADER and HEADER was chosen via user config. original_offsets will have the original offsets.

  • "mismatch_text": offsets mismatch those in HEADER and TEXT was chosen via user config. The original offsets are returned in original_offsets. Overlaps are recorded in overlaps and overflow if applicable.

  • "empty_header": HEADER offsets were empty and TEXT were parsed and found to be valid. original_offsets will have the original offsets. Overlaps are recorded in overlaps and overflow if applicable.

Variables:

  • origin_type (TEXTOffsetsOriginType) – (read-only) The discrete configuration of these offsets that produced this configuration.

  • original_offsets (OriginalOffsets | None) – (read-only) The original offsets as written in the file.

  • overlaps (list[TextToHeaderOrSuppOffsetsOverlap]) – (read-only) Overlaps between these offsets TEXT and HEADER/supp TEXT offsets.

  • overflow (TextOffsetsOverflow | None) – (read-only) Amount by which this offset exceeded $NEXTDATA or end of file.

Offset Overflow Outputs#

These classes are produced when an offset pair exceeds $NEXTDATA or EOF (end of file)

class pyreflow.api.HeaderOffsetsOverflow(offsets, overflow, dataset_len, bound_is_nextdata)#

An offset pair which exceeded $NEXTDATA or end of file

Variables:

  • offsets (HeaderNamedOffsets) – (read-only) The offsets for this overflow

  • overflow (int) – (read-only) The size of the overflow in bytes

  • dataset_len (int) – (read-only) The length of the dataset that was exceeded.

  • bound_is_nextdata (bool) – (read-only) True if the dataset length was determine by $NEXTDATA; False if determine by end of file

class pyreflow.api.TextOffsetsOverflow(offsets, overflow, dataset_len, bound_is_nextdata)#

An offset pair which exceeded $NEXTDATA or end of file

Variables:

  • offsets (TextNamedOffsets) – (read-only) The offsets for this overflow

  • overflow (int) – (read-only) The size of the overflow in bytes

  • dataset_len (int) – (read-only) The length of the dataset that was exceeded.

  • bound_is_nextdata (bool) – (read-only) True if the dataset length was determine by $NEXTDATA; False if determine by end of file

class pyreflow.api.SuppOffsetsOverflow(offsets, overflow, dataset_len, bound_is_nextdata)#

An offset pair which exceeded $NEXTDATA or end of file

Variables:

  • offsets (SuppTEXTNamedOffsets) – (read-only) The offsets for this overflow

  • overflow (int) – (read-only) The size of the overflow in bytes

  • dataset_len (int) – (read-only) The length of the dataset that was exceeded.

  • bound_is_nextdata (bool) – (read-only) True if the dataset length was determine by $NEXTDATA; False if determine by end of file

Offset-Offset Overlap Outputs#

These classes are produced when two offsets overlap each other.

class pyreflow.api.HeaderToHeaderOffsetsOverlap(offsets0, offsets1, overlap)#

Overlap between offset pairs from HEADER.

Variables:

  • offsets0 (HeaderNamedOffsets) – (read-only) The first offset pair in this overlap

  • offsets1 (HeaderNamedOffsets) – (read-only) The second offset pair in this overlap

  • overlap (int) – (read-only) The size of the overlap in bytes

class pyreflow.api.TextToHeaderOffsetsOverlap(offsets0, offsets1, overlap)#

Overlap between an offsets from HEADER and an offsets from TEXT.

Variables:

  • offsets0 (TextNamedOffsets) – (read-only) The first offset pair in this overlap

  • offsets1 (HeaderNamedOffsets) – (read-only) The second offset pair in this overlap

  • overlap (int) – (read-only) The size of the overlap in bytes

class pyreflow.api.SuppToHeaderOffsetsOverlap(offsets0, offsets1, overlap)#

Overlap between supplemental TEXT offsets and offsets from HEADER.

Variables:

  • offsets0 (SuppTEXTNamedOffsets) – (read-only) The first offset pair in this overlap

  • offsets1 (HeaderNamedOffsets) – (read-only) The second offset pair in this overlap

  • overlap (int) – (read-only) The size of the overlap in bytes

class pyreflow.api.TextToHeaderOrSuppOffsetsOverlap(offsets0, offsets1, overlap)#

Overlap between TEXT offset pair and HEADER or supp TEXT pair.

Variables:

  • offsets0 (TextNamedOffsets) – (read-only) The first offset pair in this overlap

  • offsets1 (HeaderOrSuppNamedOffsets) – (read-only) The second offset pair in this overlap

  • overlap (int) – (read-only) The size of the overlap in bytes

Misc outputs#

class pyreflow.api.ValidKeywords(std, nonstd)#

Standard and non-standard keywords.

Variables:
class pyreflow.api.IntraSegmentDarkBytes(prev, next, start, end, bytes)#

Unparsed bytes which are between two segments in an FCS file.

Variables:

  • prev (FlankingSegmentName) – (read-only) Identifier for the segment immediately prior.

  • next (FlankingSegmentName) – (read-only) Identifier for the segment immediately after.

  • start (int) – (read-only) The starting offset of this region.

  • end (int) – (read-only) The final offset of this region (one greater than offset of the last byte).

  • bytes (DarkBytes) – (read-only) The byte contents of this region.

class pyreflow.api.FlatDatasetFromKwsOutput(data, analysis, others, dataset_offsets, repair_diagnostics, schema_diagnostics, dataset_diagnostics)#

Dataset from parsing flat TEXT.

Variables:

  • data (DataFrame) – (read-only) A dataframe encoding the contents of DATA. Number of columns must match number of measurements. May be empty. Types do not necessarily need to correspond to those in the data schema but mismatches may result in truncation.

  • analysis (AnalysisBytes) – (read-only) Contents of the ANALYSIS segment.

  • others (list[OtherBytes]) – (read-only) A list of (byte) strings encoding the OTHER segments.

  • dataset_offsets (DatasetOffsets) – (read-only) Offsets used to parse DATA and ANALYSIS.

  • repair_diagnostics (RepairDiagnostics) – (read-only) Diagnostic output from repairing keyword list

  • schema_diagnostics (DataSchemaDiagnostics) – (read-only) Diagnostic output from parsing the data schema

  • dataset_diagnostics (DatasetDiagnostics) – (read-only) Diagnostic output from parsing DATA segment.

Raises:

EventDataError – If data contains columns which are not unsigned 8/16/32/64-bit integers or 32/64-bit floats

class pyreflow.api.StdDatasetFromKwsOutput(dataset_offsets, repair_diagnostics, std_diagnostics, dataset_diagnostics)#

Miscellaneous data when standardizing TEXT from keywords.

Variables:

  • dataset_offsets (DatasetOffsets) – (read-only) Offsets used to parse DATA and ANALYSIS.

  • repair_diagnostics (RepairDiagnostics) – (read-only) Diagnostic output from repairing keyword list

  • std_diagnostics (StdTEXTDiagnostics) – (read-only) Diagnostic output from TEXT standardization

  • dataset_diagnostics (DatasetDiagnostics) – (read-only) Diagnostic output from parsing DATA segment.

class pyreflow.api.KeywordVersionScore(good_req, good_opt, drop, missing_opt, missing_req, missing_absent, incompatible_widths)#

Score generated when guessing version from keywords.

Variables:

  • good_req (int) – (read-only) Number of required keywords expected to be in this version and found.

  • good_opt (int) – (read-only) Number of optional keywords expected to be in this version and found.

  • drop (int) – (read-only) Number of keywords (opt or req) that must be dropped for this version.

  • missing_opt (int) – (read-only) Number of optional keywords that are missing in this version.

  • missing_req (int) – (read-only) Number of required keywords that are missing in this version.

  • missing_absent (int) – (read-only) Number of keywords that are expected to be missing for this version.

  • incompatible_widths (bool) – (read-only) The $PnB values are incompatible with this version. This will only be True if version is 2.0 and 3.0 and the $PnB values contain multiple widths across them.

Dynamic value selectors based on keywords#

Some options can be selectively chosen based on the value of keywords present in the FCS dataset. For instance, some files name the time measurement as "HDR-T" rather than the compliant "TIME" or "Time", so the value of time_meas_pattern must be set accordingly depending on the machine in use (which often be found with the $CYT keyword).

These type aliases look complicated but are actually Lisp-like expressions for the functions if and cond with specialized identifiers which evaluated to true or false depending on if a keyword is present and/or a certain value.

The above case for "HDR-T" (with a new AI-powered cytometer from Cytodyne Sytems named "T-1000") can be written as ("if", ("key_is", "$CYT", "T-1000"), "HDR-T").

type pyreflow.typing.Selector = T | tuple[Literal['if'], Condition, Selector] | tuple[Literal['if'], Condition, Selector, Selector] | tuple[Literal['cond'], list[tuple[Condition, Selector]]]#

A dynamic selector for a type based on contents of an FCS file.

This is used to select certain configuration options based on the keywords of an FCS file.

The type can be included simply by itself, in which case no selection will occur.

Alternatively, the type can be embedded in a series of Lisp-like conditional statements represented as Python tuples.

If the tuple’s first element is "if", the second must be a condition, the third must be a another selector (the same as this type) which will be evaluated if the condition is true, and the fourth must be another selector or None which will be evaluated if the condition is false.

If the tuple’s first element is "cond", each subsequent element must be a tuple pair with a condition and and a statement to be evaluated if the condition is true. These conditions will be evaluated in series until the the first true case.

If this evaluates to None, the default for the underlying type T will be chosen.

type pyreflow.typing.AppendableSelector = Selector | list[Selector]#

Like a Selector but can include a list.

The values of the results of each individual selector will be concatenated.

type pyreflow.typing.Condition = KeyTest | tuple[Literal['and'], KeyTest, KeyTest] | tuple[Literal['or'], KeyTest, KeyTest] | tuple[Literal['not'], KeyTest]#

An expression which evaluates to true/false depending on FCS keywords.

This is either a bare KeyTest which will evaluated directly or a List-like set of tuples that represent conditional logic. The first element in each tuple is a “logical function” which will evaluate to true or false depending on the boolean outputs of the arguments that follow.

type pyreflow.typing.KeyTest = tuple[Literal['has_key'], str] | tuple[Literal['key_is'], str, str] | tuple[Literal['key_matches'], str, str]#

Evaluates to true/false depending on the value of an FCS keyword.

These are Lisp-like expressions represented as Python tuples where the first element is a “function” which is run with the arguments that follow.

If "has_key", return true if the indicated key is present.

If "key_is", return true if the indicated key (first argument) has a value exactly equal to the second argument.

If "key_matches", return true if the indicated key (first argument) has a value which matches the regular expression (second argument). The regexp must follow the syntax of the rust regexp crate.

Keys can either be standard (start with "$") or non-standard (no "$").