Configuration Interface#

Overview#

pyreflow by default will only read fully standards compliant FCS files.

However, since most files are not standards compliant, pyreflow offers several general configuration strategies in order to deal with the real world.

These are summarized as follows:

  • Scalpal strategy: This will attempt to fix as many mistakes as possible in an FCS file while preserving non-trivial metadata. Trivial metadata includes, whitespace, keys with blank values, and repeated keys. If non-trivial data cannot be preserved, this strategy will fail.

  • Sledgehammer strategy: This is optimized to read DATA at the expense of metadata. Non standard keywords (unless they are required for parsing DATA) will be dropped. Segments such as ANALYSIS and OTHER may not be read.

These strategies only apply to reading FCS files. pyreflow will only write compliant files so in this case these strategies are irrelevent.

Dictionary Configuration#

Each of these strategies is implemented in pyreflow via default configuration dictionaries which can be splatted as function arguments using **.

Since these are just dictionaries, they can be modified after creating to fine-tune the options in a given strategy.

class pyreflow.api.ReadHeaderConfig#

Config for ~pyreflow.fcs_read_header.

classmethod scalpal()#

Return non-compliant configuration optimized to preserve data.

All non-trivial metadata (ie whitespace, blank keys, etc) will be preserved.

Return type:

dict[str, Any]

classmethod sledgehammer()#

Return non-compliant configuration optimized to read data.

Metadata may be destroyed or dropped.

Return type:

dict[str, Any]

classmethod strict()#

Return standards-compliant configuration.

Return type:

dict[str, Any]

class pyreflow.api.ReadFlatTEXTConfig#

Config for reading flat TEXT.

Can be used with ~pyreflow.fcs_read_flat_text and ~pyreflow.fcs_read_flat_texts.

classmethod scalpal()#

Return non-compliant configuration optimized to preserve data.

All non-trivial metadata (ie whitespace, blank keys, etc) will be preserved.

Return type:

dict[str, Any]

classmethod sledgehammer()#

Return non-compliant configuration optimized to read data.

Metadata may be destroyed or dropped.

Return type:

dict[str, Any]

classmethod strict()#

Return standards-compliant configuration.

Return type:

dict[str, Any]

class pyreflow.api.ReadStdTEXTConfig#

Config for reading standardized TEXT.

Can be used with ~pyreflow.fcs_read_std_text and ~pyreflow.fcs_read_std_texts.

classmethod scalpal()#

Return non-compliant configuration optimized to preserve data.

All non-trivial metadata (ie whitespace, blank keys, etc) will be preserved.

Return type:

dict[str, Any]

classmethod sledgehammer()#

Return non-compliant configuration optimized to read data.

Metadata may be destroyed or dropped.

Return type:

dict[str, Any]

classmethod strict()#

Return standards-compliant configuration.

Return type:

dict[str, Any]

class pyreflow.api.ReadFlatDatasetConfig#

Config for reading flat TEXT and DATA.

Can be used with ~pyreflow.fcs_read_flat_dataset and ~pyreflow.fcs_read_flat_datasets.

classmethod scalpal()#

Return non-compliant configuration optimized to preserve data.

All non-trivial metadata (ie whitespace, blank keys, etc) will be preserved.

Return type:

dict[str, Any]

classmethod sledgehammer()#

Return non-compliant configuration optimized to read data.

Metadata may be destroyed or dropped.

Return type:

dict[str, Any]

classmethod strict()#

Return standards-compliant configuration.

Return type:

dict[str, Any]

class pyreflow.api.ReadStdDatasetConfig#

Config for reading flat TEXT and DATA.

Can be used with ~pyreflow.fcs_read_std_dataset and ~pyreflow.fcs_read_std_datasets.

classmethod scalpal()#

Return non-compliant configuration optimized to preserve data.

All non-trivial metadata (ie whitespace, blank keys, etc) will be preserved.

Return type:

dict[str, Any]

classmethod sledgehammer()#

Return non-compliant configuration optimized to read data.

Metadata may be destroyed or dropped.

Return type:

dict[str, Any]

classmethod strict()#

Return standards-compliant configuration.

Return type:

dict[str, Any]

class pyreflow.api.ReadFlatDatasetFromKeywordsConfig#

Config for ~pyreflow.fcs_read_flat_dataset_with_keywords.

classmethod scalpal()#

Return non-compliant configuration optimized to preserve data.

All non-trivial metadata (ie whitespace, blank keys, etc) will be preserved.

Return type:

dict[str, Any]

classmethod sledgehammer()#

Return non-compliant configuration optimized to read data.

Metadata may be destroyed or dropped.

Return type:

dict[str, Any]

classmethod strict()#

Return standards-compliant configuration.

Return type:

dict[str, Any]

class pyreflow.api.NewCoreTEXTConfig#

Config for CoreTEXT*.from_kws().

classmethod scalpal()#

Return non-compliant configuration optimized to preserve data.

All non-trivial metadata (ie whitespace, blank keys, etc) will be preserved.

Return type:

dict[str, Any]

classmethod sledgehammer()#

Return non-compliant configuration optimized to read data.

Metadata may be destroyed or dropped.

Return type:

dict[str, Any]

classmethod strict()#

Return standards-compliant configuration.

Return type:

dict[str, Any]

class pyreflow.api.NewCoreDatasetConfig#

Config for CoreDataset*.from_kws().

classmethod scalpal()#

Return non-compliant configuration optimized to preserve data.

All non-trivial metadata (ie whitespace, blank keys, etc) will be preserved.

Return type:

dict[str, Any]

classmethod sledgehammer()#

Return non-compliant configuration optimized to read data.

Metadata may be destroyed or dropped.

Return type:

dict[str, Any]

classmethod strict()#

Return standards-compliant configuration.

Return type:

dict[str, Any]

Pydantic#

pyreflow has an optional pydantic interface for configuration options. Configurations may be parsed from a yaml or JSON file and validated using these classes. Using this requires pydantic to be installed, which is not the case by default.

Methods on these classes are wrappers for functions defined in Reader Functions. For any method, the corresponding function in Reader Functions is fcs_<method_name>. See there for in-depth explanation for every argument, parameter, and exception.

This may be useful in large pipelines where one has many files with different configurations that one wishes to process in a type-safe manner.

Each class also has methods corresponding to the different read strategies as outlined in ref:overview. These can be used to initialize a class with the default configuration for a given strategy and then modified as needed.

class pyreflow.pydantic.PyreflowReadHeaderConfig(*, text_correction=(0, 0), data_correction=(0, 0), analysis_correction=(0, 0), other_corrections=[], max_other=None, other_width=8, guess_other_width='none', squish_offsets=False, allow_pseudoempty=False, dataset_overflow_limit=0, overlap_correction_limit=0)#
Parameters:
read_header(path, dataset_offset=0)#

Wrapper for fcs_read_header().

Parameters:
Return type:

Header

classmethod new_scalpal()#

Init to read non-compliant files without data loss.

Return type:

Self

classmethod new_sledgehammer()#

Init to read non-compliant files maybe with possible metadata loss.

Return type:

Self

class pyreflow.pydantic.PyreflowReadFlatTEXTConfig(*, text_correction=(0, 0), data_correction=(0, 0), analysis_correction=(0, 0), other_corrections=[], max_other=None, other_width=8, guess_other_width='none', squish_offsets=False, allow_pseudoempty=False, dataset_overflow_limit=0, overlap_correction_limit=0, version_override=None, supp_text_correction=(0, 0), nextdata_correction=0, allow_duplicated_supp_text='false', ignore_supp_text=False, delim_escape_mode='escaped', allow_non_ascii_delim='false', allow_nonunique='false', allow_even_delims='false', allow_odd_tokens='false', allow_empty_keys='false', allow_delim_at_boundary='false', use_encoding='utf8', allow_non_ascii_keys='false', allow_non_utf8_values='false', allow_missing_supp_text='false', allow_supp_text_own_delim='false', allow_missing_nextdata='false', trim_value_whitespace='notrim', warnings_are_errors=False, hide_warnings=False)#
Parameters:
read_flat_text(path, dataset_offset=0)#

Wrapper for fcs_read_flat_text().

Parameters:
Return type:

FlatTEXTOutput

read_flat_texts(path, skip=None, limit=None, scan=False)#

Wrapper for fcs_read_flat_texts().

Parameters:
Return type:

list[FlatTEXTOutput]

classmethod new_scalpal()#

Init to read non-compliant files without data loss.

Return type:

Self

classmethod new_sledgehammer()#

Init to read non-compliant files maybe with possible metadata loss.

Return type:

Self

to_header_config()#

Project this model to PyreflowReadHeaderConfig.

Return type:

PyreflowReadHeaderConfig

class pyreflow.pydantic.PyreflowReadStdTEXTConfig(*, text_correction=(0, 0), data_correction=(0, 0), analysis_correction=(0, 0), other_corrections=[], max_other=None, other_width=8, guess_other_width='none', squish_offsets=False, allow_pseudoempty=False, dataset_overflow_limit=0, overlap_correction_limit=0, version_override=None, supp_text_correction=(0, 0), nextdata_correction=0, allow_duplicated_supp_text='false', ignore_supp_text=False, delim_escape_mode='escaped', allow_non_ascii_delim='false', allow_nonunique='false', allow_even_delims='false', allow_odd_tokens='false', allow_empty_keys='false', allow_delim_at_boundary='false', use_encoding='utf8', allow_non_ascii_keys='false', allow_non_utf8_values='false', allow_missing_supp_text='false', allow_supp_text_own_delim='false', allow_missing_nextdata='false', trim_value_whitespace='notrim', dedup_measurement_names=False, trim_intra_value_whitespace=False, time_meas_pattern='^(TIME|Time)$', allow_missing_time='false', add_missing_timestep=None, force_linear_scale='none', ignore_optical_only_keys=[], process_optical_only_keys='demote_warn', date_pattern=None, time_pattern=None, datetime_pattern=None, last_modified_pattern=None, allow_other_feature=False, process_pseudostandard='error', process_hyper_par='error', process_other_version='error', process_extra_timestep='error', fix_log_scale_offsets=False, spillover_measurement_mode='named', disallow_localtime=False, ignore_standard_keys=[], promote_to_standard=[], demote_from_standard=[], rename_standard_keys={}, replace_standard_key_values={}, append_standard_keywords={}, substitute_standard_key_values={}, allow_repair_non_unique='false', text_data_correction=(0, 0), text_analysis_correction=(0, 0), ignore_text_data_offsets=False, ignore_text_analysis_offsets=False, allow_header_text_offset_mismatch='error', allow_missing_required_offsets='false', process_optional_failure='error', int_width_override='never', byteord_override='none', disallow_range_truncation='false', warnings_are_errors=False, hide_warnings=False)#
Parameters:
read_std_text(path, dataset_offset=0)#

Wrapper for fcs_read_std_text().

Parameters:
Return type:

tuple[AnyCoreTEXT, StdTEXTOutput]

read_std_texts(path, skip=None, limit=None, scan=False)#

Wrapper for fcs_read_std_texts().

Parameters:
Return type:

list[tuple[AnyCoreTEXT, StdTEXTOutput]]

classmethod new_scalpal()#

Init to read non-compliant files without data loss.

Return type:

Self

classmethod new_sledgehammer()#

Init to read non-compliant files maybe with possible metadata loss.

Return type:

Self

to_flat_text_config()#

Project this model to PyreflowReadFlatTEXTConfig.

Return type:

PyreflowReadFlatTEXTConfig

to_header_config()#

Project this model to PyreflowReadHeaderConfig.

Return type:

PyreflowReadHeaderConfig

class pyreflow.pydantic.PyreflowReadFlatDatasetConfig(*, text_correction=(0, 0), data_correction=(0, 0), analysis_correction=(0, 0), other_corrections=[], max_other=None, other_width=8, guess_other_width='none', squish_offsets=False, allow_pseudoempty=False, dataset_overflow_limit=0, overlap_correction_limit=0, version_override=None, supp_text_correction=(0, 0), nextdata_correction=0, allow_duplicated_supp_text='false', ignore_supp_text=False, delim_escape_mode='escaped', allow_non_ascii_delim='false', allow_nonunique='false', allow_even_delims='false', allow_odd_tokens='false', allow_empty_keys='false', allow_delim_at_boundary='false', use_encoding='utf8', allow_non_ascii_keys='false', allow_non_utf8_values='false', allow_missing_supp_text='false', allow_supp_text_own_delim='false', allow_missing_nextdata='false', trim_value_whitespace='notrim', ignore_standard_keys=[], promote_to_standard=[], demote_from_standard=[], rename_standard_keys={}, replace_standard_key_values={}, append_standard_keywords={}, substitute_standard_key_values={}, allow_repair_non_unique='false', text_data_correction=(0, 0), text_analysis_correction=(0, 0), ignore_text_data_offsets=False, ignore_text_analysis_offsets=False, allow_header_text_offset_mismatch='error', allow_missing_required_offsets='false', process_optional_failure='error', int_width_override='never', byteord_override='none', disallow_range_truncation='false', data_remainder_limit=0, allow_uneven_event_width='false', allow_tot_mismatch='false', over_bitmask_action='trunc_warn', over_range_action='warn', allow_missing_crc='false', allow_mismatch_crc='false', compute_crc='never', read_intra_segment_dark_bytes=False, read_post_dataset_dark_bytes=False, row_buffer_size=28000, warnings_are_errors=False, hide_warnings=False)#
Parameters:
read_flat_dataset(path, dataset_offset=0, scan=False)#

Wrapper for fcs_read_flat_dataset().

Parameters:
Return type:

FlatDatasetOutput

read_flat_datasets(path, skip=None, limit=None, scan=False)#

Wrapper for fcs_read_flat_datasets().

Parameters:
Return type:

list[FlatDatasetOutput]

summarize(path, skip=None, limit=None, scan=False)#

Wrapper for fcs_summarize().

Parameters:
Return type:

list[DatasetSummary]

classmethod new_scalpal()#

Init to read non-compliant files without data loss.

Return type:

Self

classmethod new_sledgehammer()#

Init to read non-compliant files maybe with possible metadata loss.

Return type:

Self

to_flat_text_config()#

Project this model to PyreflowReadFlatTEXTConfig.

Return type:

PyreflowReadFlatTEXTConfig

to_header_config()#

Project this model to PyreflowReadHeaderConfig.

Return type:

PyreflowReadHeaderConfig

class pyreflow.pydantic.PyreflowReadStdDatasetConfig(*, text_correction=(0, 0), data_correction=(0, 0), analysis_correction=(0, 0), other_corrections=[], max_other=None, other_width=8, guess_other_width='none', squish_offsets=False, allow_pseudoempty=False, dataset_overflow_limit=0, overlap_correction_limit=0, version_override=None, supp_text_correction=(0, 0), nextdata_correction=0, allow_duplicated_supp_text='false', ignore_supp_text=False, delim_escape_mode='escaped', allow_non_ascii_delim='false', allow_nonunique='false', allow_even_delims='false', allow_odd_tokens='false', allow_empty_keys='false', allow_delim_at_boundary='false', use_encoding='utf8', allow_non_ascii_keys='false', allow_non_utf8_values='false', allow_missing_supp_text='false', allow_supp_text_own_delim='false', allow_missing_nextdata='false', trim_value_whitespace='notrim', dedup_measurement_names=False, trim_intra_value_whitespace=False, time_meas_pattern='^(TIME|Time)$', allow_missing_time='false', add_missing_timestep=None, force_linear_scale='none', ignore_optical_only_keys=[], process_optical_only_keys='demote_warn', date_pattern=None, time_pattern=None, datetime_pattern=None, last_modified_pattern=None, allow_other_feature=False, process_pseudostandard='error', process_hyper_par='error', process_other_version='error', process_extra_timestep='error', fix_log_scale_offsets=False, spillover_measurement_mode='named', disallow_localtime=False, ignore_standard_keys=[], promote_to_standard=[], demote_from_standard=[], rename_standard_keys={}, replace_standard_key_values={}, append_standard_keywords={}, substitute_standard_key_values={}, allow_repair_non_unique='false', text_data_correction=(0, 0), text_analysis_correction=(0, 0), ignore_text_data_offsets=False, ignore_text_analysis_offsets=False, allow_header_text_offset_mismatch='error', allow_missing_required_offsets='false', process_optional_failure='error', int_width_override='never', byteord_override='none', disallow_range_truncation='false', data_remainder_limit=0, allow_uneven_event_width='false', allow_tot_mismatch='false', over_bitmask_action='trunc_warn', over_range_action='warn', allow_missing_crc='false', allow_mismatch_crc='false', compute_crc='never', read_intra_segment_dark_bytes=False, read_post_dataset_dark_bytes=False, row_buffer_size=28000, warnings_are_errors=False, hide_warnings=False)#
Parameters:
read_std_dataset(path, dataset_offset=0, scan=False)#

Wrapper for fcs_read_std_dataset().

Parameters:
Return type:

tuple[AnyCoreDataset, StdDatasetOutput]

read_std_datasets(path, skip=None, limit=None, scan=False)#

Wrapper for fcs_read_std_datasets().

Parameters:
Return type:

list[tuple[AnyCoreDataset, StdDatasetOutput]]

classmethod new_scalpal()#

Init to read non-compliant files without data loss.

Return type:

Self

classmethod new_sledgehammer()#

Init to read non-compliant files maybe with possible metadata loss.

Return type:

Self

to_flat_dataset_config()#

Project this model to PyreflowReadFlatDatasetConfig.

Return type:

PyreflowReadFlatDatasetConfig

to_flat_text_config()#

Project this model to PyreflowReadFlatTEXTConfig.

Return type:

PyreflowReadFlatTEXTConfig

to_header_config()#

Project this model to PyreflowReadHeaderConfig.

Return type:

PyreflowReadHeaderConfig

to_std_text_config()#

Project this model to PyreflowReadStdTEXTConfig.

Return type:

PyreflowReadStdTEXTConfig

class pyreflow.pydantic.PyreflowReadFlatDatasetFromKeywordsConfig(*, allow_pseudoempty=False, dataset_overflow_limit=0, overlap_correction_limit=0, ignore_standard_keys=[], promote_to_standard=[], demote_from_standard=[], rename_standard_keys={}, replace_standard_key_values={}, append_standard_keywords={}, substitute_standard_key_values={}, allow_repair_non_unique='false', text_data_correction=(0, 0), text_analysis_correction=(0, 0), ignore_text_data_offsets=False, ignore_text_analysis_offsets=False, allow_header_text_offset_mismatch='error', allow_missing_required_offsets='false', process_optional_failure='error', int_width_override='never', byteord_override='none', disallow_range_truncation='false', data_remainder_limit=0, allow_uneven_event_width='false', allow_tot_mismatch='false', over_bitmask_action='trunc_warn', over_range_action='warn', allow_missing_crc='false', allow_mismatch_crc='false', compute_crc='never', read_intra_segment_dark_bytes=False, read_post_dataset_dark_bytes=False, row_buffer_size=28000, warnings_are_errors=False, hide_warnings=False)#
Parameters:
read_flat_dataset_with_keywords(path, dataset_offset=0, dataset_len=None)#

Wrapper for fcs_read_flat_dataset_with_keywords().

Parameters:

  • path (Path)

  • dataset_offset (int)

  • dataset_len (int | None)

Return type:

FlatDatasetFromKwsOutput

classmethod new_scalpal()#

Init to read non-compliant files without data loss.

Return type:

Self

classmethod new_sledgehammer()#

Init to read non-compliant files maybe with possible metadata loss.

Return type:

Self

On this page