rmellipse.workflows =================== .. py:module:: rmellipse.workflows .. autoapi-nested-parse:: Workflows module, utilized by the command line interface. The workflows module exists to suppor the command line interface for running and publishing workflows. Documentation is provided for transparency and to aid development, however the python API is not intended for use. .. !! processed by numpydoc !! Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/rmellipse/workflows/archive_interface/index /autoapi/rmellipse/workflows/extras/index /autoapi/rmellipse/workflows/local_interfaces/index /autoapi/rmellipse/workflows/workflowtree/index Attributes ---------- .. autoapisummary:: rmellipse.workflows.SCHEMAS Exceptions ---------- .. autoapisummary:: rmellipse.workflows.UninitializedRMEProject rmellipse.workflows.JobFailure Classes ------- .. autoapisummary:: rmellipse.workflows.ProjectSettings rmellipse.workflows.FILE_TYPES rmellipse.workflows.WORKFLOW_SPECIAL_KEYS rmellipse.workflows.DIRECTORY_ITEMS rmellipse.workflows.WorkflowConfig rmellipse.workflows.RMESettings Package Contents ---------------- .. py:exception:: UninitializedRMEProject(**kwargs) Bases: :py:obj:`Exception` Common base class for all non-exit exceptions. Initialize self. See help(type(self)) for accurate signature. .. !! processed by numpydoc !! .. py:exception:: JobFailure(**kwargs) Bases: :py:obj:`Exception` Common base class for all non-exit exceptions. Initialize self. See help(type(self)) for accurate signature. .. !! processed by numpydoc !! .. py:class:: ProjectSettings(project: pathlib.Path | str | dict) Bases: :py:obj:`dict` Store and access project settings. Can be serialized to a JSON like file. Initialize self. See help(type(self)) for accurate signature. .. !! processed by numpydoc !! .. py:property:: requires_files :type: dict Get a list of required files :Returns: dict[str] dictionary of required files or folder patterns, keys are the new name of the datset within environment. .. !! processed by numpydoc !! .. py:property:: requires_releases Releases required by a project. :Returns: list list of version expression requirements, empty if no requirements. .. !! processed by numpydoc !! .. py:property:: requires_env Releases required by a project. :Returns: list list of version expression requirements, empty if no requirements. .. !! processed by numpydoc !! .. py:property:: rmesettings :type: RMESettings Get RMESettings for this project. :Returns: dict _description_ .. !! processed by numpydoc !! .. py:property:: project_dir :type: pathlib.Path Root directory of the project. :Returns: Path Path to the projects root directory. .. !! processed by numpydoc !! .. py:property:: active_workflow :type: pathlib.Path Return information about the active workflow. :Returns: Path: Path to the active workflow file. .. !! processed by numpydoc !! .. py:property:: project_map Returns a project map. :Returns: Path Path to the project map. .. !! processed by numpydoc !! .. py:property:: data_env_dir Directory containing the dataest environment. .. !! processed by numpydoc !! .. py:property:: data_env_packages Directory containing the full linked packages in data_env_dir. .. !! processed by numpydoc !! .. py:property:: processlogdir :type: pathlib.Path Direcotry that stores process logs. .. !! processed by numpydoc !! .. py:property:: dotrmedir Directory for storing hidden files within a project. :Returns: Path .. .. !! processed by numpydoc !! .. py:property:: wft_jsondir Directory for storing workfow solution JSON files. :Returns: _type_ _description_ .. !! processed by numpydoc !! .. py:property:: rmeproj_file Path to the project file within a directory. .. !! processed by numpydoc !! .. py:method:: is_rmeproject(directory: pathlib.Path) :staticmethod: True if directory is an RME project (i.e. has a rmeproject.yml file.) .. !! processed by numpydoc !! .. py:data:: SCHEMAS .. py:class:: FILE_TYPES(*args, **kwds) Bases: :py:obj:`enum.Enum` Create a collection of name/value pairs. Example enumeration: >>> class Color(Enum): ... RED = 1 ... BLUE = 2 ... GREEN = 3 Access them by: - attribute access: >>> Color.RED - value lookup: >>> Color(1) - name lookup: >>> Color['RED'] Enumerations can be iterated over, and know how many members they have: >>> len(Color) 3 >>> list(Color) [, , ] Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. .. !! processed by numpydoc !! .. py:attribute:: WORKFLOW :value: 'workflow' .. py:attribute:: CODE :value: 'code' .. py:attribute:: DOC :value: 'doc' .. py:attribute:: MISC :value: 'misc' .. py:attribute:: DATASET :value: 'dataset' .. py:class:: WORKFLOW_SPECIAL_KEYS(*args, **kwds) Bases: :py:obj:`enum.Enum` Create a collection of name/value pairs. Example enumeration: >>> class Color(Enum): ... RED = 1 ... BLUE = 2 ... GREEN = 3 Access them by: - attribute access: >>> Color.RED - value lookup: >>> Color(1) - name lookup: >>> Color['RED'] Enumerations can be iterated over, and know how many members they have: >>> len(Color) 3 >>> list(Color) [, , ] Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. .. !! processed by numpydoc !! .. py:attribute:: JOB :value: 'jobs' .. py:attribute:: RELEASE :value: 'release' .. py:attribute:: DATASETS :value: 'datasets' .. py:class:: DIRECTORY_ITEMS(*args, **kwds) Bases: :py:obj:`enum.Enum` Create a collection of name/value pairs. Example enumeration: >>> class Color(Enum): ... RED = 1 ... BLUE = 2 ... GREEN = 3 Access them by: - attribute access: >>> Color.RED - value lookup: >>> Color(1) - name lookup: >>> Color['RED'] Enumerations can be iterated over, and know how many members they have: >>> len(Color) 3 >>> list(Color) [, , ] Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. .. !! processed by numpydoc !! .. py:attribute:: DIRECTORY :value: 'Directory' .. py:attribute:: FILE :value: 'File' .. py:attribute:: H5_FILE :value: 'H5_File' .. py:attribute:: H5_GROUP :value: 'H5_Group' .. py:attribute:: H5_DATASET :value: 'H5_DATASET' .. py:class:: WorkflowConfig(workflow_file: pathlib.Path | str | dict, project_settings: ProjectSettings) Bases: :py:obj:`dict` Store and access a workflow configuration. Parameters ---------- dict : _type_ _description_ Returns ------- _type_ _description_ Raises ------ Exception _description_ :Parameters: **workflow_file** : Path | str | dict Workflow file. **project_settings** : ProjectSettings Project settings that contain the workflow. :Raises: ValueError Missing environment variables. ValueError Empry workflow file. .. !! processed by numpydoc !! .. py:attribute:: project_settings .. py:attribute:: raw .. py:attribute:: env_variables .. py:method:: expand_variables(obj: str | dict | list, expand_env: bool = True, expand_datasets: bool = True) -> str | dict | list Expand variables of any object that can be serialized to JSON. Encodes obj to a JSON string, replaces any variable expressions, then de-encodes the stirng. Order of operations is encode -> env variables -> dataset names -> decode. JSON is a faster serialization format, so using it here instead of YAML. :Parameters: **obj** : str | dict | list _description_ **expand_env: bool, optional** If True, expands environment variables. Default is true. **expand_datasets: bool, optional** If True, expands dataset variables. Default is True :Returns: object Same type as input with variables expanded. .. !! processed by numpydoc !! .. py:property:: title :type: str .. py:property:: release_version :type: str .. py:property:: requires_files :type: dict Get a list of required files Includes required files of the project, any name-space conflicts are overwritten with the workspace requires_files. :Returns: dict[str] dictionary of required files or folder patterns, keys are the new name of the datset within environment. .. !! processed by numpydoc !! .. py:property:: requires_releases :type: None Releases requirements defined in the workflow file. :Returns: _type_ _description_ .. !! processed by numpydoc !! .. py:property:: requires_env :type: None Environment variable requirements. :Returns: _type_ _description_ .. !! processed by numpydoc !! .. py:property:: formatting_keys :type: dict Dictionary of kwargs for string formating. .. !! processed by numpydoc !! .. py:property:: datasets List of defined datsets in the workflow config. :Returns: _type_ _description_ .. !! processed by numpydoc !! .. py:property:: jobs List of defined datsets in the workflow config. :Returns: _type_ _description_ .. !! processed by numpydoc !! .. py:property:: release Release related settings :Returns: _type_ _description_ .. !! processed by numpydoc !! .. py:class:: RMESettings(project_dir: pathlib.Path) Bases: :py:obj:`dict` Settings file for the RME program. rme.yml files are used for storing settings that can be defined at the user level. For example, what archives to use by default, and aliases for the archives. Settings files are overloaded by higher priority settings files. Discovery order (lowest to highest priority) User config -> project config Parameters ---------- dict : _type_ _description_ Initialize self. See help(type(self)) for accurate signature. .. !! processed by numpydoc !! .. py:method:: get_host_settings(host_name: str = None) Get the host settings by host name. User and password are returned as None if they aren't present in the settings. Does not attempt to get credentials based on host. :Parameters: **host_name** : str, optional _description_, by default None :Returns: host: str user: str password: str :Raises: LookupError: If the host isn't in the settings. .. !! processed by numpydoc !! .. py:property:: hosts .. py:property:: default_host