rmellipse.workflows =================== .. py:module:: rmellipse.workflows .. autoapi-nested-parse:: Workflows module, utilized by the command line interface. The workflows module exists to suppor the command line interface for running and publishing workflows. Documentation is provided for transparency and to aid development, however the python API is not intended for use. Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/rmellipse/workflows/archive_interface/index /autoapi/rmellipse/workflows/extras/index /autoapi/rmellipse/workflows/local_interfaces/index /autoapi/rmellipse/workflows/workflowtree/index Attributes ---------- .. autoapisummary:: rmellipse.workflows.SCHEMAS Exceptions ---------- .. autoapisummary:: rmellipse.workflows.UninitializedRMEProject rmellipse.workflows.JobFailure Classes ------- .. autoapisummary:: rmellipse.workflows.ProjectSettings rmellipse.workflows.FILE_TYPES rmellipse.workflows.WORKFLOW_SPECIAL_KEYS rmellipse.workflows.DIRECTORY_ITEMS rmellipse.workflows.WorkflowConfig rmellipse.workflows.RMESettings Package Contents ---------------- .. py:exception:: UninitializedRMEProject(**kwargs) Bases: :py:obj:`Exception` Common base class for all non-exit exceptions. Initialize self. See help(type(self)) for accurate signature. .. py:exception:: JobFailure(**kwargs) Bases: :py:obj:`Exception` Common base class for all non-exit exceptions. Initialize self. See help(type(self)) for accurate signature. .. py:class:: ProjectSettings(project: pathlib.Path | str | dict) Bases: :py:obj:`dict` Store and access project settings. Can be serialized to a JSON like file. Initialize self. See help(type(self)) for accurate signature. .. py:property:: requires_files :type: dict Get a list of required files :returns: dictionary of required files or folder patterns, keys are the new name of the datset within environment. :rtype: dict[str] .. py:property:: requires_releases Releases required by a project. :returns: list of version expression requirements, empty if no requirements. :rtype: list .. py:property:: requires_env Releases required by a project. :returns: list of version expression requirements, empty if no requirements. :rtype: list .. py:property:: rmesettings :type: RMESettings Get RMESettings for this project. :returns: _description_ :rtype: dict .. py:property:: project_dir :type: pathlib.Path Root directory of the project. :returns: Path to the projects root directory. :rtype: Path .. py:property:: active_workflow :type: pathlib.Path Return information about the active workflow. :returns: Path to the active workflow file. :rtype: Path .. py:property:: project_map Returns a project map. :returns: Path to the project map. :rtype: Path .. py:property:: data_env_dir Directory containing the dataest environment. .. py:property:: data_env_packages Directory containing the full linked packages in data_env_dir. .. py:property:: processlogdir :type: pathlib.Path Direcotry that stores process logs. .. py:property:: dotrmedir Directory for storing hidden files within a project. :rtype: Path .. py:property:: wft_jsondir Directory for storing workfow solution JSON files. :returns: _description_ :rtype: _type_ .. py:property:: rmeproj_file Path to the project file within a directory. .. py:method:: is_rmeproject(directory: pathlib.Path) :staticmethod: True if directory is an RME project (i.e. has a rmeproject.yml file.) .. py:data:: SCHEMAS .. py:class:: FILE_TYPES(*args, **kwds) Bases: :py:obj:`enum.Enum` Create a collection of name/value pairs. Example enumeration: >>> class Color(Enum): ... RED = 1 ... BLUE = 2 ... GREEN = 3 Access them by: - attribute access: >>> Color.RED - value lookup: >>> Color(1) - name lookup: >>> Color['RED'] Enumerations can be iterated over, and know how many members they have: >>> len(Color) 3 >>> list(Color) [, , ] Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. .. py:attribute:: WORKFLOW :value: 'workflow' .. py:attribute:: CODE :value: 'code' .. py:attribute:: DOC :value: 'doc' .. py:attribute:: MISC :value: 'misc' .. py:attribute:: DATASET :value: 'dataset' .. py:class:: WORKFLOW_SPECIAL_KEYS(*args, **kwds) Bases: :py:obj:`enum.Enum` Create a collection of name/value pairs. Example enumeration: >>> class Color(Enum): ... RED = 1 ... BLUE = 2 ... GREEN = 3 Access them by: - attribute access: >>> Color.RED - value lookup: >>> Color(1) - name lookup: >>> Color['RED'] Enumerations can be iterated over, and know how many members they have: >>> len(Color) 3 >>> list(Color) [, , ] Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. .. py:attribute:: JOB :value: 'jobs' .. py:attribute:: RELEASE :value: 'release' .. py:attribute:: DATASETS :value: 'datasets' .. py:class:: DIRECTORY_ITEMS(*args, **kwds) Bases: :py:obj:`enum.Enum` Create a collection of name/value pairs. Example enumeration: >>> class Color(Enum): ... RED = 1 ... BLUE = 2 ... GREEN = 3 Access them by: - attribute access: >>> Color.RED - value lookup: >>> Color(1) - name lookup: >>> Color['RED'] Enumerations can be iterated over, and know how many members they have: >>> len(Color) 3 >>> list(Color) [, , ] Methods can be added to enumerations, and members can have their own attributes -- see the documentation for details. .. py:attribute:: DIRECTORY :value: 'Directory' .. py:attribute:: FILE :value: 'File' .. py:attribute:: H5_FILE :value: 'H5_File' .. py:attribute:: H5_GROUP :value: 'H5_Group' .. py:attribute:: H5_DATASET :value: 'H5_DATASET' .. py:class:: WorkflowConfig(workflow_file: pathlib.Path | str | dict, project_settings: ProjectSettings) Bases: :py:obj:`dict` Store and access a workflow configuration. :param dict: _description_ :type dict: _type_ :returns: _description_ :rtype: _type_ :raises Exception: _description_ :raises :param workflow_file: Workflow file.: :raises :type workflow_file: Path | str | dict: :raises :param project_settings: Project settings that contain the: workflow. :raises :type project_settings: ProjectSettings: :raises :raises ValueError: Missing environment variables.: :raises :raises ValueError: Empry workflow file.: .. py:attribute:: project_settings .. py:attribute:: raw .. py:attribute:: env_variables .. py:method:: expand_variables(obj: str | dict | list, expand_env: bool = True, expand_datasets: bool = True) -> str | dict | list Expand variables of any object that can be serialized to JSON. Encodes obj to a JSON string, replaces any variable expressions, then de-encodes the stirng. Order of operations is encode -> env variables -> dataset names -> decode. JSON is a faster serialization format, so using it here instead of YAML. :param obj: _description_ :type obj: str | dict | list :param expand_env: If True, expands environment variables. Default is true. :type expand_env: bool, optional :param expand_datasets: If True, expands dataset variables. Default is True :type expand_datasets: bool, optional :returns: Same type as input with variables expanded. :rtype: object .. py:property:: title :type: str .. py:property:: release_version :type: str .. py:property:: requires_files :type: dict Get a list of required files Includes required files of the project, any name-space conflicts are overwritten with the workspace requires_files. :returns: dictionary of required files or folder patterns, keys are the new name of the datset within environment. :rtype: dict[str] .. py:property:: requires_releases :type: None Releases requirements defined in the workflow file. :returns: _description_ :rtype: _type_ .. py:property:: requires_env :type: None Environment variable requirements. :returns: _description_ :rtype: _type_ .. py:property:: formatting_keys :type: dict Dictionary of kwargs for string formating. .. py:property:: datasets List of defined datsets in the workflow config. :returns: _description_ :rtype: _type_ .. py:property:: jobs List of defined datsets in the workflow config. :returns: _description_ :rtype: _type_ .. py:property:: release Release related settings :returns: _description_ :rtype: _type_ .. py:class:: RMESettings(project_dir: pathlib.Path) Bases: :py:obj:`dict` Settings file for the RME program. rme.yml files are used for storing settings that can be defined at the user level. For example, what archives to use by default, and aliases for the archives. Settings files are overloaded by higher priority settings files. Discovery order (lowest to highest priority) User config -> project config :param dict: _description_ :type dict: _type_ :param Initialize self. See help(type(self)) for accurate signature.: .. py:method:: get_host_settings(host_name: str = None) Get the host settings by host name. User and password are returned as None if they aren't present in the settings. Does not attempt to get credentials based on host. :param host_name: _description_, by default None :type host_name: str, optional :returns: * *host* -- str * *user* -- str * *password* -- str :raises LookupError:: If the host isn't in the settings. .. py:property:: hosts .. py:property:: default_host