Changelog

v1.4.1 (2023-09-20)

Bug fixes

  • Resolved issue where text files that can’t be opened with any encoding caused the record builder to crash (#134)

Documentation improvements

  • Documented internal release and deploy process on Wiki

v1.4.0 (2023-09-19)

New features

  • Added ability to generate previews for “plain” image files (e.g. .jpg, .png, etc.) and plain text files. (#128)

Bug fixes

  • Fix problem arising from NEMO API change that removed username keyword. (#131)

v1.3.1 (2023-05-19)

Bug fixes

  • Fixed issue where “process new records” script was emailing an error alert on conditions that were not errors. (#126)

Miscellaneous/Development changes

  • Fixed pipeline runner to not run tests when they’re not needed. (#129)

v1.3.0 (2023-04-14)

New features

  • Add support for reading .spc and .msa EDS spectrum files produced by EDAX acquisition softwares. (#122)

Documentation improvements

  • Add towncrier to manage documentation of changes in a semi-automated manner. (#125)

v1.2.0 (2023-03-31)

New features

  • Added new “default” extractor for filetypes we don’t know how to read that will add very basic file-based metadata otherwise (#121)

  • Added a configuration environment variable for file finding (NexusLIMS_file_strategy). A value of "inclusive" will add all files found in the time range of a session to the record (even if we don’t know how to parse it beyond basic metadata). A value of "exclusive" will exlcude files that do not have an explicit extractor defined (this was the previous behavior) (#121)

  • Added a way to “ignore” files during the file finding routine via an environment variable named NexusLIMS_ignore_patterns. It should be a JSON-formatted list provided as a string. Each item of the list will be passed to the GNU find command as a pattern to ignore. (#121)

Bug fixes

  • Fixed Poetry not installing due to change in upstream installer location (#117)

  • Fixed issue where record builder would not run (and we wouldn’t even be alerted!) if the network shares for mmfnexus_path and nexusLIMS_path were not mounted. (#118)

  • Fixed bug introduced by change to API response for reservation questions in NEMO 4.3.2 (#119)

  • Fix for development bug introduced by upgrade of tox package to 4.0.

Enhancements

  • Added support for "NO_CONSENT" and "NO_RESERVATION" statuses in the session_log table of the NexusLIMS database (#105)

  • Harvesters (and other parts of the code that use network resources) will now retry their requests if they fail in order to make the record building process more resilient (#112)

  • Harvester will now read periodic table element information from NEMO reservation questions and include them in the XML records. Also updated the schema and CDCS XSLT to allow for and display this information in the front end. (#120)

  • File finding now works on a directory of symbolic links (in addition to a regular folder hierarchy).

Documentation improvements

Miscellaneous/Development changes

  • If the record building delay has not passed and no files were found, a RECORD_GENERATION event will no longer be added to the session_log table in the database to avoid cluttering things up. (#107)

  • Public facing branches are now excluded from CI/CD pipeline to prevent test failures (#114)

  • Updated code to use various linters, including isort, black, pylint, and ruff. (#121)

  • Add support for Python 3.10(.9)

  • Moved URL configration to environment variables

  • Updated third-party dependencies to recent latest versions

Deprecations and/or Removals

  • Remove support for Python 3.7.X

  • Removed unused LDAP code

v1.1.1 (2022-06-15)

Bug fixes

  • Fixed issue where record builder would crash if only one file was found during the activity (and added explicit test for this condition). (#96)

  • Fix issue in NEMO harvester where not-yet-ended sessions would cause the harvester to try to insert rows that violated database constraints. (#99)

  • Implemented a “lockfile” system so concurrent runs of the record builder will not be allowed, preventing extra entries in the session_log table that were causing errors. (#101)

  • Fix reading reservation and usage event times from NEMO servers with differing datetime formats. (#103)

  • The NEMO harvester no longer attempts to build records without explicit “data consent” supplied by the user during the reservation questions (previously, if no reservation was found, the harvester would return a generic event and a record would still be built). (#104)

  • Fixed bug where null bytes in a TIFF file caused an error in metadata extraction (#110)

Enhancements

  • Add ability for record builder to insert a link to reservation information in the summary node (modified schema to hold this and record builder to insert it). (#90)

  • Contributed a PR to the upstream NEMO project to allow for displaying of a single reservation, so that we may link to it and include it as a reference in records built by NexusLIMS. (#92)

  • Made the default data_consent value for the NEMO harvester False, so we will not harvest data from sessions that do not have reservation questions defined (users now have to opt-in to have their data curated by NexusLIMS). (#93)

  • NEMO harvester now limits its API requests to only tools defined in the NexusLIMS database, which is more efficient and greatly speeds up the harvesting process. (#94)

  • The record builder will now retry for a configurable number of days if it does not find any files for a session (useful for machines that have a delay in data syncing to centralized file storage). Configured via the nexusLIMS_file_delay_days environment variable. (#102)

  • Made datetime formats for NEMO API harvester configurable (both sending and receiving) so that it can work regardless of configuration on the NEMO server. (#103)

  • Record generation events in the database now have timezone information for better specificity in multi-timezone setups. (#106)

  • Add pid attribute to Experiment schema to allow for integration with CDCS’s handle implementation. (#109)

Miscellaneous/Development changes

  • Configured tests to run on-premises, which speeds up various testing operations. (#57)

  • Drastically restructured repository to look more like a proper Python library than just a collection of files and scripts. (#60)

  • Migrated project organization and packaging from pipenv to poetry. (#88)

  • Fixed some tests that started failing due to tool ID changes on our local NEMO server. (#97)

  • Improved logging from NEMO harvester making it easier to debug issues when they occur. (#98)

  • Session processing script is now smarter about email alerts. (#100)

  • CI/CD pipeline will now retry failed tests (should be more resilient against transient failures due to network issues).

  • Made some changes to the codebase in preparation of making it public-facing on Github.

Deprecations and/or Removals

  • Removed a variety of associated files that were not important for the Python package (old presentations, diagrams, reports, etc.) (#60)

  • nexusLIMS.harvesters.sharepoint_calendar module was deprecated after the SharePoint calendaring system was decommissioned in the Nexus facility. All harvester development will center around NEMO for the foreseeable future. (#108)

  • Removed enumeration restriction on PIDs from the schema so it is more general (and easier to add new instruments without having to do an XML schema migration). (#110)

v1.1.0 (2021-12-12)

New features

  • Major new feature in this release is the implementation of a reservation and metadata harvester for the NEMO facility management system. All planned future feature development will focus on this harvester, and the SharePoint calendar harvester will be deprecated in a future release. See the Record building workflow docs and the nexusLIMS.harvesters.nemo package docs for more details. (#89)

Enhancements

  • Add support to NEMO harvester for multiple samples in a set of reservation questions. The required structure for reservation questions is documented in the nexusLIMS.harvesters.nemo.res_event_from_session() function.

  • Added ability to specify timezone information for instruments in the NexusLIMS database, which helps fully qualify all dates and times so file finding works as expected when inspecting files stored on servers in different timezones.

  • Updated detail XLST to display multiple samples for a record if present (since this is now possible using the NEMO reservation questions).

Documentation improvements

  • Documented new NEMO harvester and updated record generation documentation to describe how the process works with multiple harvesters.

  • Fixed broken image paths in README.

Miscellaneous/Development changes

  • Migrated project structure from pipenv to poetry for better dependency resolution, easier and faster deployment, and configuration of project via pyproject.toml. Also implemented tox for the running of tests, doc builds, and pipelines. (#88)

  • Refactored some functions from the SharePoint harvester into the nexusLIMS.utils module for easier use throughout the rest of the codebase.

Deprecations and/or Removals

  • Removed the “Session Logger” application in favor of using NEMO and its usage events to track session timestamps.

v1.0.1 (2021-09-15)

New features

  • Implemented a “file viewer” on the front-end NexusLIMS application which also allows for downloading single, multiple, or all data files from a particular record in .zip archives. (#61)

  • Implemented a metadata extractor for .ser and .emi files produced by the TIA application on FEI TEMs. (#62)

  • Added ability to export a record as XML in the front end NexusLIMS application. (#65)

  • Added a “tutorial” feature to the front-end of the NexusLIMS application, which leads users through a tour describing what the various parts of the application do. (#71)

  • Added new “dry run” mode and additional verbosity options to record builder that allow one to see what records would be built without actually doing anything. (#77)

Bug fixes

  • Fixed issue where Session Logger app was failing due to incompatibilities between the code and certain database states. (#53)

  • Fixed issue where Session Logger app was leaving behind a temporary file on the microscope computers by making it clean up after itself. (#55)

  • Fixed issue where multiple copies of the Session Logger app were able to be run at the same time, which shouldn’t have been possible. (#59)

  • Fixed the “back to previous” button in the front-end application that was broken. (#64)

  • Fixed issue with SharePoint harvester where records were being assigned to the person who created a calendar event, not the person whose name was on the actual event. (#72)

  • Fixed a deployment issue related to pipenv and how it specifies packages to be installed. (#73)

  • Fixed issues with .ser file handling (and contributed various fixes upstream to the HyperSpy project: 1, 2, 3). (#74)

Enhancements

  • Added customized loading text while the list of records is loading in the front-end NexusLIMS application. (#58)

  • Tweaked heuristic in SharePoint harvester to better match sessions to calendar events (previously, if there were multiple reservations in one day, they may have been incorrectly attributed to a session). (#67)

  • Added explicit support for Python 3.8.X versions. (#75)

  • Implemented bash script to run record builder automatically, which can then be scheduled via a tool such as cron. (#76)

  • Added version information to Session Logger app to make it easier for users to know if they are up to date or not. (#79)

  • Small tweak to make acquisition activity links easier to click in record display. (#81)

Documentation improvements

  • Added “taxonomy” of terms used in the NexusLIMS project to the documentation (see NexusLIMS taxonomy for details). (#40)

  • Added XML Schema documentation for the Nexus Experiment schema to the documentation (see Schema documentation for details). (#51)

  • Added links to NexusLIMS documentation in the front-end NexusLIMS CDCS application. (#68)

  • Added many documentation pages to more thoroughly explain how NexusLIMS works, including improvements to the project README, as well as the following pages: NexusLIMS database, Session logging (deprecated), Developer documentation, and pages about data security.

Miscellaneous/Development changes

  • Improvement to logging to make it easier to debug records not being built correctly. (#80)

  • Added new ReservationEvent class to abstract the concept of a calendar reservation event. This reduces dependencies on the SharePoint-specific way things were written before and will help in the future implementation of the NEMO harvester.

  • Fix some issues with tests not running correctly due to changes of paths in mmfnexus_path.

  • Improvements to the CI/CD pipelines so multiple pipelines can run at once without error.

v0.0.9 (2020-02-24) - First real working release

New features

  • Added extractor for TIFF image files produced by FEI SEM and FIB instruments.

  • Record builder can now be run automatically and will do the whole process (probing database, finding files, extracting metadata, building record XML, and uploading to CDCS frontend).

Enhancements

  • Acquisition activities are now split up by clustering of file acquisition times, rather than inspecting when an instrument switches modes. This is more realistic to how microscopes are used in practice (see Activity Clustering for more details).

  • Added “instrument specific” parsing for DigitalMicrograph files

  • Added nexusLIMS.cdcs module to handle interactions with the CDCS front-end.

  • Added a “Data Type” to all metadata extractions that will attempt to classify what sort of data a file is (“STEM EELS”, “TEM Image”, “SEM EDS”, etc.).

  • Configuration of program options is now mostly done via environment variables rather than values hard-coded into the source code.

  • Drastically improved file finding time by utilizing GNU find to identify files in a record rather than a pure Python implementation.

  • Records now use a “dummy title” if no matching reservation is found for a session.

  • Thumbnail previews of DigitalMicrograph files will now include image annotations.

  • Updated SharePoint calendar harvester to be compatible with SharePoint 2016.

  • Various XSLT enhancements for better display of record data.

Documentation improvements

Miscellaneous/Development changes

  • Added helper script to update XLST on NexusLIMS front-end via API, making things easier for development.

  • Fully implemented tests to ensure 100% of codebase is covered by test functions.

  • Refactored record builder and harvester to use Instrument instances rather than string parsing.

v0.0.2 (2020-01-08) - Pre-release version

New features

  • Added ability to use custom CA certificate for network communications (useful if communicating with servers with self-signed certificates). (#49)

  • Added javascript-powered XLST for more interactive and fully-featured display of records in the CDCS front-end (T. Bina).

  • Added session logging .exe application that can be deployed on individual microscope PCs to specify when a session starts and ends (used to get timestamps for file finding during record building). See Session logging (deprecated) for more details.

  • Finished implementation of building AcquisitionActivity representations of experiments, which are then translated into XML for the final record.

  • Implemented prototype record builder script for automated record generation (T. Bina).

  • Preview images are now generated for known dataset types during the creation of acquisition activities in record building.

Bug fixes

  • Updated endpoint for sharepoint calendar API that had broken due to a change in that service. (#37)

  • Fixed issue where schema had duplicate “labels” for certain fields that was causing a confusing display in the CDCS “curate” page. (#42)

Enhancements

  • Added other Nexus instruments to the schema so we can have records from more than just a single Instrument (as it was in initial testing). (#32)

  • Added a unit attribute to parameter values in the Nexus Experiment schema. (#41)

  • Added a place to insert “project” information into Nexus Experiment schema (#44)

  • Improved the implementation of the Instrument type within the Nexus Experiment schema. (#45)

  • Added spiffy logo for NexusLIMS (see the README for more details on its origins).

  • Formatted repository as a proper Python package that can be installed via pip.

  • Generalized metadata extraction process in anticipation of implementing extractors for additional file types.

  • Instrument configuration is now fully pulled from NexusLIMS DB, rather than hard-coded in the application code.

  • XSLT now properly displays preview images rather than a placeholder for each dataset.

Documentation improvements

  • Fix links in README to point to new upstream project location. (#36)

  • Added basic documentation for NexusLIMS package and link via a badge on the README. (#43)

Miscellaneous/Development changes

  • Moved project out of a personal account and into it’s own NexusLIMS group on Gitlab. (#29)

  • Added dedicated folder (separate form data storage location) for NexusLIMS to write dumps of extracted metadata and preview images. (#38)

  • Added database initialization script to create correct NexusLIMS DB structure.

  • Refactored record builder and calendar harvesting into separate submodules to better delineate functionality of the various parts of NexusLIMS

v0.0.1 (2019-03-26) - Pre-release version

New features

  • Implemented SharePoint calendar metadata harvesting for equipment reservations. (#26)

  • Added metadata extractor for FEI TIFF image files produced by SEMs and FIBs. (#35)

  • Created repository to hold initial work on NexusLIMS.

Enhancements

  • Added a concept of “role” (experimental, calibration, etc.) to datasets in the Nexus Experiment schema (#23)

Miscellaneous/Development changes

  • Added CI/CD pipeline for backend tests. (#30)