The TopHatFilter struct represents a zero-sum symmetric second-derivative-like filter that when applied to spectral data has the property of suppressing constant and slowly varying signals (like the continuum) while retaining a linear signal for faster changing signals like the characteristic peaks.

See

• F. H. Schamber Proc Symposium of "X-ray Fluorscence Analysis on Environmental Samples" Chapel Hill 1976 T Dzubay Ed.
• P. Statham Anal Chem 49 no 14 Dec 1977

The TopHatFilter struct optimizes the memory and CPU use when applying top-hat filters to spectrum data.

The easiest way to implement a top-hat filter is as matrix F. The rows represent the filters. The product of the filter and the data vector is the filtered spectrum. The product of the filter times a diagnonal matrix constructed from the data times the transpose of the filter is the covariance of the filtered data. The diagonal matrix constructed from the spectrum data is the covariance matrix associated with the spectrum data because the channels in the spectrum data are independent (thus the matrix is diagnonal) and the magnitude equals the counts in each channels because the spectrum data is nominally Poissonian and in the large number limit, the variance of a Poissonian random variable is the number itself (σ=sqrt(N) => Var = N)

Notes on memory and code optimization: The filter matrix is banded diagonal. Approximately, 2.5% of the elements are non-zero. This suggest use of the BandedMatrix type. The most expensive operation is calculating F⋅D⋅Fᵀ, the covariance matrix of the filtered data. D is a diagonal matrix and so computing each element in F⋅D⋅Fᵀ reduces to a sum over a single variable. Furthermore, the weighted least squares fit doesn't require the full F⋅D⋅Fᵀ, just diag(F⋅D⋅Fᵀ). However, it turns out that we can do better implementing our own banded matrix type largely because D is fully diagonal and the matrix product F⋅D⋅Fᵀ reduces down to a sum over a single variable. The product F⋅d and F⋅D⋅Fᵀ are readily implemented as element-by-element multiplies and sums. Thus storing the filter as offsets and row filters is efficient in both memory and CPU use.

ConstantWidthFilter

A top-hat filter that has constant width determined by FWHM at Mn Kα for all channels.

GaussianFilter

A Gaussian-shaped filter that varies in width with the FWHM of the detector. The Gaussian is offset to ensure the sum of the filter elements is zero.

VariableWidthFilter

A top-hat filter that varies in width with the FWHM of the detector.

tophatfilter(
    charLabel::CharXRayLabel,
    thf::TopHatFilter,
    scale::Float64 = 1.0,
    tol::Float64 = 1.0e-6,
)::FilteredReference

For filtering an ROI on a reference spectrum. Process a portion of a Spectrum with the specified filter. Use a simple edge-based background model.

tophatfilter(
  reflabel::ReferenceLabel,
  roi::UnitRange{Int},
  thf::TopHatFilter,
  scale = 1.0,
  tol = 1.0e-6
)::FilteredReference

For filtering an ROI on a reference spectrum. Process a portion of a Spectrum with the specified filter. Use a naive linear background model.

tophatfilter(spec::Spectrum, thf::TopHatFilter, scale::Float64=1.0, tol::Float64 = 1.0e-4)::FilteredUnknown

For filtering the unknown spectrum. Defaults to the weighted fitting model.

tophatfilter(::Type{FilteredUnknownW}, spec::Spectrum, thf::TopHatFilter, scale::Float64=1.0, tol::Float64 = 1.0e-4)::FilteredUnknownW

For filtering the unknown spectrum. Process the full Spectrum with the specified filter for use with the weighted least squares model.

tophatfilter(spec::Spectrum, thf::TopHatFilter, scale::Float64=1.0, tol::Float64 = 1.0e-4)::FilteredDatum

For filtering the unknown spectrum. Process the full Spectrum with the specified filter.

buildfilter(det::Detector, a::Float64=1.0, b::Float64=2.0)::TopHatFilter

Build the default top-hat filter for the specified detector with the specified top and base parameters.

buildfilter(::Type{<:TopHatFilterType}, det::Detector, a::Float64=1.0, b::Float64=1.0)::TopHatFilter

Build a top-hat-style filter for the specified detector with the specified top and base parameters. The VariableWidthFilter and ConstantWidthFilter types are currently supported.

buildfilter(::Type{GaussianFilter}, det::Detector, a::Float64=1.0, b::Float64=5.0)::TopHatFilter

Build a top-hat filter with Gaussian shape whose width varies with the detector's resolution as a function of X-ray energy for the specified detector with the specified top and base parameters. The a parameter corresponds to the filter width relative to the detector resolution expressed as Gaussian width. So a=1 is a filter whose width equals the detector resolution at each energy. The b parameter is the extent of the filter in Gaussian widths. The default a=1, b=5 corresponds to a filter that has the same resolution as the detector and an extent of 2.5 Gaussian widths above and below the center channel.

Represents the filtered reference spectrum over an ROI. Carries the minimal data necessary to support filter-fitting a single region-of-interest (continguous range of channles) and computing useful output statistics.

Represents the unknown in a filter fit using the weighted fitting model. This is an approximation that produces over optimistic resulting covariance matrix.

FilteredUnknownG

Represents the unknown in a filter fit using the full generalized fitting model. This model is expensive to calculate but uses the full generalized linear fitting model which produces the correct fit covariances.

filterfit(unk::FilteredUnknownW, ffs::AbstractVector{FilteredReference}, alg=fitcontiguousww)::UncertainValues

Filter fit the unknown against ffs, an array of FilteredReference and return the result as an FilterFitResult object. By default use the generalized LLSQ fitting (pseudo-inverse implementation).

This function is designed to reperform the fit if one or more k-ratio is less-than-or-equal-to zero. The FilteredReference corresponding to the negative value is removed from the fit and the fit is reperformed. How the non-positive value is handled is determine by forcezeros. If forcezeros=true, then the returned k-ratio for the non-positive value will be set to zero (but the uncertainty remains the fitted one). However, if forcezeros=false, then the final non-positive k-ratio is returned along with the associated uncertainty. forcezeros=false is better when a number of fit k-ratio sets are combined to produce an averaged k-ratio with reduced uncertainty. forcezeros=true would bias the result positive.

filterfit(unk::FilteredUnknownG, ffs::AbstractVector{FilteredReference}, alg=fitcontiguousw)::UncertainValues

Filter fit the unknown against ffs, an array of FilteredReference and return the result as an FilterFitResult object. By default use the generalized LLSQ fitting (pseudo-inverse implementation).

The result of a spectrum fit.

FilterFitResult

Represents the result of fitting either FilteredUnknownG or FilteredUnknownW to a FilteredUnknown.

Struct elements

label::UnknownLabel  # Identifies the unknown
kratios::UncertainValues # Labeled with ReferenceLabel objects
roi::UnitRange{Int} # Range of channels fit
raw::Vector{Float64} # Raw spectrum data
residual::Vector{Float64} # Residual spectrum

kratios(ffr::FitResult)::Vector{KRatio}

The k-ratios associated with each CharXRayLabel as a vector 'KRatio' objects.

spectrum(fl::FilteredLabel)::Spectrum

The spectrum associated with a FilteredLabel-based type.

residual(ffr::FilterFitResult)::Spectrum

A Spectrum containing the histogram representing the unknown spectrum minus the fitted characteristic peaks shapes times the best fit coefficient.

covariance(fd::FilteredUnknownW, roi::UnitRange{Int})

Like extract(fd,roi) except extracts the covariance diagnonal elements over the specified range of channels. roi must be fully contained within the data in fd.

covariance(fd::FilteredUnknownG, roi::UnitRange{Int})::AbstractMatrix{Float64}

Like extract(fd,roi) except extracts the covariance matrix over the specified range of channels. roi must be fully contained within the filtered edata in fd.

filteredresidual(fit::FilterFitResult, unk::FilteredUnknown, ffs::AbstractVector{FilteredReference})::Vector{Float64}

Computes the difference between the best fit and the unknown filtered spectral data.

A label associated with reference spectra. The label encapsulates the original spectrum and the range of channels represented by this reference object. structs that extend ReferenceLabel should have .roi and .spec members.

SpectrumFeature

A union representing the different type of peak-like features (helpful and harmful) that can appear in a spectrum.

A ReferenceLabel<:FilteredLabel that Represents a reference spectrum associated with a set of characteristic x-rays (CharXRay) objects over a contiguous range of spectrum channels.

A ReferenceLabel<:FilteredLabel that Represents a reference spectrum associated with an escape peak from a set of characteristic x-rays (CharXRay) objects over a contiguous range of spectrum channels.

A FilteredLabel that represents the unknown spectrum.

SpectrumScaling types are designed to rescale spectrum data primarily for plotting.

Implement

Base.show(io::IO, scn::SpectrumScaling)
scalefactor(sc::SpectrumScaling, spec::Spectrum)

Don't scale the spectrum data.

Scale to a fixed total integral.

Scale to a constant dose (Counts/(nA⋅s)). Requires a spectrum has both :ProbeCurrent & :LiveTime.

Scale to a constant dose⋅width (Counts/(nA⋅s/eV)) Requires a spectrum has both :ProbeCurrent & :LiveTime.

Scale to a default sum in the specified ROI.

Scale to a fixed peak intensity

Scale to a constant dose⋅width (Counts/(nA⋅s/eV)) Requires a spectrum has both :ProbeCurrent & :LiveTime.

EnergyScale

An EnergyScale is a way of representing the energy axis associated with X-ray data. The scale may be linear, polynomial or ??? to handle the various different non-linearities that happen with EDS detectors plus we can also handle WDS wavescans.

Implements:

channel(eV::AbstractFloat, sc::EnergyScale)::Int
energy(ch::Int, sc::EnergyScale)::Float64

LinearEnergyScale

An EnergyScale implementation parameterized by channel width and offset.

PolyEnergyScale

An energy scale based on a polynomial function of the channel index.

Detector

An abstract type defining the characteristics of an X-ray detector.

Implements:

channelcount(det::Detector)::Int
scale(det::Detector)::EnergyScale
resolution(eV::Float64, det::Detector)::Float64 # FWHM at eV
energy(ch::Int, det::Detector)::Float64
channel(eV::Float64, det::Detector)::Int
profile(energy::Float64, xrayE::Float64, det::Detector)
lld(det::Detector)::Int
visible(sf::SpectrumFeature, det::Detector)

EDSDetector

Types extending EDSDetector must have member variables

channelcount::Int # Number of channels
scale::EnergyScale # Detector calibration funtion
resolution::Resolution # Detector lineshape function
lld::Int # low level discriminator

" resolution(eV::Float64, res::Resolution) resolution(eV::Float64, det::EDSDetector)

The FWHM at eV for the <:Resolution model.

Resolution

An abstract type describing the channel dependence of the resolution of an EDS detector.

Implements:

resolution(eV::Float64, res::Resolution)::Float # Resolution at specified energy
profile(energy::Float64, xrayE::Float64, res::Resolution) # Amplitude for a signal at the specified energy at the specified energy
extent(xrayE::Float64, res::Resolution, ampl::Float64)::Tuple{2,Float} # The range of channels over which the signal exceeds ampl

simpleEDSwICC(chCount::Integer, width::Float64, offset::Float64, fwhmatmnka::Float64, lld::Int=channel(150.0 eV))

Construct simple model of an EDS detector with incomplete charge collection at low X-ray energies.

The multidimensional dataset with an EnergyScale and Symbol indexed properties. A Signal contains N spatial/temporal axes and 1 data axis. An object might be constructed as Signal(energy, props, (4096, 1024, 2048)) where there are 4096 channels, 1024 rows and 2048 columns. Signal may be 1, 2,.. N dimensional but since they reside in memory, there are practical limits.

A type for data sets containing multiple closely related spectra. A Signal is slighly more restricted than an Array{Spectrum,N} because all the Spectra in a Signal are assumed to have certain properties in common - the EnergyScale and a set of common Spectrum properties. You can also specify an efficient packing of the data by using UInt8, UInt16, ... etc as required to hold the data.

Special property tags:

:Elapse  # Total elapse time for map (so sig[:RealTime] ≈ sig[:Elapse]/length(sig)
:Axes # Names for axes [ "Data", "Y", "X" ]

HyperSpectrum(sig::Signal)

HyperSpectrum is a wrapper around Signal to facilitate access to the the data as Spectrum objects.

readrplraw(rplfilename::AbstractString, rawfilename::AbstractString, scale::EnergyScale, props::Dict{Symbol,Any}=Dict{Symbol,Any}())
readrplraw(rplio::IO, rawio::IO, scale::EnergyScale, props::Dict{Symbol,Any}=Dict{Symbol,Any}())

Read a RPL/RAW file pair from IO or filename into a Signal obect. The reader supports :bigendian, :littleendian ordering and :vector or :image alignment. Since the Signal can be very large it is beneficial to release and collected the associated memory when you are done with the data by assigning nothing to the variable (or allowing it to go out of scope) and then calling GC.gc() to force a garbage collection.

* Ordering: The individual data items may be in `:littleendian` or `:bigendian`.
** `:littleendian` => Intel/AMD and others
** `:bigendian` => ARM/PowerPC/Motorola
* Alignment:  The data in the file can be organized as `:vector` or `:image`.  However, the data will be

reorganized into 'vector' format when returned as a Signal. **:vector => Spectrum/data vector as contiguous blocks by pixel ** :image => Each channel of data organized in image planes * Data types: signed/unsigned 8/16/32-bit integers or 16-bit/32-bit/64-bit floats

Standard LISPIX Parameters in .rpl File

.rpl files consist of 9 lines. Each line consists of a 'key'<tab>'value' where there is one and only one tab and possibly other space between the parameter name and parameter value. Parameter names are case-insensitive. The first line in the files is "key<tab>value". Subsequent lines contain the keys and values described in this table.

This .rpl file indicates the image is 849 pixels wide and 846 pixels high, with 4096 levels in the depth dimension.

ashyperspectrum(sig::Signal, name::AbstractString="Hyper-Spectrum")

Convert the Array{<:Real, N} perspective into a Array{Spectrum{<:Real}, N} perspective.

Special property tags:

:Cartesian # The pixel index of a Spectrum extracted from a HyperSpectrum

plane(hss::Signal, chs::UnitRange, normalize=false)

Sums a contiguous range of data planes into an Array. The dimension of the result is one less than the dimension of the Signal.

plane(hss::Signal, ch::Int, normalize=false)

Sums a contiguous range of data planes into an Array. The dimension of the result is one less than the dimension of the Signal.

plane(hss::HyperSpectrum, chs::Union{Int,UnitRange{Int}}, norm=false) =

Extract as an Array the sum of the data in chs.

roiimage(hss::Signal, chs::UnitRange)

Create a count map from the specified contiguous range of channels.

roiimage(hss::Signal, cxr::CharXRay, n=5)

Create a count map for the specified characteristic X-ray.

asimage(vqr::HyperspectrumQuant, idx::Int; transform=identity)

Create an image that represents the data associated with the idx label. transform is a function nominally from x -> [0,1] which is applied to the raw quantified results.

Examples:

transform=x->0.8x                  # linear scaling
transform=x->log10(1.0+99.0x)/2.0  # log scaling
transform=x->1.0-x                 # invert

compressed(sig::Signal)

Returns a Signal with smaller or equal storage space to sig without losing any infomation.

maxpixel(sig::Signal)

Compute Bright's Max-Pixel derived signal.

maxpixel(hss::HyperSpectrum)

Produce a maxpixel spectrum.

generated(cm::ContinuumModel, ea::Float64)

Compute the intensity of the measured continuum generated from the material and conditions specified in the continuum model object at the specified measured energy ea.

continuumrois(elms, det::EDSDetector, minE::Float64, maxE::Float64)

Compute the ROIs for the contiguous continuum regions for the specified elements elms on an EDSDetector for the specified range of energies.

emitted(cm::ContinuumModel, ea::Float64)

Compute the intensity of the measured continuum emitted from the material and conditions specified in the continuum model object at the specified measured energy ea.

fitcontinuum(
  spec::Spectrum,
  resp::AbstractArray,
  rois::Vector{UnitRange};
  brem::Type{<:NeXLBremsstrahlung} = Castellano2004a,
  mc::Type{<:MatricCorrection} = Riveros1993,
)

Fit a continuum model to the specified range of channels (`rois`).  The `resp` argument is a matrix which describes

the detector response on a channel-by-channel basis. It can be calculated from an EDSDetector and an DetectorEfficiency using resp = NeXLSpectrum.detectorresponse(det, eff). The Spectrum object must have the :Composition, :BeamEnergy and :TakeOffAngle properties defined.

fitcontinuum(
  spec::Spectrum,
  det::EDSDetector,
  resp::AbstractArray{<:Real,2}; #
  minE::Float64 = 1.5e3,
  maxE::Float64 = 0.95 * spec[:BeamEnergy],
  brem::Type{<:NeXLBremsstrahlung} = Castellano2004a,
  mc::Type{<:MatrixCorrection} = Riveros1993,
)

Fit the continuum from ROIs determined from the data within the spectrum (:Composition, :BeamEnergy & :TakeOffAngle). The ROIs are computed using continuumrois(...) and each roi is fit seperately.

fittedcontinuum(
  spec::Spectrum,
  det::EDSDetector,
  resp::AbstractArray{<:Real,2}; #
  mode = :Global [ | :Local ] # Fit to all ROIs simultaneously (:Global) or to each roi independently (:Local)
  minE::Float64 = 1.5e3,
  maxE::Float64 = 0.95 * spec[:BeamEnergy],
  brem::Type{<:NeXLBremsstrahlung} = Castellano2004a,
  mc::Type{<:MatrixCorrection} = Riveros1993,
)::Spectrum

Fit the continuum under the characteristic peaks by fitting the closest continuum ROIs. The low energy peaks are fit using the continuum immediately higher in energy and the high energy peaks are fit using the continuum on both sides.

indexofmaxpixel(sig::Signal, ch::Int) # at channel `ch`
indexofmaxpixel(sig::Signal) # all channels
indexofmaxpixel(sig::Signal, ch::Int, cis::CartesianIndices)
indexofmaxpixel(sig::Signal, cis::CartesianIndices)
indexofmaxpixel(hs::HyperSpectrum, ch::Int) # at channel `ch`
indexofmaxpixel(hs::HyperSpectrum) # all channels
indexofmaxpixel(hs::HyperSpectrum, ch::Int, cis::CartesianIndices)
indexofmaxpixel(hs::HyperSpectrum, cis::CartesianIndices)

Find the coordinates producing the maximum value in data[ch] or data[:] within 'cis' or full spatial dimensions.

roiimages(hss::Signal, achs::Vector{UnitRange{Int}})

Create an array of Gray images representing the intensity in each range of channels in in achs. They are normalized such the the most intense pixel in any of them defines white.

roiimages(hss::HyperSpectrum, achs::Vector{UnitRange{Int}})

Create an array of Gray images representing the intensity in each range of channels in in achs. They are normalized such the the most intense pixel in any of them defines white.

roiimages(hss::HyperSpectrum, cxrs::Vector{CharXRay}, n=5)

Create an array of Gray images representing the intensity in each of the CharXRay lines in cxrs. They are normalized such the the most intense pixel in any of them defines white.

detectorresponse(det::EDSDetector, eff::DetectorEfficiency, incidence::Float64=π/2)::AbstractMatrix

Build a matrix which models the detector response including aspects like the detector efficiency, the resolution, the escape peaks. All the warts that can be modeled within a linear model but not things like coincidence peaks that are non-linear. This function can (!should!) be specialized for more sophisticated detector models that include more warts.

Example:

genint = computegeneratedintensity(....) # Either characteristic or Bremsstrahlung...
det = simpleEDS(4096, 5.0, 0.0, 132.0, 10)
eff = SDDEfficiency(AP33Tabulation(); thickness=0.0370, deadlayer=30.0e-7, entrance=Film(pure(n"Al"), 10.0e-7))
resp = detectorresponse(det, eff)
# finally compute the measured signal
measured = genint*resp

weight(esc::EscapeArtifact, factor=0.01)

The weight of an EscapeArtifact which is factor * weight(esc.xray).

extents(cxrs::AbstractVector{<:SpectrumFeature},det::Detector,ampl::Float64)::Vector{UnitRange{Int}}

Determine the contiguous ranges of channels over which the specified collection of X-rays will be measured on the specified detector. The ampl determines the extent of each peak.

" profile(energy::Float64, xrayE::Float64, res::Resolution)

Calculates a Gaussian profile for an X-ray of xrayE (eV) for a detector with the specified resolution. Maintains normalization to a sum of unity.

" profile(ch::Int, xrayE::Float64, det::EDSDetector)

Calculates the profile for an X-ray of xrayE (eV) for a detector with the specified resolution. Performs a crude integration to account for the channel width.

subtractcontinuum(
  spec::Spectrum,
  det::EDSDetector,
  resp::AbstractArray{<:Real,2}; #
  minE::Float64 = 1.5e3,
  maxE::Float64 = 0.95 * spec[:BeamEnergy],
  brem::Type{<:NeXLBremsstrahlung} = Castellano2004a,
  mc::Type{<:MatrixCorrection} = Riveros1993,
)::Spectrum

Computes the characteristic-only spectrum by subtracting the .

heterogeneity(lbl::ReferenceLabel, ffrs::Vector{FilterFitResult})

Computes the ratio of the standard deviation of the measured values over the mean calculated uncertainty from the fit. A value near 1 means the sample appears homogeneous and a value greater than 1 means the sample appears heterogeneous.

visible(cxrs::AbstractVector{<:SpectrumFeature}, det::Detector)

Returns the characteristic x-rays that are visible on the specified detector (ie. Between the LLD and the maximum channel).

visible(sf::SpectrumFeature, det::Detector)

Is sf visible on the specified Detector?

extract(fd::FilteredReference, roi::UnitRange{Int})

Extract the filtered data representing the specified range. roi must fully encompass the filtered data in fd.

extract(fd::FilteredUnknown, roi::UnitRange{Int})::AbstractVector{Float64}

Extract the filtered data representing the specified range. roi must be fully contained within the filtered data in fd.

extent(xrayE::Float64, res::Resolution, ampl::Float64)

Calculates the extent of the peak interval for an x-ray of the specified energy.

extent(escape::EscapeArtifact, res::Resolution, ampl::Float64)::Tuple{2,Float64}

The extent of an escape artifact is determined by the resolution of the detector at the energy of the escape peak.

extent(escape::ComptonArtifact, res::Resolution, ampl::Float64)::Tuple{2,Float64}

The extent of a Compton artifact is determined by the resolution of the detector at the energy of the Compton peak.

extent(sf::SpectrumFeature, det::Detector, ampl::Float64)::Tuple{Float64, Float64}

Computes the channel range encompassed by the specified set of x-ray transitions down to an intensity of ampl. Relative line weights are taken into account.

characteristiccounts(ffr::FiterFitResult, strip)

Number of spectrum counts that were accounted for by the fitted elements with the strip Element(s) removed.

scale(det::Detector)

EnergyScale associated with this detector.

HyperspectrumQuant

Represents the result from a fit(...) of a HyperSpectrum object.

charXRayLabels(#
  spec::Spectrum, #
  elm::Element, #
  allElms::AbstractVector{Element}, #
  det::Detector, #
  ampl::Float64, #
  maxE::Float64=1.0e6)::Vector{SpectrumFeature}

Creates a vector CharXRayLabel objects associated with 'elm' for a spectrum containing the elements 'allElms' assuming that it was collected on 'det'. ROIs in which other elements from 'allElms' interfere with 'elm' will not be included.

channelcount(det::Detector)

Number of detector channels.

Beryllium(thickness=5.0e-4)

Construct a beryllium window.

AP33Tabulation()
AP5Tabulation()

Construct tabulated window models for the Moxtek AP3.3 and AP5 windows.