Submodule to work with grouped data.

Functions:

block_by(ndat, block[, mode])

Get group by array for block reduction.

factor_by(by[, sort])

Factor by to codes and groups.

factor_by_to_index(by, **kwargs)

Transform group_idx to quantities to be used with reduce_data_indexed().

reduce_data_grouped(data, by, *[, axis, ...])

Reduce data by group.

reduce_data_indexed(data, *, index, ...[, ...])

Reduce data by index

resample_data_indexed(data, sampler, *[, ...])

Resample using indexed reduction.
cmomy.grouped.block_by(ndat, block, mode='drop_last')[source]#

Get group by array for block reduction.

Parameters:

  • ndat (int) – Size of by.

  • block (int) – Block size. Negative values is a single block.

  • mode ({"drop_first", "drop_last", "expand_first", "expand_last"}) –

    What to do if ndat does not divide equally by block.

    • ”drop_first” : drop first samples

    • ”drop_last” : drop last samples

    • ”expand_first”: expand first block size

    • ”expand_last”: expand last block size

Returns:

by (ndarray) – Group array for block reduction.

See also

cmomy.grouped.reduce_data_grouped

Examples

>>> block_by(5, 2)
array([ 0,  0,  1,  1, -1])

>>> block_by(5, 2, mode="drop_first")
array([-1,  0,  0,  1,  1])

>>> block_by(5, 2, mode="expand_first")
array([0, 0, 0, 1, 1])

>>> block_by(5, 2, mode="expand_last")
array([0, 0, 1, 1, 1])
cmomy.grouped.factor_by(by, sort=True)[source]#

Factor by to codes and groups.

Parameters:

  • by (sequence) – Values to group by. Negative or None values indicate to skip this value. Note that if by is a pandas pandas.Index object, missing values should be marked with None only.

  • sort (bool, default True) – If True (default), sort groups. If False, return groups in order of first appearance.

Returns:

  • groups (list or pandas.Index) – Unique group names (excluding negative or None Values.)

  • codes (ndarray of int) – Indexer into groups.

Examples

>>> by = [1, 1, 0, -1, 0, 2, 2]
>>> groups, codes = factor_by(by, sort=False)
>>> groups
[1, 0, 2]
>>> codes
array([ 0,  0,  1, -1,  1,  2,  2])

Note that with sort=False, groups are in order of first appearance.

>>> groups, codes = factor_by(by)
>>> groups
[0, 1, 2]
>>> codes
array([ 1,  1,  0, -1,  0,  2,  2])

This also works for sequences of non-integers.

>>> by = ["a", "a", None, "c", "c", -1]
>>> groups, codes = factor_by(by)
>>> groups
['a', 'c']
>>> codes
array([ 0,  0, -1,  1,  1, -1])

And for pandas.Index objects

>>> import pandas as pd
>>> by = pd.Index(["a", "a", None, "c", "c", None])
>>> groups, codes = factor_by(by)

>>> groups
Index(['a', 'c'], dtype='object')
>>> codes
array([ 0,  0, -1,  1,  1, -1])
cmomy.grouped.factor_by_to_index(by, **kwargs)[source]#

Transform group_idx to quantities to be used with reduce_data_indexed().

Parameters:
Returns:

  • groups (list or pandas.Index) – Unique groups in group_idx (excluding Negative or None values in group_idx if exclude_negative is True).

  • index (ndarray) – Indexing array. index[start[k]:end[k]] are the index with group groups[k].

  • start (ndarray) – See index

  • end (ndarray) – See index.

See also

cmomy.grouped.reduce_data_indexed, factor_by

Examples

>>> factor_by_to_index([0, 1, 0, 1])
([0, 1], array([0, 2, 1, 3]), array([0, 2]), array([2, 4]))

>>> factor_by_to_index(["a", "b", "a", "b"])
(['a', 'b'], array([0, 2, 1, 3]), array([0, 2]), array([2, 4]))

Also, missing values (None or negative) are excluded:

>>> factor_by_to_index([None, "a", None, "b"])
(['a', 'b'], array([1, 3]), array([0, 1]), array([1, 2]))

You can also pass pandas.Index objects:

>>> factor_by_to_index(pd.Index([None, "a", None, "b"], name="my_index"))
(Index(['a', 'b'], dtype='object', name='my_index'), array([1, 3]), array([0, 1]), array([1, 2]))
cmomy.grouped.reduce_data_grouped(data, by, *, axis=MISSING, dim=MISSING, mom_ndim=None, mom_axes=None, mom_dims=None, mom_params=None, out=None, dtype=None, casting='same_kind', order=None, parallel=None, axes_to_end=False, group_dim=None, groups=None, keep_attrs=None, apply_ufunc_kwargs=None)[source]#

Reduce data by group.

Parameters:

  • data (ndarray or DataArray or Dataset) – Moments array(s). It is assumed moment dimensions are last.

  • by (array-like of int) – Group by values of same length as data along sampled dimension. Negative values indicate no group (i.e., skip this index).

  • axis (int, optional) – Axis to reduce/sample along. Note that negative values are relative to data.ndim - mom_ndim. It is assumed that the last dimensions are for moments. For example, if data.shape == (1,2,3) with mom_ndim=1, axis = -1 `` would be equivalent to ``axis = 1. Defaults to axis=-1.

  • dim (hashable) – Dimension to reduce/sample along.

  • mom_ndim ({1, 2}, optional) – Value indicates if moments (mom_ndim = 1) or comoments (mom_ndim=2). If not specified and data is an xarray object attempt to infer mom_ndim from mom_dims. Otherwise, default to mom_ndim = 1.

  • mom_axes (int or tuple of int, optional) – Location of the moment dimensions. Default to (-mom_ndim, -mom_ndim+1, ...). If specified and mom_ndim is None, set mom_ndim to len(mom_axes). Note that if mom_axes is specified, negative values are relative to the end of the array. This is also the case for axes if mom_axes is specified.

  • mom_dims (hashable or tuple of hashable) – Name of moment dimensions. If specified, infer mom_ndim from mom_dims. If also pass mom_ndim, check that mom_dims is consistent with mom_dims. If not specified, defaults to data.dims[-mom_ndim:]. This is primarily used if data is a Dataset, or if mom_dims are not the last dimensions.

  • mom_params (cmomy.MomParams or cmomy.MomParamsDict or dict, optional) – Moment parameters. You can set moment parameters axes and dims using this option. For example, passing mom_params={"dim": ("a", "b")} is equivalent to passing mom_dims=("a", "b"). You can also pass as a cmomy.MomParams object with mom_params=cmomy.MomParams(dims=("a", "b")).

  • out (ndarray) – Optional output array. If specified, output will be a reference to this array. Note that if the output if method returns a Dataset, then this option is ignored.

  • dtype (dtype) – Optional dtype for output data.

  • casting ({'no', 'equiv', 'safe', 'same_kind', 'unsafe'}, optional) –

    Controls what kind of data casting may occur.

    • ’no’ means the data types should not be cast at all.

    • ’equiv’ means only byte-order changes are allowed.

    • ’safe’ means only casts which can preserve values are allowed.

    • ’same_kind’ means only safe casts or casts within a kind, like float64 to float32, are allowed.

    • ’unsafe’ (default) means any data conversions may be done.

  • order ({"C", "F"}, optional) – Order argument. See numpy.zeros().

  • axes_to_end (bool) – If True, place sampled dimension (if exists in output) and moment dimensions at end of output. Otherwise, place sampled dimension (if exists in output) at same position as input axis and moment dimensions at same position as input (if input does not contain moment dimensions, place them at end of array).

  • parallel (bool, optional) – If True, use parallel numba numba.njit or numba.guvectorized code if possible. If None, use a heuristic to determine if should attempt to use parallel method.

  • group_dim (str, optional) – Name of the output group dimension. Defaults to dim.

  • groups (sequence, optional) – Sequence of length by.max() + 1 to assign as coordinates for group_dim.

  • keep_attrs ({"drop", "identical", "no_conflicts", "drop_conflicts", "override"} or bool, optional) –

    • ‘drop’ or False: empty attrs on returned xarray object.

    • ’identical’: all attrs must be the same on every object.

    • ’no_conflicts’: attrs from all objects are combined, any that have the same name must also have the same value.

    • ’drop_conflicts’: attrs from all objects are combined, any that have the same name but different values are dropped.

    • ’override’ or True: skip comparing and copy attrs from the first object to the result.

  • apply_ufunc_kwargs (dict-like) – Extra parameters to xarray.apply_ufunc(). One useful option is on_missing_core_dim, which can take the value "copy" (the default), "raise", or "drop" and controls what to do with variables of a Dataset missing core dimensions. Other options are join, dataset_join, dataset_fill_value, and dask_gufunc_kwargs. Unlisted options are handled internally.

Returns:

out (ndarray or DataArray or Dataset) – Reduced data of same type as input data, with shape out.shape = (..., shape[axis-1], ngroup, shape[axis+1], ..., mom0, ...) where shape = data.shape and ngroups = by.max() + 1.

See also

cmomy.grouped.factor_by

Examples

>>> import cmomy
>>> data = np.ones((5, 3))
>>> by = [0, 0, -1, 1, -1]
>>> reduce_data_grouped(data, mom_ndim=1, axis=0, by=by)
array([[2., 1., 1.],
       [1., 1., 1.]])

This also works for DataArray objects. In this case, the groups are added as coordinates to group_dim

>>> xout = xr.DataArray(data, dims=["rec", "mom"])
>>> reduce_data_grouped(xout, mom_ndim=1, dim="rec", by=by, group_dim="group")
<xarray.DataArray (group: 2, mom: 3)> Size: 48B
array([[2., 1., 1.],
       [1., 1., 1.]])
Dimensions without coordinates: group, mom

Note that if by skips some groups, they will still be included in The output. For example the following by skips the value 0.

>>> by = [1, 1, -1, 2, 2]
>>> reduce_data_grouped(xout, mom_ndim=1, dim="rec", by=by)
<xarray.DataArray (rec: 3, mom: 3)> Size: 72B
array([[0., 0., 0.],
       [2., 1., 1.],
       [2., 1., 1.]])
Dimensions without coordinates: rec, mom

If you want to ensure that only included groups are used, use cmomy.grouped.factor_by(). This has the added benefit of working with non integer groups as well

>>> by = ["a", "a", None, "b", "b"]
>>> groups, codes = cmomy.grouped.factor_by(by)
>>> reduce_data_grouped(xout, mom_ndim=1, dim="rec", by=codes, groups=groups)
<xarray.DataArray (rec: 2, mom: 3)> Size: 48B
array([[2., 1., 1.],
       [2., 1., 1.]])
Coordinates:
  * rec      (rec) <U1 8B 'a' 'b'
Dimensions without coordinates: mom
cmomy.grouped.reduce_data_indexed(data, *, index, group_start, group_end, scale=None, axis=MISSING, dim=MISSING, mom_ndim=None, mom_axes=None, mom_dims=None, mom_params=None, out=None, dtype=None, casting='same_kind', order=None, parallel=None, axes_to_end=False, coords_policy='first', group_dim=None, groups=None, keep_attrs=None, apply_ufunc_kwargs=None)[source]#

Reduce data by index

Parameters:

  • data (ndarray or DataArray or Dataset) – Moments array(s). It is assumed moment dimensions are last.

  • index (ndarray) – Index into data.shape[axis].

  • group_start (ndarray) – Start, end of index for a group. index[group_start[group]:group_end[group]] are the indices for group group.

  • group_end (ndarray) – Start, end of index for a group. index[group_start[group]:group_end[group]] are the indices for group group.

  • scale (ndarray, optional) – Weights of same size as index.

  • axis (int, optional) – Axis to reduce/sample along. Note that negative values are relative to data.ndim - mom_ndim. It is assumed that the last dimensions are for moments. For example, if data.shape == (1,2,3) with mom_ndim=1, axis = -1 `` would be equivalent to ``axis = 1. Defaults to axis=-1.

  • dim (hashable) – Dimension to reduce/sample along.

  • mom_ndim ({1, 2}, optional) – Value indicates if moments (mom_ndim = 1) or comoments (mom_ndim=2). If not specified and data is an xarray object attempt to infer mom_ndim from mom_dims. Otherwise, default to mom_ndim = 1.

  • mom_axes (int or tuple of int, optional) – Location of the moment dimensions. Default to (-mom_ndim, -mom_ndim+1, ...). If specified and mom_ndim is None, set mom_ndim to len(mom_axes). Note that if mom_axes is specified, negative values are relative to the end of the array. This is also the case for axes if mom_axes is specified.

  • mom_dims (hashable or tuple of hashable) – Name of moment dimensions. If specified, infer mom_ndim from mom_dims. If also pass mom_ndim, check that mom_dims is consistent with mom_dims. If not specified, defaults to data.dims[-mom_ndim:]. This is primarily used if data is a Dataset, or if mom_dims are not the last dimensions.

  • mom_params (cmomy.MomParams or cmomy.MomParamsDict or dict, optional) – Moment parameters. You can set moment parameters axes and dims using this option. For example, passing mom_params={"dim": ("a", "b")} is equivalent to passing mom_dims=("a", "b"). You can also pass as a cmomy.MomParams object with mom_params=cmomy.MomParams(dims=("a", "b")).

  • out (ndarray) – Optional output array. If specified, output will be a reference to this array. Note that if the output if method returns a Dataset, then this option is ignored.

  • dtype (dtype) – Optional dtype for output data.

  • casting ({'no', 'equiv', 'safe', 'same_kind', 'unsafe'}, optional) –

    Controls what kind of data casting may occur.

    • ’no’ means the data types should not be cast at all.

    • ’equiv’ means only byte-order changes are allowed.

    • ’safe’ means only casts which can preserve values are allowed.

    • ’same_kind’ means only safe casts or casts within a kind, like float64 to float32, are allowed.

    • ’unsafe’ (default) means any data conversions may be done.

  • order ({"C", "F", "A", "K"}, optional) – Order argument. See numpy.asarray().

  • parallel (bool, optional) – If True, use parallel numba numba.njit or numba.guvectorized code if possible. If None, use a heuristic to determine if should attempt to use parallel method.

  • axes_to_end (bool) – If True, place sampled dimension (if exists in output) and moment dimensions at end of output. Otherwise, place sampled dimension (if exists in output) at same position as input axis and moment dimensions at same position as input (if input does not contain moment dimensions, place them at end of array).

  • coords_policy ({'first', 'last', 'group', None}) –

    Policy for handling coordinates along dim if by is specified for DataArray data. If no coordinates do nothing, otherwise use:

    • ’first’: select first value of coordinate for each block.

    • ’last’: select last value of coordinate for each block.

    • ’group’: Assign unique groups from group_idx to dim

    • None: drop any coordinates.

    Note that if coords_policy is one of first or last, parameter groups will be ignored.

  • group_dim (str, optional) – Name of the output group dimension. Defaults to dim.

  • groups (sequence, optional) – Sequence of length by.max() + 1 to assign as coordinates for group_dim.

  • keep_attrs ({"drop", "identical", "no_conflicts", "drop_conflicts", "override"} or bool, optional) –

    • ‘drop’ or False: empty attrs on returned xarray object.

    • ’identical’: all attrs must be the same on every object.

    • ’no_conflicts’: attrs from all objects are combined, any that have the same name must also have the same value.

    • ’drop_conflicts’: attrs from all objects are combined, any that have the same name but different values are dropped.

    • ’override’ or True: skip comparing and copy attrs from the first object to the result.

  • apply_ufunc_kwargs (dict-like) – Extra parameters to xarray.apply_ufunc(). One useful option is on_missing_core_dim, which can take the value "copy" (the default), "raise", or "drop" and controls what to do with variables of a Dataset missing core dimensions. Other options are join, dataset_join, dataset_fill_value, and dask_gufunc_kwargs. Unlisted options are handled internally.

Returns:

out (ndarray or DataArray or Dataset) – Reduced data of same type as input data, with shape out.shape = (..., shape[axis-1], ngroup, shape[axis+1], ..., mom0, ...), where shape = data.shape and ngroup = len(group_start).

See also

cmomy.grouped.factor_by_to_index

Examples

This is a more general reduction than reduce_data_grouped(), but it can be used similarly.

>>> import cmomy
>>> data = np.ones((5, 3))
>>> by = ["a", "a", "b", "b", "c"]
>>> groups, index, start, end = cmomy.grouped.factor_by_to_index(by)
>>> reduce_data_indexed(
...     data, mom_ndim=1, axis=0, index=index, group_start=start, group_end=end
... )
array([[2., 1., 1.],
       [2., 1., 1.],
       [1., 1., 1.]])

This also works for DataArray objects

>>> xout = xr.DataArray(data, dims=["rec", "mom"])
>>> reduce_data_indexed(
...     xout,
...     mom_ndim=1,
...     dim="rec",
...     index=index,
...     group_start=start,
...     group_end=end,
...     group_dim="group",
...     groups=groups,
...     coords_policy="group",
... )
<xarray.DataArray (group: 3, mom: 3)> Size: 72B
array([[2., 1., 1.],
       [2., 1., 1.],
       [1., 1., 1.]])
Coordinates:
  * group    (group) <U1 12B 'a' 'b' 'c'
Dimensions without coordinates: mom
cmomy.grouped.resample_data_indexed(data, sampler, *, mom_ndim=None, axis=MISSING, mom_axes=None, out=None, dtype=None, casting='same_kind', order=None, parallel=True, axes_to_end=False, dim=MISSING, mom_dims=None, coords_policy='first', rep_dim='rep', groups=None, keep_attrs=None)[source]#

Resample using indexed reduction.