GitHub

Cross-sectional transforms

Cross-sectional transforms

A cross-sectional transform rescales one observation of an observations × assets matrix against the other assets of that same observation. No member reads a second observation and no member is fitted, so a transform is configuration alone and it runs on a plain matrix.

The estimation set of an observation is what its statistics are computed from: the finite cells carrying a positive benchmark weight when w is given, and the finite cells otherwise. A cell outside that set is still transformed against it, so an asset the benchmark does not hold is scored on the same scale as one it does. An observation whose estimation set is empty returns a NaN at every asset.

The benchmark weights and the group labels are arguments of cross_sectional_transform, never fields, because one transform runs against a different benchmark and a different classification at every call site. cross_sectional_groups reads the labels off the codes of a CategoricalPanelField.

Types

PortfolioOptimisers.AbstractCrossSectionalTransform — Type
abstract type AbstractCrossSectionalTransform <: AbstractEstimator

Abstract supertype for all cross-sectional transform types.

A member rescales or reshapes one observation of an observations × assets matrix against the other assets of that same observation, so no member reads a second observation and no member is fitted. Two members treat an outlier and three members turn a raw quantity into a score.

All concrete and/or abstract types representing cross-sectional transforms should be subtypes of AbstractCrossSectionalTransform.

Interfaces

In order to implement a new concrete type that works seamlessly with the library, subtype AbstractCrossSectionalTransform and implement the following method:

cross_sectional_transform

Arguments

  • ct: The concrete subtype instance.
  • X: Data matrix observations × assets, where a NaN marks a missing cell.
  • w: Benchmark weight matrix observations × assets, or nothing.
  • groups: Group label matrix observations × assets, or nothing.

Returns

  • Y::Matrix{<:AbstractFloat}: Transformed matrix observations × assets.

Related

source
PortfolioOptimisers.CrossSectionalWinsoriser — Type
struct CrossSectionalWinsoriser{__T_low, __T_high} <: AbstractCrossSectionalTransform

Clips every value of an observation into the band between two percentiles of that observation's cross-section.

The clip is a hard one, so a value beyond a band edge takes the edge itself and every value inside the band is untouched. It is the cheapest way to stop one asset's extreme reading from dominating a cross-sectional fit.

Mathematical definition

\[\begin{align} x_{t,i}' &= \min\left(\max\left(x_{t,i},\, q_{t}^{\mathrm{lo}}\right),\, q_{t}^{\mathrm{hi}}\right)\,. \end{align}\]

Where:

  • $x_{t,i}$: Value of asset $i$ at observation $t$.
  • $q_{t}^{\mathrm{lo}}$, $q_{t}^{\mathrm{hi}}$: Percentiles of the estimation set of observation $t$, at the levels low and high.

The percentiles are equal-weighted over the estimation set. An asset outside that set is still clipped to the same band, and a NaN stays a NaN.

Fields

  • low: Percentile level of the lower edge of the band.
  • high: Percentile level of the upper edge of the band.

Constructors

CrossSectionalWinsoriser(; low::Real = 0.01, high::Real = 0.99) -> CrossSectionalWinsoriser

Keywords correspond to the struct's fields.

Validation

  • 0 <= low < high <= 1.

Examples

julia> CrossSectionalWinsoriser()CrossSectionalWinsoriser   low ┼ Float64: 0.01  high ┴ Float64: 0.99

Related

source
PortfolioOptimisers.CrossSectionalTanhShrinker — Type
struct CrossSectionalTanhShrinker{__T_knee, __T_atol} <: AbstractCrossSectionalTransform

Compresses every value of an observation towards the centre of that observation's cross-section, through a hyperbolic tangent.

The map is smooth and strictly increasing, so it creates no jump at a threshold and it keeps the order of the tail. A value near the centre moves almost not at all, and a value far from it is pulled in hard. The output keeps the units of the input.

Mathematical definition

\[\begin{align} x_{t,i}' &= m_{t} + h_{t} \tanh\left(\frac{x_{t,i} - m_{t}}{h_{t}}\right) \\ h_{t} &= c \, \gamma \, \mathrm{med}_{j}\left(\left\lvert x_{t,j} - m_{t} \right\rvert\right)\,. \end{align}\]

Where:

  • $x_{t,i}$: Value of asset $i$ at observation $t$.
  • $m_{t}$: Median of the estimation set of observation $t$.
  • $\gamma$: The constant 1.4826…, which makes a median absolute deviation consistent with a standard deviation under normality.
  • $c$: Compression knee.
  • $h_{t}$: Half-width of the near-linear region of observation $t$.

The median and the median absolute deviation are equal-weighted over the estimation set. An asset outside that set is still compressed against the same statistics, and a NaN stays a NaN. An observation whose robust scale is at or below atol is returned unchanged, because it carries no dispersion to compress against.

Fields

  • knee: Half-width of the near-linear region, counted in robust standard deviations. A larger knee compresses less.
  • atol: Absolute tolerance below which a cross-sectional scale counts as zero. An observation at or below it carries no dispersion, so its finite cells score zero rather than dividing by that scale.

Constructors

CrossSectionalTanhShrinker(; knee::Real = 3.0, atol::Real = 1e-12) -> CrossSectionalTanhShrinker

Keywords correspond to the struct's fields.

Validation

  • isfinite(knee) and knee > 0.
  • isfinite(atol) and atol >= 0.

Examples

julia> CrossSectionalTanhShrinker()CrossSectionalTanhShrinker  knee ┼ Float64: 3.0  atol ┴ Float64: 1.0e-12

Related

source
PortfolioOptimisers.CrossSectionalStandardiser — Type
struct CrossSectionalStandardiser{__T_min_group_size, __T_atol} <: AbstractCrossSectionalTransform

Scores every value of an observation as a cross-sectional z-score, optionally inside its own group first.

The centre is weighted and the scale is equal-weighted, so the score says how far an asset sits from the benchmark's own centre, measured in the dispersion of the cross-section. A grouped score is then brought back to a common centre and scale, so scores from different groups are comparable.

Mathematical definition

\[\begin{align} z_{t,i} &= \frac{x_{t,i} - \mu_{t}}{\sigma_{t}} \\ \mu_{t} &= \frac{\sum_{j \in \mathcal{E}_{t}} w_{t,j} x_{t,j}}{\sum_{j \in \mathcal{E}_{t}} w_{t,j}} \\ \sigma_{t} &= \sqrt{\frac{1}{\lvert \mathcal{E}_{t} \rvert - 1} \sum_{j \in \mathcal{E}_{t}} \left(x_{t,j} - \mu_{t}\right)^{2}}\,. \end{align}\]

Where:

  • $x_{t,i}$: Value of asset $i$ at observation $t$.
  • $w_{t,j}$: Benchmark weight of asset $j$ at observation $t$.
  • $\mathcal{E}_{t}$: Estimation set of observation $t$.
  • $\mu_{t}$, $\sigma_{t}$: Centre and scale of observation $t$.

With group labels the same pair is computed inside each group, and the score is recentred and rescaled over the whole observation afterwards. A group holding fewer than min_group_size estimation assets, and every asset labelled CS_MISSING_GROUP, takes the whole-observation pair instead. An observation whose scale is at or below atol scores zero, and a NaN stays a NaN.

Fields

  • min_group_size: Smallest estimation set a group may carry and still be estimated from. A group below it, and every asset that carries no group, takes the whole observation's statistics instead.
  • atol: Absolute tolerance below which a cross-sectional scale counts as zero. An observation at or below it carries no dispersion, so its finite cells score zero rather than dividing by that scale.

Constructors

CrossSectionalStandardiser(; min_group_size::Integer = 8, atol::Real = 1e-12) -> CrossSectionalStandardiser

Keywords correspond to the struct's fields.

Validation

  • min_group_size >= 1.
  • isfinite(atol) and atol >= 0.

Examples

julia> CrossSectionalStandardiser()CrossSectionalStandardiser  min_group_size ┼ Int64: 8            atol ┴ Float64: 1.0e-12

Related

source
PortfolioOptimisers.CrossSectionalGaussianRank — Type
struct CrossSectionalGaussianRank{__T_min_group_size, __T_scale, __T_atol} <: AbstractCrossSectionalTransform

Scores every value of an observation by the inverse normal of its cross-sectional percentile rank.

The score depends on the order of the cross-section and not on the size of its gaps, so one extreme reading moves no other asset's score. The inverse normal spreads the ranks the way a normal sample would be spread, which is what makes the output usable as a factor exposure.

Mathematical definition

\[\begin{align} z_{t,i} &= \frac{\Phi^{-1}\left(p_{t,i}\right) - \mu_{t}}{\sigma_{t}}\,. \end{align}\]

Where:

  • $p_{t,i}$: Percentile rank of asset $i$ at observation $t$, as CrossSectionalPercentileRank defines it.
  • $\Phi^{-1}$: Inverse cumulative distribution function of the standard normal distribution.
  • $\mu_{t}$: Weighted centre of the inverse normals of observation $t$.
  • $\sigma_{t}$: Equal-weighted scale of the inverse normals of observation $t$, which is divided by only when scale is true.

The ranking is equal-weighted over the estimation set, and the recentring is weighted. The recentring and the rescaling always run over the whole observation, never inside a group. An observation whose scale is at or below atol scores zero after the recentring, and a NaN stays a NaN.

Fields

  • min_group_size: Smallest estimation set a group may carry and still be estimated from. A group below it, and every asset that carries no group, takes the whole observation's statistics instead.
  • scale: Whether to divide the recentred scores by their equal-weighted scale. Leave it false to feed a scale-invariant consumer, which then reads no dispersion noise from the estimate of that scale.
  • atol: Absolute tolerance below which a cross-sectional scale counts as zero. An observation at or below it carries no dispersion, so its finite cells score zero rather than dividing by that scale.

Constructors

CrossSectionalGaussianRank(; min_group_size::Integer = 8, scale::Bool = true, atol::Real = 1e-12) -> CrossSectionalGaussianRank

Keywords correspond to the struct's fields.

Validation

  • min_group_size >= 1.
  • isfinite(atol) and atol >= 0.

Examples

julia> CrossSectionalGaussianRank()CrossSectionalGaussianRank  min_group_size ┼ Int64: 8           scale ┼ Bool: true            atol ┴ Float64: 1.0e-12

Related

source
PortfolioOptimisers.CrossSectionalPercentileRank — Type
struct CrossSectionalPercentileRank{__T_min_group_size} <: AbstractCrossSectionalTransform

Scores every value of an observation by its percentile rank inside that observation's cross-section.

The rank is centred inside its own bin, so the score sits strictly inside the open unit interval and an inverse normal of it is always finite. A tie shares the average of the ranks its members would otherwise occupy.

Mathematical definition

\[\begin{align} p_{t,i} &= \mathrm{clamp}\left(\frac{\#\left\{j \in \mathcal{E}_{t} : x_{t,j} < x_{t,i}\right\} + \#\left\{j \in \mathcal{E}_{t} : x_{t,j} \le x_{t,i}\right\}}{2 \lvert \mathcal{E}_{t} \rvert},\, \frac{1}{2 \lvert \mathcal{E}_{t} \rvert},\, 1 - \frac{1}{2 \lvert \mathcal{E}_{t} \rvert}\right)\,. \end{align}\]

Where:

  • $x_{t,i}$: Value of asset $i$ at observation $t$.
  • $\mathcal{E}_{t}$: Estimation set of observation $t$.
  • $p_{t,i}$: Percentile rank of asset $i$ at observation $t$.

The ranking is equal-weighted over the estimation set, so a benchmark weight selects that set and nothing else. With group labels an asset is ranked inside its own group. A group holding fewer than min_group_size estimation assets, and every asset labelled CS_MISSING_GROUP, is ranked against the whole observation instead. A NaN stays a NaN.

Fields

  • min_group_size: Smallest estimation set a group may carry and still be estimated from. A group below it, and every asset that carries no group, takes the whole observation's statistics instead.

Constructors

CrossSectionalPercentileRank(; min_group_size::Integer = 8) -> CrossSectionalPercentileRank

Keywords correspond to the struct's fields.

Validation

  • min_group_size >= 1.

Examples

julia> CrossSectionalPercentileRank()CrossSectionalPercentileRank  min_group_size ┴ Int64: 8

Related

source

Functions

PortfolioOptimisers.cross_sectional_transform — Function
cross_sectional_transform(ct::CrossSectionalWinsoriser, X::MatNum; w::Option{<:MatNum} = nothing, groups::Option{<:AbstractMatrix{<:Integer}} = nothing)
cross_sectional_transform(ct::CrossSectionalTanhShrinker, X::MatNum; w::Option{<:MatNum} = nothing, groups::Option{<:AbstractMatrix{<:Integer}} = nothing)
cross_sectional_transform(ct::CrossSectionalStandardiser, X::MatNum; w::Option{<:MatNum} = nothing, groups::Option{<:AbstractMatrix{<:Integer}} = nothing)
cross_sectional_transform(ct::CrossSectionalGaussianRank, X::MatNum; w::Option{<:MatNum} = nothing, groups::Option{<:AbstractMatrix{<:Integer}} = nothing)
cross_sectional_transform(ct::CrossSectionalPercentileRank, X::MatNum; w::Option{<:MatNum} = nothing, groups::Option{<:AbstractMatrix{<:Integer}} = nothing)

Transform each observation of a matrix against the other assets of that same observation.

The benchmark weights and the group labels are arguments and never fields, because one transform runs against a different benchmark and a different classification at every call site, while the transform itself is one configuration.

Four rules are shared by every member:

  • The estimation set of an observation is its finite cells carrying a positive weight when w is given, and its finite cells otherwise. It is what the statistics are computed from.
  • A cell outside the estimation set is still transformed against it, so an asset the benchmark does not hold is scored on the same scale as one it does.
  • An observation whose estimation set is empty returns a missing marker at every asset, because it has nothing to be transformed against.
  • The two outlier members accept groups and ignore it: an outlier is extreme against the whole cross-section, not against a sector.

Algorithm

  1. Check X, w and groups, then promote them to one floating point element type.
  2. Build the finiteness mask and the estimation mask.
  3. Walk the observations, and apply the member's own map to each.

Arguments

  • ct: Cross-sectional transform.
  • X::MatNum: Data matrix observations × assets, where a NaN marks a missing cell.
  • w: Benchmark weight matrix observations × assets, or nothing.
  • groups: Group label matrix observations × assets, or nothing. CS_MISSING_GROUP labels an asset that carries no group.

Validation

  • !isempty(X), and every cell of X is finite or NaN.
  • size(w) == size(X), and every weight is finite and non-negative.
  • size(groups) == size(X), and every label is at least CS_MISSING_GROUP.

Returns

  • Y::Matrix{<:AbstractFloat}: Transformed matrix observations × assets, carrying a missing marker wherever X did.

Examples

julia> X = [1.0 NaN 3.0 4.0; 4.0 3.0 2.0 1.0];julia> cross_sectional_transform(CrossSectionalWinsoriser(; low = 0.1, high = 0.9), X)2×4 Matrix{Float64}: 1.4  NaN    3.0  3.8 3.7    3.0  2.0  1.3julia> cross_sectional_transform(CrossSectionalPercentileRank(), X)2×4 Matrix{Float64}: 0.166667  NaN      0.5    0.833333 0.875       0.625  0.375  0.125

Related

source
PortfolioOptimisers.cross_sectional_groups — Function
cross_sectional_groups(B::AbstractArray{<:Real, 3}) -> Matrix{Int}
cross_sectional_groups(pnl::AssetPanel, name::AbstractString) -> Matrix{Int}

Derive the group labels of a cross-sectional transform from a one-hot block, or from a categorical Panel Field of an Asset Panel.

Both forms return the observations × assets label matrix cross_sectional_transform takes, where a label is the position of a level in the level order and CS_MISSING_GROUP marks an asset with no level.

In the one-hot form, an asset carries a one in the column of the level it belongs to, and an asset whose row sets no level has no group. In the panel form, the codes of a CategoricalPanelField are the labels, and a cell its fill policy wrote (omsk is false) has no group: the fill resolved a blank so that the carrier holds no blank, and a fill value is not a membership, so the read undoes it, as OneHotExposure and panel_field_values do for the same cell.

Algorithm

One-hot form:

  1. Label every asset CS_MISSING_GROUP.
  2. Walk the levels of each asset, and label the asset with the first level it sets.

Panel form:

  1. Look the Panel Field up with panel_field, and check its kind and its shape.
  2. Copy its codes.
  3. Write CS_MISSING_GROUP into every cell whose observed mask is false, when the field carries one.

Arguments

  • B::AbstractArray{<:Real, 3}: One-hot block observations × assets × levels.
  • pnl::AssetPanel: Asset Panel holding the categorical Panel Field.
  • name::AbstractString: Name of the categorical Panel Field to read.

Validation

  • The named Panel Field is a CategoricalPanelField. Raises an ArgumentError.
  • The named Panel Field is time-varying. Raises a DimensionMismatch.

Returns

  • groups::Matrix{Int}: Group label matrix observations × assets. A label is the position of the level in the Panel Field's own level order, and CS_MISSING_GROUP marks an asset that sets none, or whose level was written by a fill policy.

Examples

julia> B = reshape([1.0, 0.0, 0.0, 1.0, 0.0, 0.0], 1, 2, 3)1×2×3 Array{Float64, 3}:[:, :, 1] = 1.0  0.0[:, :, 2] = 0.0  1.0[:, :, 3] = 0.0  0.0julia> cross_sectional_groups(B)1×2 Matrix{Int64}: 1  2

Related

source