Cross-sectional transforms: private API

const CS_MISSING_GROUP = -1

The group label that says an asset carries no group at an observation.

A cell labelled this way never joins a group statistic, and it takes the whole-row statistics instead. Every other label is a group of its own, so the labels need not be contiguous and need not repeat between observations.

Related

• cross_sectional_transform
• cross_sectional_groups

const CS_MAD_CONSISTENCY = 1.4826022185056018

The factor that makes a median absolute deviation consistent with a standard deviation under normality.

It is the reciprocal of the third quartile of the standard normal distribution, so a normal sample's scaled median absolute deviation estimates the same quantity its standard deviation does.

Related

• CrossSectionalTanhShrinker

assert_cross_sectional_matrix(X::MatNum) -> nothing

Check that a cross-sectional data matrix is non-empty and holds no infinite cell.

A NaN is the marker for a missing cell, so it is admitted and preserved. An infinity is neither a value nor a marker: it survives a quantile, a median and a mean, and it turns every statistic of its observation into an infinity or a NaN without saying why.

Algorithm

1. Refuse an empty matrix.
2. Find the first cell that is neither finite nor NaN, and refuse it by name.

Arguments

• X: Data matrix observations × assets.

Validation

• !isempty(X). Raises an IsEmptyError.
• Every cell of X is finite or NaN. Raises a DomainError naming the observation and the asset.

Returns

• nothing.

Related

• cross_sectional_transform
• assert_cross_sectional_weights
• assert_cross_sectional_groups

assert_cross_sectional_weights(X::MatNum, w::Nothing) -> nothing
assert_cross_sectional_weights(X::MatNum, w::MatNum) -> nothing

Check a benchmark weight matrix against the data matrix it selects the estimation set of.

A weight is a selector first and a weight second: a positive weight puts the cell in the estimation set of its observation, and a zero weight leaves it out. A NaN weight therefore says neither, and a negative weight says less than nothing.

Algorithm

1. Return when w is nothing, because the estimation set is then the finite cells alone.
2. Check the shape against X, then the finiteness and the sign of every weight.

Arguments

• X: Data matrix observations × assets.
• w: Benchmark weight matrix observations × assets, or nothing.

Validation

• size(w) == size(X). Raises a DimensionMismatch.
• Every weight is finite. Raises an IsNonFiniteError.
• Every weight is non-negative. Raises a DomainError.

Returns

• nothing.

Related

• cross_sectional_transform
• assert_cross_sectional_matrix
• assert_cross_sectional_groups

assert_cross_sectional_groups(X::MatNum, groups::Nothing) -> nothing
assert_cross_sectional_groups(X::MatNum, groups::AbstractMatrix{<:Integer}) -> nothing

Check a group label matrix against the data matrix it partitions.

A label is an identity, not a quantity, so only CS_MISSING_GROUP carries a meaning of its own. A label below it names no group and no missing cell, so it would silently join a partition of its own.

Algorithm

1. Return when groups is nothing, because every cell then takes the whole-row statistics.
2. Check the shape against X, then that no label sits below CS_MISSING_GROUP.

Arguments

• X: Data matrix observations × assets.
• groups: Group label matrix observations × assets, or nothing.

Validation

• size(groups) == size(X). Raises a DimensionMismatch.
• Every label is at least CS_MISSING_GROUP. Raises a DomainError.

Returns

• nothing.

Related

• cross_sectional_transform
• cross_sectional_groups
• CS_MISSING_GROUP

cross_sectional_groups_observed!(G::AbstractMatrix{Int}, omsk::Nothing) -> nothing
cross_sectional_groups_observed!(G::AbstractMatrix{Int}, omsk::AbstractMatrix{Bool}) -> nothing

Write CS_MISSING_GROUP into every group label whose cell a fill policy wrote, in place.

A categorical Panel Field with no observed mask carried no blank, so there is nothing to undo and the method over nothing returns at once.

Arguments

• G: Group label matrix observations × assets, the copied codes of the Panel Field.
• omsk: The Panel Field's observed mask, or nothing.

Returns

• nothing.

Related

• cross_sectional_groups
• CS_MISSING_GROUP
• CategoricalPanelField

cross_sectional_estimation_mask(fin::AbstractMatrix{Bool}, w::Nothing) -> AbstractMatrix{Bool}
cross_sectional_estimation_mask(fin::AbstractMatrix{Bool}, w::MatNum) -> AbstractMatrix{Bool}

Return the estimation mask of every observation of a cross-sectional transform.

The estimation set of an observation is what its statistics are computed from. Without benchmark weights it is the finite cells, and with them it is the finite cells carrying a positive weight. A cell outside the set is still transformed against it, so the mask selects the estimator and never the output.

Algorithm

1. Return the finiteness mask when w is nothing.
2. Otherwise intersect it with the cells of positive weight.

Arguments

• fin::AbstractMatrix{Bool}: Finiteness mask observations × assets.
• w: Benchmark weight matrix observations × assets, or nothing.

Returns

• est::AbstractMatrix{Bool}: Estimation mask observations × assets. It is fin itself when w is nothing, and neither mask is written to.

Related

• cross_sectional_transform
• cross_sectional_indices

cross_sectional_indices(msk::AbstractMatrix{Bool}, t::Integer) -> Vector{Int}

Return the asset indices an observations × assets mask selects at one observation.

Arguments

• msk::AbstractMatrix{Bool}: Mask observations × assets.
• t::Integer: Observation.

Returns

• idx::Vector{Int}: Selected asset indices, in ascending order.

Examples

julia> PortfolioOptimisers.cross_sectional_indices([true false true; false true false], 1)2-element Vector{Int64}: 1 3

Related

• cross_sectional_estimation_mask
• cross_sectional_transform

cross_sectional_weight_type(w::Nothing) -> Type
cross_sectional_weight_type(w::MatNum) -> Type

Return the element type a benchmark weight matrix contributes to the output of a transform.

An absent weight matrix contributes Bool, which promotes against every numeric type without widening it, so the caller needs no branch of its own.

Arguments

• w: Benchmark weight matrix observations × assets, or nothing.

Returns

• T::Type: Bool when w is nothing, and eltype(w) otherwise.

Related

• cross_sectional_transform

cross_sectional_weighted_mean(A::AbstractMatrix, w::Nothing, t::Integer, idx::AbstractVector{<:Integer})
cross_sectional_weighted_mean(A::AbstractMatrix, w::MatNum, t::Integer, idx::AbstractVector{<:Integer})

Return the centre of one observation's estimation set.

The centre is weighted when benchmark weights are given, so a large benchmark holding pulls it towards itself, and it is the plain mean otherwise.

Arguments

• A::AbstractMatrix: Data matrix observations × assets, already floating point.
• w: Benchmark weight matrix observations × assets, or nothing.
• t::Integer: Observation.
• idx::AbstractVector{<:Integer}: Estimation set of the observation, which must not be empty.

Returns

• mu::Number: The centre, in the element type of A. It is zero when every weight of the estimation set is zero, which the estimation set's own definition excludes.

Related

• cross_sectional_equal_std
• cross_sectional_transform

cross_sectional_equal_std(A::AbstractMatrix, t::Integer, idx::AbstractVector{<:Integer}, mu::Number)

Return the equal-weighted dispersion of one observation's estimation set around a centre.

The scale is equal-weighted even where the centre is weighted, because a benchmark weight says how much of the market an asset is and not how precisely its value is measured. The divisor is the sample size less one.

Arguments

• A::AbstractMatrix: Data matrix observations × assets, already floating point.
• t::Integer: Observation.
• idx::AbstractVector{<:Integer}: Estimation set of the observation.
• mu::Number: Centre to measure the dispersion around.

Returns

• sigma::Number: The dispersion, in the element type of A. It is zero when the estimation set holds fewer than two assets, and every consumer then reads the cell as having no dispersion rather than an undefined one.

Related

• cross_sectional_weighted_mean
• cross_sectional_transform

cross_sectional_stat(v::AbstractVector, i::Integer) -> Number
cross_sectional_stat(v::Number, i::Integer) -> Number

Read the statistic that applies to one asset, whether it is per asset or shared by the observation.

An ungrouped transform holds one centre and one scale for the whole observation, and a grouped one holds a pair per asset. This reader lets one scoring loop serve both.

Arguments

• v: Statistic, either one number per asset or one number for the observation.
• i::Integer: Asset.

Returns

• val::Number: The statistic of asset i.

Related

• cross_sectional_zscore_row!

cross_sectional_blank_row!(Y::AbstractMatrix, t::Integer) -> nothing

Write a missing marker into every asset of one observation.

An observation whose estimation set is empty has nothing to be transformed against, so no cell of it carries a value, not even a cell that was finite.

Arguments

• Y::AbstractMatrix: Output matrix observations × assets.
• t::Integer: Observation.

Returns

• nothing.

Related

• cross_sectional_transform

cross_sectional_zscore_row!(Y::AbstractMatrix, A::AbstractMatrix, fin::AbstractMatrix{Bool}, t::Integer, idx::AbstractVector{<:Integer}, mu, sigma, atol::Real) -> nothing

Score one observation against a centre and a scale.

An observation with no dispersion scores every cell zero rather than dividing by it, so a cross-section that carries the same value everywhere reads as a neutral exposure instead of a missing one.

Algorithm

1. Blank the whole observation when its estimation set is empty.
2. Otherwise write a missing marker at every non-finite cell.
3. Write the centred and rescaled value where the scale is above atol, and zero where it is not.

Arguments

• Y::AbstractMatrix: Output matrix observations × assets.
• A::AbstractMatrix: Data matrix observations × assets, already floating point.
• fin::AbstractMatrix{Bool}: Finiteness mask observations × assets.
• t::Integer: Observation.
• idx::AbstractVector{<:Integer}: Estimation set of the observation.
• mu: Centre, one number for the observation or one per asset.
• sigma: Scale, one number for the observation or one per asset.
• atol::Real: atol: Absolute tolerance below which a cross-sectional scale counts as zero. An observation at or below it carries no dispersion, so its finite cells score zero rather than dividing by that scale.

Returns

• nothing.

Related

• cross_sectional_stat
• CrossSectionalStandardiser

cross_sectional_recentre_rescale!(Y::AbstractMatrix, fin::AbstractMatrix{Bool}, est::AbstractMatrix{Bool}, w::Option{<:MatNum}, atol::Real, scale::Bool) -> nothing

Recentre, and optionally rescale, an already scored matrix over the whole cross-section.

A grouped score is comparable inside its group and not between groups, so the score is brought back to a weighted centre of zero over the whole observation, and to a unit equal-weighted scale when the caller asks for one.

Algorithm

1. Blank an observation whose estimation set is empty.
2. Subtract the weighted centre of the estimation set from every finite cell.
3. Return when scale is false.
4. Divide by the equal-weighted scale of the estimation set, writing zero where that scale is at or below atol.

Arguments

• Y::AbstractMatrix: Score matrix observations × assets, written in place.
• fin::AbstractMatrix{Bool}: Finiteness mask observations × assets of the matrix the score came from.
• est::AbstractMatrix{Bool}: Estimation mask observations × assets.
• w: Benchmark weight matrix observations × assets, or nothing.
• atol::Real: atol: Absolute tolerance below which a cross-sectional scale counts as zero. An observation at or below it carries no dispersion, so its finite cells score zero rather than dividing by that scale.
• scale::Bool: Whether to divide by the equal-weighted scale.

Returns

• nothing.

Related

• CrossSectionalStandardiser
• CrossSectionalGaussianRank

cross_sectional_midranks!(P::AbstractMatrix, A::AbstractMatrix, t::Integer, idx::AbstractVector{<:Integer}, qry::AbstractVector{<:Integer}) -> nothing

Write the percentile rank of each queried asset against one estimation set.

A tie shares the average of the ranks its members would otherwise occupy, and the rank is centred inside its bin, so a percentile sits strictly inside the open unit interval and an inverse normal of it is always finite.

Algorithm

1. Write a missing marker at every queried asset when the estimation set is empty.
2. Otherwise sort the estimation values once.
3. For each queried asset, count the estimation values strictly below it and those at or below it, and average the two counts.
4. Divide by the size of the estimation set, and clamp into the closed interval between half a bin and one less half a bin.

Arguments

• P::AbstractMatrix: Percentile matrix observations × assets, written in place.
• A::AbstractMatrix: Data matrix observations × assets, already floating point.
• t::Integer: Observation.
• idx::AbstractVector{<:Integer}: Estimation set to rank against.
• qry::AbstractVector{<:Integer}: Assets to rank.

Returns

• nothing.

Related

• CrossSectionalPercentileRank
• cross_sectional_percentile_ranks

cross_sectional_rank_counts(v::AbstractVector, x::Number) -> Tuple{Int, Int}

Return how many entries of a sorted vector sit below a value, and how many sit at or below it.

The pair is what a midrank needs: their average is the rank a tie shares. Two binary searches read it in logarithmic time, and the second starts where the first stopped, because a value at or below x is never below one that is below it.

The searches are written out rather than taken from Base, because a loaded dependency adds its own methods to searchsortedfirst and searchsortedlast, and the static analysis gate reads those arms and refuses the call.

Algorithm

1. Binary search for the largest prefix of v whose entries are all below x.
2. Binary search again, from that prefix to the end, for the largest prefix whose entries are all at or below x.

Arguments

• v::AbstractVector: Values to count against, sorted in ascending order.
• x::Number: Value to count around.

Returns

• nlt::Int: Number of entries strictly below x.
• nle::Int: Number of entries at or below x.

Examples

julia> PortfolioOptimisers.cross_sectional_rank_counts([1.0, 2.0, 2.0, 3.0], 2.0)(1, 3)julia> PortfolioOptimisers.cross_sectional_rank_counts([1.0, 2.0, 2.0, 3.0], 0.5)(0, 0)

Related

• cross_sectional_midranks!
• CrossSectionalPercentileRank

cross_sectional_row_groups(est::AbstractMatrix{Bool}, groups::AbstractMatrix{<:Integer}, t::Integer) -> Dict{Int, Vector{Int}}

Return the estimation set of each group of one observation.

A cell labelled CS_MISSING_GROUP joins no group, so it contributes to no group statistic while still belonging to the observation's own estimation set.

Arguments

• est::AbstractMatrix{Bool}: Estimation mask observations × assets.
• groups::AbstractMatrix{<:Integer}: Group label matrix observations × assets.
• t::Integer: Observation.

Returns

• gidx::Dict{Int, Vector{Int}}: Estimation asset indices of each label present at the observation, in ascending order.

Related

• cross_sectional_transform
• CS_MISSING_GROUP

cross_sectional_group_split(fin::AbstractMatrix{Bool}, groups::AbstractMatrix{<:Integer}, t::Integer, gidx::Dict{Int, Vector{Int}}, mgs::Integer)

Split the finite assets of one observation into the groups that stand and the ones that fall back.

A group stands when its own estimation set is large enough to estimate from. Every other finite asset takes the whole-row statistics, which is what the fallback list carries.

Algorithm

1. Walk the finite assets of the observation.
2. Send an asset whose label is CS_MISSING_GROUP, or whose group holds fewer than mgs estimation assets, to the fallback list.
3. Send every other asset to the query list of its own group.

Arguments

• fin::AbstractMatrix{Bool}: Finiteness mask observations × assets.
• groups::AbstractMatrix{<:Integer}: Group label matrix observations × assets.
• t::Integer: Observation.
• gidx::Dict{Int, Vector{Int}}: Estimation set of each group of the observation.
• mgs::Integer: min_group_size: Smallest estimation set a group may carry and still be estimated from. A group below it, and every asset that carries no group, takes the whole observation's statistics instead.

Returns

• qry::Dict{Int, Vector{Int}}: Finite assets of each group that stands.
• fb::Vector{Int}: Finite assets that take the whole-row statistics.

Related

• cross_sectional_row_groups
• cross_sectional_transform

cross_sectional_percentile_ranks(A::AbstractMatrix, fin::AbstractMatrix{Bool}, est::AbstractMatrix{Bool}, groups::Nothing, mgs::Integer)
cross_sectional_percentile_ranks(A::AbstractMatrix, fin::AbstractMatrix{Bool}, est::AbstractMatrix{Bool}, groups::AbstractMatrix{<:Integer}, mgs::Integer)

Return the percentile rank of every finite cell of a matrix, against its own cross-section.

Without group labels a cell is ranked against the whole estimation set of its observation. With them it is ranked inside its own group, and it falls back to the whole estimation set when its group is too small to rank inside.

Algorithm

1. Walk the observations.
2. Rank every finite asset against the estimation set of the observation when no labels are given.
3. Otherwise split the finite assets into the groups that stand and the ones that fall back, rank each standing group against its own estimation set, and rank the fallback list against the whole estimation set.

Arguments

• A::AbstractMatrix: Data matrix observations × assets, already floating point.
• fin::AbstractMatrix{Bool}: Finiteness mask observations × assets.
• est::AbstractMatrix{Bool}: Estimation mask observations × assets.
• groups: Group label matrix observations × assets, or nothing.
• mgs::Integer: min_group_size: Smallest estimation set a group may carry and still be estimated from. A group below it, and every asset that carries no group, takes the whole observation's statistics instead.

Returns

• P::Matrix{<:AbstractFloat}: Percentile matrix observations × assets, carrying a missing marker at every cell that is not finite and at every cell whose estimation set is empty.

Related

• CrossSectionalPercentileRank
• CrossSectionalGaussianRank
• cross_sectional_midranks!

cross_sectional_standardise!(Y::AbstractMatrix, A::AbstractMatrix, fin::AbstractMatrix{Bool}, est::AbstractMatrix{Bool}, w::Option{<:MatNum}, groups::Nothing, mgs::Integer, atol::Real) -> nothing
cross_sectional_standardise!(Y::AbstractMatrix, A::AbstractMatrix, fin::AbstractMatrix{Bool}, est::AbstractMatrix{Bool}, w::Option{<:MatNum}, groups::AbstractMatrix{<:Integer}, mgs::Integer, atol::Real) -> nothing

Score a matrix as cross-sectional z-scores, with or without groups.

Without group labels one centre and one scale serve the whole observation. With them each group that stands carries its own pair, every other asset takes the observation's pair, and the scores are recentred and rescaled over the whole observation afterwards.

Algorithm

1. Walk the observations, and take the estimation set of each.
2. Compute the observation's own centre and scale.
3. Without group labels, score every finite cell against that pair and stop.
4. With them, compute the pair of each group that stands, give every other finite cell the observation's pair, score, and then recentre and rescale over the whole observation.

Arguments

• Y::AbstractMatrix: Output matrix observations × assets.
• A::AbstractMatrix: Data matrix observations × assets, already floating point.
• fin::AbstractMatrix{Bool}: Finiteness mask observations × assets.
• est::AbstractMatrix{Bool}: Estimation mask observations × assets.
• w: Benchmark weight matrix observations × assets, or nothing.
• groups: Group label matrix observations × assets, or nothing.
• mgs::Integer: min_group_size: Smallest estimation set a group may carry and still be estimated from. A group below it, and every asset that carries no group, takes the whole observation's statistics instead.
• atol::Real: atol: Absolute tolerance below which a cross-sectional scale counts as zero. An observation at or below it carries no dispersion, so its finite cells score zero rather than dividing by that scale.

Returns

• nothing.

Related

• CrossSectionalStandardiser
• cross_sectional_zscore_row!
• cross_sectional_recentre_rescale!

cross_sectional_cell_stats(A::AbstractMatrix, w::Option{<:MatNum}, t::Integer, groups::AbstractMatrix{<:Integer}, gidx::Dict{Int, Vector{Int}}, mgs::Integer, mu, sigma)

Return the centre and the scale that apply to each asset of one grouped observation.

An asset whose group stands takes its group's pair, and every other asset takes the observation's own pair. Building both vectors first keeps the scoring loop free of the fallback rule.

Algorithm

1. Compute the pair of each group holding at least mgs estimation assets.
2. Fill both vectors with the observation's pair.
3. Overwrite the entries of every asset whose group stands.

Arguments

• A::AbstractMatrix: Data matrix observations × assets, already floating point.
• w: Benchmark weight matrix observations × assets, or nothing.
• t::Integer: Observation.
• groups::AbstractMatrix{<:Integer}: Group label matrix observations × assets.
• gidx::Dict{Int, Vector{Int}}: Estimation set of each group of the observation.
• mgs::Integer: min_group_size: Smallest estimation set a group may carry and still be estimated from. A group below it, and every asset that carries no group, takes the whole observation's statistics instead.
• mu: Centre of the whole observation.
• sigma: Scale of the whole observation.

Returns

• M::Vector{<:Number}: Centre of each asset.
• S::Vector{<:Number}: Scale of each asset.

Related

• cross_sectional_standardise!
• cross_sectional_row_groups