Coverage Policy
By default a moment estimator fits on the assets whose returns are finite and active at every observation of the window, the coverage universe. Put a CoveragePolicy in the cvg field of the estimator, and it fits each entry of its result on the observations where every asset of that entry is finite and active. An asset with a short history then still gets an estimate. An asset whose share of the observations is below min_coverage is NaN in the result.
An AbstractCoverageAlgorithm decides what happens to a delisted asset. DecayCoverage keeps the history of a delisted asset, and drops the asset from the result as soon as it becomes inactive. ResetCoverage deletes its history, so a relisted asset starts again from zero. ExpireCoverage keeps the asset in the result for after more observations. During an incremental fit, fold_inactive! applies the rule when an asset becomes inactive, and admits decides whether the asset appears in the result when you read the estimate. The observation count of each entry is stored in a PortfolioOptimisers.CoverageCounts, which the state of the incremental fit carries.
Two admitted assets can each have a variance and share too few observations for a covariance. Such a pair is undetermined. An AbstractPeel in the peel field of the policy removes a set of assets that leaves no undetermined pair, and the covariance warns and names them, or refuses under strict = true. MinimalPeel removes the fewest assets, GreedyPeel removes the asset with the most undetermined pairs first, and peel = nothing skips the peel, so the matrix repair refuses the NaN. A rule of your own implements peel_assets.
PortfolioOptimisers.CoveragePolicy — Type
struct CoveragePolicy{__T_min_coverage, __T_alg, __T_peel} <: AbstractEstimatorFits each cell of a moment on the observations that cell has, instead of on the Coverage Universe.
A policy replaces the all-or-nothing Coverage Universe with available-case estimation. It goes in the cvg field of a moment estimator, where the default is nothing. With nothing the estimator takes the reduce-and-expand path by dispatch, and the policy costs nothing. With a policy set, the estimator fits every cell of the answer on the observations at which every asset of that cell is finite and active, and each cell carries its own denominator. An asset reaches the answer only where admits lets it in.
min_coverage is also the share that a scenario_fill passes in silence. admits reads the coverage share of an asset as its own observation count over the number of observations folded. The fill counts the non-finite entries of the column over the same denominator. So a column that DecayCoverage, ResetCoverage or ExpireCoverage admits is at most a 1 - min_coverage share non-finite. Where the fill_limit of a prior is nothing, resolve_fill_limit derives 1 - min_coverage, and no admitted column goes past it. A policy therefore turns on no warning, and an available-case walk-forward gives no warning for the gaps that it was set up to fill.
The default min_coverage = 0 admits every column, and a prior fills in silence every column that has a variance. Under the default Bessel correction that is a column with two or more observations. A column with one observation has a NaN variance, so the Investable Mask drops the asset and nothing is filled. To admit broadly and still get the warning, set the fill_limit of the prior explicitly, tighter than 1 - min_coverage.
A covariance peels the assets of an undetermined pair at admission. Two admitted assets can each have a variance and share too few observations for a covariance. The peel rule then removes a set of assets that leaves no such pair, and coverage_peel reports it through strict_diagnostic: a warning names the peeled assets, and strict = true refuses. A peeled asset is NaN across its row and column, so it leaves the Investable Mask, and the matrix repair meets no NaN inside the finite block. The default MinimalPeel keeps the most assets. GreedyPeel is the oracle's rule, and peel = nothing skips the peel and keeps the NaN for the matrix repair to refuse. A mean and a variance have no pairs, so the rule binds on a covariance and a correlation alone. On a fold, the peel runs each time the folded estimator answers a cov or a cor call, on the counts folded so far, and not in partial_fit!. So a walk-forward warns at each step that holds an undetermined pair, and the peeled assets can change from one step to the next.
Fields
min_coverage:min_coverage: Coverage floor, a share in[0, 1]. An asset whose own observation count over the number of observations folded falls below it isNaNin the answer. It is also the share that ascenario_fillfills in silence, so under the default of0a prior fills every column that has a variance and says nothing.
alg:alg: Coverage algorithm, the rule that decides what happens to a delisted asset.
peel:peel: Peel rule, which chooses the assets an available-case covariance removes so that no pair of admitted assets with a variance is left without a covariance, ornothingto skip the peel and leave theNaNof each undetermined pair for the matrix repair to refuse.
Constructors
CoveragePolicy(; min_coverage::Real = 0.0, alg::AbstractCoverageAlgorithm = DecayCoverage(), peel::Option{<:AbstractPeel} = MinimalPeel()) -> CoveragePolicyKeywords correspond to the struct's fields.
Validation
0 <= min_coverage <= 1.
Examples
julia> CoveragePolicy(; min_coverage = 0.25)CoveragePolicy min_coverage ┼ Float64: 0.25 alg ┼ DecayCoverage() peel ┴ MinimalPeel()Related
PortfolioOptimisers.AbstractCoverageAlgorithm — Type
abstract type AbstractCoverageAlgorithm <: AbstractAlgorithmAbstract supertype of the rules that decide what an available-case fit does with a delisted asset.
All concrete subtypes should subtype AbstractCoverageAlgorithm. A member of the family goes in the alg field of a CoveragePolicy. The family exists because a delisting has more than one correct treatment. One caller keeps the history of the asset and drops the asset at once. Another caller throws the history away, so that a relisting starts cold. A third holds the asset in the frame for a stated number of observations after it goes.
Interfaces
In order to implement a new concrete type that works seamlessly with the library, subtype AbstractCoverageAlgorithm and implement the following methods:
fold_inactive!
fold_inactive!(alg::AbstractCoverageAlgorithm, state::AbstractPartialFitState, ni::AbstractVector{<:Bool}) -> AbstractPartialFitState: The fold-time hook, called once per observation with the assets that have just become inactive. Only the arms that fold call it. The two-pass arms, which are the semi-covariance, the coskewness and the cokurtosis, read the window whole and never call it, so a new subtype keeps the history in those arms.
Arguments
alg: The concrete subtype instance.state: The partial-fit state being folded, mutated in place.ni: One entry per asset,truewhere the asset was active at the previous observation and is inactive at this one.
Returns
state::AbstractPartialFitState: The state to fold the observation into.
admits
admits(alg::AbstractCoverageAlgorithm, share::Real, active::Bool, stale::Integer, min_coverage::Real) -> Bool: The predicate that the estimate from the state calls once per asset.
Arguments
alg: The concrete subtype instance.share: The asset's coverage share, its own observation count over the number of observations folded.active: Whether the asset was active at the last observation.stale: The number of observations folded since the asset was last finite and active.min_coverage: The coverage floor of the policy.
Returns
admitted::Bool: Whether the asset appears in the answer, rather than asNaN.
Examples
julia> struct KeepEverything <: PortfolioOptimisers.AbstractCoverageAlgorithm endjulia> PortfolioOptimisers.admits(::KeepEverything, args...) = true;julia> PortfolioOptimisers.admits(KeepEverything(), 0.0, false, 99, 1.0)trueRelated
PortfolioOptimisers.DecayCoverage — Type
struct DecayCoverage <: AbstractCoverageAlgorithmKeeps a delisted asset's history, and drops the asset from the frame the moment it goes inactive.
This is the default rule. It does nothing at fold time, and it does not read the staleness when the estimate is made from the state. A relisting resumes the history. An asset that lists, delists and lists again folds into the counts and accumulators that it left behind, and does not start from zero. ExpireCoverage keeps the history in the same way, and it differs only in how long a delisted asset stays in the frame.
Related
PortfolioOptimisers.ResetCoverage — Type
struct ResetCoverage <: AbstractCoverageAlgorithmThrows a delisted asset's history away, so that a relisting starts the asset cold.
The reset zeroes the counts of the asset, and the row and the column of every accumulator that touches it. ExpWeightedCovariance resets an inactive asset in the same way. Use it when a relisted ticker is a different company, or when a corporate action makes the old series incomparable with the new one.
Every arm applies the reset. A folding arm applies it through fold_inactive! at the observation where the asset goes inactive, and a two-pass arm applies it to the whole window through coverage_inactive_block!. The reset needs an active mask. With no mask no asset goes inactive, a gap reads as a holiday, and the answer is the answer of DecayCoverage. The reset does not zero the number of observations folded, so the coverage share of a relisted asset is its count since the relisting over every observation folded.
Related
PortfolioOptimisers.ExpireCoverage — Type
struct ExpireCoverage{__T_after} <: AbstractCoverageAlgorithmHolds a delisted asset in the frame for a stated number of observations, then drops it.
The count is the staleness of the asset, the number of observations folded since the asset was last finite and active. A holiday raises the staleness as a delisting does, so a holiday just before a delisting shortens the hold. A finite return at an active observation sets the staleness back to zero. after = 0 admits exactly what DecayCoverage admits. An observation at which an asset is inactive raises its staleness, so no fold leaves an inactive asset at a staleness of zero.
Fields
after:after: Number of observations an inactive asset stays in the frame for, counted from the last observation at which it was finite and active.
Constructors
ExpireCoverage(; after::Integer = 0) -> ExpireCoverageKeywords correspond to the struct's fields.
Validation
after >= 0.
Examples
julia> ExpireCoverage(; after = 21)ExpireCoverage after ┴ Int64: 21Related
PortfolioOptimisers.AbstractPeel — Type
abstract type AbstractPeel <: AbstractAlgorithmAbstract supertype of the rules that choose which assets an available-case covariance peels away.
An available-case covariance divides each pair by the observations that the pair shares. Two admitted assets can each have a variance and share too few observations for a covariance, so their cell is NaN inside the finite block. Such a pair is undetermined. The undetermined pairs make a graph, with one vertex per admitted asset that has a variance and one edge per undetermined pair. A set of assets removes every undetermined pair exactly when it is a vertex cover of this graph. A member of the family goes in the peel field of a CoveragePolicy and chooses the cover, and nothing skips the peel. The peel runs at admission, so a peeled asset is NaN across its row and column and leaves the Investable Mask.
Interfaces
In order to implement a new concrete type that works seamlessly with the library, subtype AbstractPeel and implement the following method:
peel_assets
peel_assets(peel::AbstractPeel, U::AbstractMatrix{Bool}) -> Vector{Int}: The vertices to remove from the graph of undetermined pairs.
Arguments
peel: The concrete subtype instance.U: The adjacency matrix of the graph. It is symmetric,trueat each undetermined pair, andfalseon the diagonal.
Returns
idx::Vector{Int}: The vertices to remove, sorted. A set that does not cover every edge leaves aNaNinside the block, and the matrix repair refuses it with anIsNonFiniteError.
Examples
julia> struct PeelEveryEnd <: PortfolioOptimisers.AbstractPeel endjulia> PortfolioOptimisers.peel_assets(::PeelEveryEnd, U) = findall(vec(any(U; dims = 2)));julia> PortfolioOptimisers.peel_assets(PeelEveryEnd(), Bool[0 1 0; 1 0 0; 0 0 0])2-element Vector{Int64}: 1 2Related
PortfolioOptimisers.MinimalPeel — Type
struct MinimalPeel <: AbstractPeelPeels the fewest assets that remove every undetermined pair, and keeps the greedy peel whenever it is one of them.
This is the default rule. It solves the minimum vertex cover of the graph of undetermined pairs exactly, so it keeps the most assets that a peel can keep. When the cover of GreedyPeel has the minimum size, the rule returns it, so it gives the greedy answer on every graph where the greedy answer is optimal. Otherwise it returns the minimum cover whose sorted index vector is lexicographically smallest.
Mathematical definition
\[\begin{align} \mathcal{P} &= \underset{\substack{\mathcal{C} \subseteq \mathcal{V} \\ \forall (i, j) \in \mathcal{E},\ i \in \mathcal{C} \lor j \in \mathcal{C}}}{\operatorname{arg\,min}} \left\lvert \mathcal{C} \right\rvert\,. \end{align}\]
Where:
- $\mathcal{P}$: The assets the rule peels.
- $\mathcal{V}$: Vertices of the graph of undetermined pairs, the admitted assets that have a variance.
- $\mathcal{E}$: Edges of the graph of undetermined pairs, the pairs of vertices that share too few observations for a covariance.
- $\mathcal{C}$: A candidate set of assets to remove.
Algorithm
- Take the greedy cover
gwithGreedyPeel. - Find the minimum cover size
kwithpeel_cover_within. Start atlength(g)and lower the size while a smaller cover exists, so that only the size below the minimum needs a full search. Returngwhen no smaller cover exists. - Otherwise build the lexicographically smallest cover of size
kwithpeel_lexicographic.
The search is exact. Minimum vertex cover is NP-hard in general, but the search is exponential in the cover size alone, of order $1.47^{k}$ with $O(N^{2})$ work at each step. A tree, a path and a cycle need no branch at all. The graph holds only the pairs of admitted assets that share too few observations, so k is small in practice.
Related
PortfolioOptimisers.GreedyPeel — Type
struct GreedyPeel <: AbstractPeelPeels assets greedily, the asset with the most undetermined pairs first.
It is the oracle's rule. It removes the asset with the most undetermined pairs to the assets that are still in the graph, the smallest index on a tie, and repeats until no undetermined pair is left. The greedy cover can be larger than the minimum. On a star whose three leaves each carry one more pendant edge it removes four assets, where three suffice. MinimalPeel returns the minimum.
Algorithm
- Find the asset of the largest degree among the assets still in the graph, the smallest index on a tie, with
peel_max_degree. - Stop when that degree is zero. Otherwise remove the asset and go to step 1.
Related
PortfolioOptimisers.peel_assets — Function
peel_assets(peel, U)Returns the assets to remove from a graph of undetermined pairs.
It is the verb of the AbstractPeel interface. coverage_peel calls it once per available-case covariance, on the graph of the admitted assets that have a variance.
Arguments
peel: The peel rule, ornothingto skip the peel.U: The adjacency matrix of the graph. It is symmetric,trueat each undetermined pair, andfalseon the diagonal.
Returns
idx::Vector{Int}: The vertices to remove, sorted.
Examples
julia> U = falses(7, 7);julia> for (i, j) in ((1, 2), (1, 3), (1, 4), (2, 5), (3, 6), (4, 7)) U[i, j] = U[j, i] = true endjulia> PortfolioOptimisers.peel_assets(GreedyPeel(), U)4-element Vector{Int64}: 1 2 3 4julia> PortfolioOptimisers.peel_assets(MinimalPeel(), U)3-element Vector{Int64}: 2 3 4julia> PortfolioOptimisers.peel_assets(nothing, U)Int64[]Related
PortfolioOptimisers.peel_assets — Method
peel_assets(
_::Nothing,
_::AbstractMatrix{Bool}
) -> Vector{Int64}
nothing method of peel_assets. It skips the peel and removes no vertex.
An undetermined pair then stays NaN in the covariance, and the matrix repair refuses the NaN inside the finite block with an IsNonFiniteError that names the cell. Skip the peel to see the undetermined pairs, or to keep a covariance whose gaps a later step fills.
Related
PortfolioOptimisers.peel_assets — Method
peel_assets(
_::GreedyPeel,
U::AbstractMatrix{Bool}
) -> Vector{Int64}
GreedyPeel method of peel_assets. It removes the vertex of the largest degree, the smallest index on a tie, until no edge is left.
Related
PortfolioOptimisers.peel_assets — Method
peel_assets(
_::MinimalPeel,
U::AbstractMatrix{Bool}
) -> Vector{Int64}
MinimalPeel method of peel_assets. It returns the greedy cover when that cover has the minimum size, and the lexicographically smallest minimum cover otherwise.
Related
PortfolioOptimisers.fold_inactive! — Function
fold_inactive!(alg, state, ni)Applies a coverage algorithm's fold-time rule to the assets that have just become inactive.
It is the first of the two verbs of the AbstractCoverageAlgorithm interface. It runs once per observation, before the fold of that observation, so a rule that throws the history away runs while the state still holds that history. Only the folding arms call it. A two-pass arm applies the rule of a built-in algorithm to the whole window through coverage_inactive_block!.
Arguments
alg: The coverage algorithm.state: The partial-fit state, mutated in place.ni: One entry per asset,truewhere the asset was active at the previous observation and is inactive at this one.
Returns
state: The state to fold the observation into.
Related
PortfolioOptimisers.fold_inactive! — Method
fold_inactive!(
_::Union{DecayCoverage, ExpireCoverage},
state::AbstractPartialFitState,
_::AbstractVector{<:Bool}
) -> AbstractPartialFitState
DecayCoverage and ExpireCoverage method of fold_inactive!. Both keep the history of a delisted asset, so neither does anything at fold time, and the state passes through.
Related
PortfolioOptimisers.admits — Function
admits(alg, share, active, stale, min_coverage) -> BoolDecides whether an asset reaches the answer of an available-case fit.
It is the second of the two verbs of the AbstractCoverageAlgorithm interface. It runs once per asset when the estimate is made from the state, and the estimate holds NaN across every asset that admits refuses.
Arguments
alg: The coverage algorithm.share: The asset's coverage share, its own observation count over the number of observations folded.active: Whether the asset was active at the last observation.stale: The number of observations folded since the asset was last finite and active.min_coverage: The coverage floor of theCoveragePolicy.
Returns
admitted::Bool: Whether the asset reaches the answer.
Related
PortfolioOptimisers.admits — Method
admits(
_::Union{DecayCoverage, ResetCoverage},
share::Real,
active::Bool,
_::Integer,
min_coverage::Real
) -> Any
DecayCoverage and ResetCoverage method of admits. An asset is admitted while it is active and its coverage share reaches the floor. So both algorithms drop a delisted asset at once, and they differ only in the history they keep.
Mathematical definition
\[\begin{align} i \in \mathcal{A} &\iff a_{i} = 1 \land s_{i} \geq c\,. \end{align}\]
Where:
- $\mathcal{A}$: Admitted assets, those that
admitslets into the answer. A cell is admitted when every asset of the cell is in $\mathcal{A}$. - $a_{i}$: Active flag of asset $i$ at the last observation folded, $1$ when the asset is active and $0$ when it is not.
- $s_{i}$: Coverage share of asset $i$, its own observation count over the number of observations folded.
- $c$: Coverage floor of the policy, its
min_coverage.
Related
PortfolioOptimisers.admits — Method
admits(
alg::ExpireCoverage,
share::Real,
active::Bool,
stale::Integer,
min_coverage::Real
) -> Any
ExpireCoverage method of admits. An inactive asset stays in the frame while its staleness is at most after, and its coverage share must reach the floor in both cases.
Mathematical definition
\[\begin{align} i \in \mathcal{A} &\iff s_{i} \geq c \land \left(a_{i} = 1 \lor \tau_{i} \leq h\right)\,. \end{align}\]
Where:
- $\mathcal{A}$: Admitted assets, those that
admitslets into the answer. A cell is admitted when every asset of the cell is in $\mathcal{A}$. - $s_{i}$: Coverage share of asset $i$, its own observation count over the number of observations folded.
- $c$: Coverage floor of the policy, its
min_coverage. - $a_{i}$: Active flag of asset $i$ at the last observation folded, $1$ when the asset is active and $0$ when it is not.
- $\tau_{i}$: Staleness of asset $i$, the number of observations folded since the asset was last finite and active.
- $h$: The
afterfield of the algorithm.
Related