PortfolioOptimisersCovariance: private API
PortfolioOptimisers.find_uncorrelated_indices — Function
find_uncorrelated_indices(X::MatNum;
ce::StatsBase.CovarianceEstimator = PortfolioOptimisersCovariance(),
t::Number = 0.95, absolute::Bool = false,
measure::Num_VecToScaM = MeanValue(),
scores::Option{<:VecNum} = nothing)Find the indices of a subset of the assets of a data matrix in which no pair has a correlation of t or more.
The function compares each pair of columns of X. If absolute is true, it compares the absolute value of the correlation. When two assets are too correlated, the function removes the asset with the higher drop score. It returns the indices of the assets that remain.
By default, the drop score of an asset is the summary of the finite entries of its column of the correlation matrix, so the asset that is redundant with the largest part of the universe goes first. The scores keyword replaces that criterion, and a higher score removes an asset first. RedundancySelector uses scores to keep, from each correlated pair, the asset with the better score under a risk measure.
This function is internal. A caller uses RedundancySelector with a PairwiseCorrelation algorithm.
Algorithm
- Compute the correlation matrix
rhowithce, and take its absolute value whenabsoluteistrue. - When
scoresisnothing, collapse the finite entries of each column ofrhowithmeasureintosummary_rho, the default drop score, throughfinite_vec_to_real_measure. The correlation with a constant column is not defined. Acethat returns it asNaN, such asCovariance(), puts oneNaNin each column for each constant column, and the summary skips it. Otherwise takesummary_rhofromscores. - Read the strict lower triangle of
rhointotril_idx, so that each pair occurs one time. - Keep the pairs whose correlation is at least
t, and sort them from the most to the least correlated. - When
scoresis given, remove every asset whose score is not finite. ANaNscore marks an asset that has no score, and that asset never survives. Step 6 then skips each pair of that asset, so its partner survives that pair. Two non-finite scores in one pair remove both assets. The default drop score of step 2 does not take this step. - Go through that list in order. For a pair whose two assets are both still present, remove the asset with the higher drop score. When the two scores are equal, remove both assets. This is the tie policy of the library: if the function cannot tell two assets apart, it keeps neither of them. Two identical columns leave no survivor.
- Return the indices that steps 5 and 6 did not remove, in ascending order.
Step 6 skips a pair in which one asset or both assets are already removed. Thus the result depends on the order that step 4 sets. The greedy pass does not return a removed asset to the subset, so the result is not always the largest such subset. A tie can also remove an asset that the subset could keep.
Arguments
X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.ce: Covariance estimator.t: The correlation threshold. Two assets with a correlation oftor more are too correlated.absolute: Iftrue, the function compares the absolute value of the correlation.measure: The summary measure that the function applies to the finite entries of each column of the correlation matrix, for example the mean. The result is the default drop score. The function ignoresmeasureifscoresis given.scores: The drop score of each asset. From a correlated pair, the function removes the asset with the higher score. The function removes an asset with a non-finite score in every case, also when the asset is in no correlated pair.
Validation
- If
scoresis notnothing,length(scores) == size(X, 2). ADimensionMismatchis thrown otherwise.
Returns
idx::Vector{Int}: The indices of the assets that remain, in ascending order. No pair of these assets has a correlation oftor more.
Related
PortfolioOptimisers.gap_fill_value — Method
gap_fill_value(ce::PortfolioOptimisersCovariance) -> NumberReturn the gap fill value of ce.ce, because the composite passes the sample and the mask to ce.ce without change.
The composite adds matrix processing to an inner estimator and reads no cell of the sample, so the inner estimator alone decides what a gap does to the result. A composite that wraps a gap-aware estimator is gap-aware. A composite that wraps a plain estimator is not.
Arguments
ce: The composite covariance estimator with a processing step.
Returns
fv::Number:gap_fill_valueofce.ce.
Related
PortfolioOptimisers.show_fields — Method
show_fields(
_::PortfolioOptimisersCovariance
) -> Tuple{Symbol, Symbol}
Renders every field of a PortfolioOptimisersCovariance except cache.
A cache holds the state of an incremental fit. That state is not part of the configuration, and it prints under the estimator at each site that renders the estimator. Set set_show_nothing_fields!(:PortfolioOptimisersCovariance, true) to render it.
Arguments
::PortfolioOptimisersCovariance: The covariance estimator. The method reads only its type.
Returns
fields::Tuple: The field names to render, which is(:ce, :mp).
Related
PortfolioOptimisers.observation_count — Method
observation_count(
ce::PortfolioOptimisersCovariance{<:Any, <:Any, Nothing}
) -> Any
PortfolioOptimisersCovariance method of observation_count, for a composite whose own cache is nothing.
The composite then folds through ce.ce and keeps no state of its own, so the method reads the count of ce.ce. A composite whose cache holds a SampleBufferState reads the count of its buffer through the generic method instead.
Arguments
ce: Covariance estimator.
Validation
ce.ceholds a partial-fit state with a count, asobservation_countstates. AnArgumentErroris thrown otherwise.
Returns
n::Integer: The number of observations thatce.cefolded.
Related
PortfolioOptimisers.processing_observation_count — Function
processing_observation_count(
ce::PortfolioOptimisersCovariance,
X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}};
kwargs...
) -> Any
Return the number of observations that the matrix processing of a PortfolioOptimisersCovariance reads in a batch fit.
Only the denoising step reads the count, so the function reads effective_observation_count of ce.ce only when matrix_processing_reads_count answers true. Otherwise it returns size(X, 1), which no step reads. A windowed ce.ce then makes no copy of its window, and the count costs nothing when no step needs it. The fold route reads its count by the same rule.
Arguments
ce: Covariance estimator.X: The data matrix, oriented toobservations × assets.kwargs...: The keywords that the fit gives toce.ce,active_masktoo when the caller gives it. The function passes them toeffective_observation_count.
Returns
T::Real: The effective number of observations ofce.ce, orsize(X, 1).
Related
PortfolioOptimisers.series_folds — Function
series_folds(ce::AbstractCovarianceEstimator) -> Bool
series_folds(ce::Covariance{<:Any, <:Any, <:FullMoment, <:Any, <:CoveragePolicy}) -> Bool
series_folds(ce::Union{ExpWeightedCovariance, RegimeAdjustedExpWeightedCovariance}) -> Bool
series_folds(ce::PortfolioOptimisersCovariance) -> BoolAnswer whether a variance series of ce can fold the observations one at a time, where the read after observation t equals the batch fit over the observations 1 to t.
A variance series of a PortfolioOptimisersCovariance fits the processed covariance at each row. A refit reads the observations 1 to t again at each row t, so the series costs a time that grows as the square of the number of observations. A fold reads each observation one time, so fold_variance_series costs a time that grows with it. The fold is a route only where it gives the series of the refit.
The fallback answers false. A mask-aware covariance estimator whose fold over the active mask is its batch fit, and that holds no state yet, answers true with a method in its own file. These are a Covariance with a FullMoment, a CoveragePolicy and no observation weights w, which partial_fit! refuses, an ExpWeightedCovariance and a RegimeAdjustedExpWeightedCovariance. A windowed estimator is not one: its fold keeps every observation after its first fold, and a row of its series fits the window alone. The composite answers true when it holds no state, when its mp runs from the shape of the sample, as supports_shape_processing states, and when its inner estimator answers true.
Arguments
ce: Covariance estimator.
Returns
flag::Bool:truewhen the series can fold.
Related
PortfolioOptimisers.fold_variance_series — Function
fold_variance_series(
ce::PortfolioOptimisersCovariance,
X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
amsk::Union{Nothing, AbstractMatrix{<:Bool}},
emsk::Union{Nothing, AbstractMatrix{<:Bool}};
kwargs...
) -> Any
Compute the variance series of a PortfolioOptimisersCovariance in one pass, with partial_fit! at each observation.
series_folds answers true for ce, so the read of the fold after observation t is the batch fit over the observations 1 to t. The one-argument Statistics.cov of the composite applies mp to the folded matrix at each row, so each row runs mp, as the refit does.
Algorithm
- Allocate the series with the element type of
X. - For each observation
t, fold rowtofXinto the composite, with rowtof the masks thatfold_mask_keywordsgives for the inner estimator. - Read the processed covariance with
Statistics.covof the folded composite, and write its diagonal into rowt. - Return the series.
Arguments
ce: The composite covariance estimator with a processing step, which holds no state.X: The data matrix, oriented toobservations × assets.amsk: The active mask,observations × assets, ornothing.emsk: The estimation mask,observations × assets, ornothing.kwargs...: More keyword arguments for the fold of the inner estimator.
Returns
val::Matrix{<:Number}: The variance series,observations × assets.
Related
PortfolioOptimisers.fold_mask_keywords — Function
fold_mask_keywords(ce::AbstractCovarianceEstimator, amsk, emsk) -> NamedTuple
fold_mask_keywords(ce::RegimeAdjustedExpWeightedCovariance, amsk, emsk) -> NamedTupleGive the mask keywords of one fold of ce, the masks that the panel method of its cov reads.
The panel method of cov of most mask-aware estimators reads the active mask alone, and the fold of Covariance takes no other keyword. The regime-adjusted estimator reads the estimation mask too. So fold_variance_series gives each fold the masks of its batch fit.
Arguments
ce: Covariance estimator.amsk: The active mask of the fold, ornothing.emsk: The estimation mask of the fold, ornothing.
Returns
kw::NamedTuple: The keywords of the fold.
Related
PortfolioOptimisers.variance_series — Method
variance_series(ce::PortfolioOptimisersCovariance, X::MatNum; dims::Int = 1, kwargs...)
variance_series(ce::PortfolioOptimisersCovariance, X::MatNum, pnl::Option{<:AssetPanel};
dims::Int = 1, kwargs...)Compute the point-in-time variance series of a PortfolioOptimisersCovariance, the diagonal of its processed covariance over the observations 1 to t at each row t.
When series_folds answers true, the series folds each observation one time with fold_variance_series. It reads the masks from the keywords active_mask and estimation_mask, or from the Asset Panel. Otherwise the series refits the composite at each row. The matrix method takes the expanding series of variance_series(est, w::Option{<:ObsWeights}, window::Option{<:Int_VecInt}, X::MatNum, pnl::Option{<:AssetPanel}). The panel method fits the panel method of Statistics.var of the composite at each row with panel_series_row, so the inner estimator chooses its route: a mask-aware one reads the window, and a plain one reduces it to its Coverage Universe. A row whose window covers no asset of a plain inner estimator is NaN. When the inner estimator is a plain leaf, by reduces_each_window, the panel method takes the reduce-and-expand series of coverage_variance_series in place of the refit, because the two give the same values and the refit costs more for each row.
The reduce-and-expand series of variance_series(ce::AbstractCovarianceEstimator, X::MatNum, pnl::Option{<:AssetPanel}; dims::Int = 1, kwargs...) would reduce each window whatever the inner estimator is, so the composite overrides it. The estimators that are not composites keep their own methods, and they do no work of this method.
Algorithm
- Orient
X, and the masks, toobservations × assetswithdims_oriented. The panel method allocates theNaNframe withcoverage_series_frame, which checks the size of the active mask. - If
series_foldsanswerstrue, compute the series withfold_variance_series. - Otherwise refit. The matrix method returns the expanding windowed series. The panel method returns the series of
coverage_variance_serieswhenreduces_each_windowanswerstruefor the inner estimator, and otherwise it writes the row ofpanel_series_rowon the rows1totinto each rowt. - Return the series, transposed when
dims == 2.
Arguments
ce: The composite covariance estimator with a processing step.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.pnl: OptionalAssetPanel. The Coverage Universe of the fit comes from its active mask. If it isnothing, the rule reads only the finiteness of the returns.dims: Dimension along which the computation runs.kwargs...: More keyword arguments for the inner estimator, which can carryactive_maskandestimation_maskin the matrix method.
Validation
dims in (1, 2).- The active mask of
pnlmust have the size of the orientedX.
Returns
val::Matrix{<:Number}: The variance series, shaped as(T, N)ifdims == 1or(N, T)ifdims == 2.
Related