PortfolioOptimisersCovariance
PortfolioOptimisers.PortfolioOptimisersCovariance — Type
struct PortfolioOptimisersCovariance{__T_ce, __T_mp, __T_cache} <: AbstractCovarianceEstimatorRuns a covariance estimator, then applies a matrix processing step to its result.
ce computes the raw matrix. mp repairs or filters it with the positive definite repair, the denoise step and the detone step. The library uses this composite as its default covariance estimator.
The variance of the composite is the diagonal of its processed covariance, so each fit of var runs mp. variance_series fits var on the observations 1 to t for each row t. The fit of row 1 reads one observation, so its covariance is NaN. Posdef, Denoise and Detone refuse a NaN matrix, so the series raises ArgumentError: matrix contains Infs or NaNs under the default mp. A CrossSectionalFactorPrior whose ve is the composite reads this series, and so does an EWVolatility or an EWResidualVolatility whose ce is the composite. So both raise too, and so does each one with a WindowedCovariance over the composite, as WindowedCovariance() is. For a variance series, set no dn or dt, and remove the positive definite repair with mp = MatrixProcessing(; pdm = nothing) or change pdm to a step of your own that returns a non-finite matrix without change. The series then has NaN in row 1. With no step in mp, the inner estimator ce gives the same variance to rounding, so it can be the variance estimator in place of the composite.
Fields
ce: Covariance estimator.
mp: Matrix processing estimator.
cache: Optional partial-fit state. It isnothinguntilpartial_fit!writes one. When the caller gives no data matrix, the batch verb of the estimator reads it. Each propagation channel does one thing with it.factorykeeps it unchanged, because a factory call resolves the configuration, not the sample.port_opt_viewcopies the entries of the selected assets into a new state, so the viewed estimator holds the state of those assets alone.obs_weights_viewremoves it, because a state has no slice on the observation axis. Only a type with an observation-weights field has that channel. A type with no such field falls through to the identity method, so it keeps its state. A family whose state has no exact asset slice removes it on both axes and names the reason.
Constructors
PortfolioOptimisersCovariance(; ce::StatsBase.CovarianceEstimator = Covariance(), mp::AbstractMatrixProcessingEstimator = MatrixProcessing(), cache::Option{<:AbstractPartialFitState} = nothing) -> PortfolioOptimisersCovarianceKeywords correspond to the struct's fields.
Validation
ceis not anAbstractVarianceEstimator. A variance estimator has nocovorcor, soassert_covariance_estimatorraises anArgumentErrorthat namesce.DenoiseCovariance,DetoneCovarianceandProcessedCovariancebuild this type, so they refuse one too.
The incremental fit
The composite folds by composition. The state in cache selects one of two routes.
If cache holds nothing, partial_fit! passes the observation to ce.ce and keeps no state of its own. This route works because mp needs only a moment and a count. pdm and dt read only sigma, and dn reads the effective sample ratio T / N. The one-argument Statistics.cov reads the folded matrix of the inner estimator. Then it applies mp. When mp denoises, it reads the count from effective_observation_count, which is the same count that the batch fit reads. For an unweighted ce that count is size(X, 1). For a weighted ce it is the count of its weights. An mp.alg of the caller is the one step that needs the sample itself. The fold refuses it with an error that names it.
If cache holds a SampleBufferState, which Online seeds, the composite takes the buffer route. Then cov(ce) with no data runs the batch verb over the observations that the buffer kept. An mp.alg needs this route.
Propagated parameters
factory passes its arguments on to these @fprop fields:
View parameters
port_opt_view takes the view of these @vprop fields at the selected indices:
ce: Recursively viewed throughport_opt_view.cache: Sliced to the selected assets throughport_opt_view.
Examples
julia> PortfolioOptimisersCovariance()PortfolioOptimisersCovariance ce ┼ Covariance │ me ┼ SimpleExpectedReturns │ │ w ┴ nothing │ ce ┼ GeneralCovariance │ │ ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true) │ │ w ┴ nothing │ alg ┼ FullMoment() │ w ┴ nothing mp ┼ MatrixProcessing │ pdm ┼ Posdef │ │ alg ┼ UnionAll: NearestCorrelationMatrix.Newton │ │ kwargs ┴ @NamedTuple{}: NamedTuple() │ dn ┼ nothing │ dt ┼ nothing │ alg ┼ nothing │ order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg)Related
Statistics.cov — Method
Statistics.cov(ce::PortfolioOptimisersCovariance, X::MatNum; dims = 1,
active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing, kwargs...)Compute the covariance matrix with a PortfolioOptimisersCovariance estimator, and process the matrix.
The method computes the covariance matrix of X with the inner estimator ce.ce. Then it applies the matrix processing step ce.mp.
The composite does not change a gap. It passes X and active_mask to ce.ce without change, so its gap_fill_value is the gap_fill_value of ce.ce. A composite that wraps a gap-aware estimator does not refuse a sample that is not finite. It repairs the finite block and not the whole frame. The AssetPanel method of this file does the same.
Algorithm
- Check
dims, and orientXandactive_masktoobservations × assets. Transpose them whendims == 2. - Refuse a gapped sample with
assert_finite_samplewhengap_fill_valueonceis finite, because thence.ceis a plain estimator that cannot use a gapped sample. - Compute
sigmawithStatistics.cov(library_covariance_estimator(ce.ce), X; kwargs...). Passactive_masktoo if it is given. - When
sigmais immutable, copy it into aMatrix, because step 5 writes in place. - Take the effective number of observations
Twithprocessing_observation_count, with the keywords of step 3. Ameanof the caller then centres the data of the count and of the fit. - Apply
sample_matrix_processing!withce.mp,XandTtosigmain place. Ifce.ceis gap-aware, applymatrix_processing_block!instead, so that an asset outside the Coverage Universe keeps itsNaNrow and column. - Return
sigma.
ce.ce runs before ce.mp. ce.mp.order sets the order of the steps inside the processing. Step 1 orients X one time, so the estimator and the processing read the same orientation. Neither of them takes its own dims.
A weighted ce.ce is worth fewer observations than X has rows. So the denoising step reads the ratio T / N of the count of its weights, from effective_observation_count, and not of size(X, 1). An unweighted ce.ce gives T = size(X, 1), so its result does not change.
Arguments
ce: The composite covariance estimator with a processing step.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.dims: Dimension along which the computation runs.active_mask: The optional boolean matrix with the same size asX. The method passes it toce.ceif it is given.kwargs...: More keyword arguments for the inner covariance estimator and for the matrix processing step.
Validation
dims in (1, 2).
Returns
sigma::Matrix{<:Number}: The processed covariance matrix.
Related
Statistics.cor — Method
Statistics.cor(ce::PortfolioOptimisersCovariance, X::MatNum; dims = 1,
active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing, kwargs...)Compute the correlation matrix with a PortfolioOptimisersCovariance estimator, and process the matrix.
The method computes the correlation matrix of X with the inner estimator ce.ce. Then it applies the matrix processing step ce.mp.
Algorithm
- Check
dimsand orientXtoobservations × assets. Transpose it whendims == 2. - Compute
rhowithStatistics.cor(library_covariance_estimator(ce.ce), X; kwargs...). - When
rhois immutable, copy it into aMatrix, because step 4 writes in place. - Apply
matrix_processing_block!withce.mp,Xand the countTofprocessing_observation_counttorho, in place. When every diagonal entry ofrhois finite, it appliessample_matrix_processing!to the whole matrix. A gap-awarece.cegives aNaNrow for an asset outside the Coverage Universe, and an estimator that rescales a covariance, or that ranks, gives one for an asset with a zero variance, seecov2cor_nan!. The processing then runs on the other assets, and theNaNrow stays for the distance reader to refuse, seeassert_distance_reader_rows. - Return
rho.
ce.ce runs before ce.mp. ce.mp.order sets the order of the steps inside the processing. Step 1 orients X one time, so the estimator and the processing read the same orientation. Neither of them takes its own dims.
Arguments
ce: The composite covariance estimator with a processing step.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.dims: Dimension along which the computation runs.active_mask: The optional boolean matrix with the same size asX. The method passes it toce.ceif it is given.kwargs...: More keyword arguments for the inner covariance estimator and for the matrix processing step.
Validation
dims in (1, 2).
Returns
rho::Matrix{<:Number}: The processed correlation matrix.
Related
Statistics.cov — Method
Statistics.cov(ce::PortfolioOptimisersCovariance, X::MatNum,
pnl::Option{<:AssetPanel}; dims = 1, kwargs...) -> MatNum
Statistics.cor(ce::PortfolioOptimisersCovariance, X::MatNum,
pnl::Option{<:AssetPanel}; dims = 1, kwargs...) -> MatNumPass the Asset Panel to the inner estimator, then repair the finite block of the frame that it returns.
This method is the override of the composite for the reduce-and-expand root. The inner estimator does the reduction, because only the inner estimator knows if it is plain or mask-aware. This method does the repair. The repair runs on the finite block through matrix_processing_block!, so an asset outside the Coverage Universe keeps its NaN row and column, and no entry outside the finite block changes.
Algorithm
- Check
dimsand orientXtoobservations × assets. - Compute the matrix with the inner estimator. Pass
pnlas its third positional argument. - When the matrix is immutable, copy it into a
Matrix, because step 4 writes in place. - Repair its finite block with
matrix_processing_block!, withce.mp, the columns ofXthat the block names, and the countTofprocessing_observation_count. - Return the matrix.
Arguments
ce: The composite covariance estimator with a processing step.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.pnl: OptionalAssetPanel. The Coverage Universe of the fit comes from its active mask. If it isnothing, the rule reads only the finiteness of the returns.dims: Dimension along which the computation runs.kwargs...: More keyword arguments for the inner estimator and for the matrix processing step.
Validation
dims in (1, 2).
Returns
sigma::MatNum: The processed covariance matrix or correlation matrix, on the full asset universe.
Related
Statistics.var — Method
Statistics.var(ce::PortfolioOptimisersCovariance, X::MatNum,
pnl::Option{<:AssetPanel}; dims::Int = 1, kwargs...) -> MatNum
Statistics.std(ce::PortfolioOptimisersCovariance, X::MatNum,
pnl::Option{<:AssetPanel}; dims::Int = 1, kwargs...) -> MatNumTake the variance or the standard deviation of the composite over an Asset Panel from the diagonal of its processed covariance over the same panel.
The variance of the composite is the diagonal of its processed covariance, as the matrix method of var states. Over an Asset Panel, the covariance comes from the panel method of cov of the composite, which lets the inner estimator choose the plain route or the mask-aware route. So this method does the same, and mp runs on the fit. The reduce-and-expand root of var would reduce the window to its Coverage Universe whatever the inner estimator is. When the composite does not fold, by series_folds, variance_series over an Asset Panel fits this method at each row.
Algorithm
- Compute the processed covariance with the panel method of
Statistics.covof the composite, withdimsand the keywords. - Read its diagonal, and take its square root for
std. - Reshape it to a
1 × Nrow vector whendims == 1, and to anN × 1column vector otherwise.
Arguments
ce: The composite covariance estimator with a processing step.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.pnl: OptionalAssetPanel. The Coverage Universe of the fit comes from its active mask. If it isnothing, the rule reads only the finiteness of the returns.dims: Dimension along which the computation runs.kwargs...: More keyword arguments for the inner estimator and for the matrix processing step.
Validation
dims in (1, 2).
Returns
val::Matrix{<:Number}: The variance or the standard deviation of each asset,NaNoutside the assets that the inner estimator fits. Its shape is(1, N)ifdims == 1, or(N, 1)ifdims == 2.
Related
PortfolioOptimisers.supports_partial_fit — Method
supports_partial_fit(
ce::PortfolioOptimisersCovariance
) -> Any
Answers whether partial_fit! folds a PortfolioOptimisersCovariance.
The composite folds by composition, so it folds exactly when the inner estimator folds and mp runs from the shape of the sample, as supports_shape_processing states. A MatrixProcessing passes when it has no alg that reads the sample. A matrix processing estimator of the caller never passes. A composite that a wrapper gave a buffer folds in all cases. This is the default buffer route of supports_partial_fit, and it is the only route that lets an mp.alg run online.
Arguments
ce: Covariance estimator.
Returns
folds::Bool:truewhenpartial_fit!folds this composite.
Related
PortfolioOptimisers.partial_fit! — Method
partial_fit!(ce::PortfolioOptimisersCovariance{<:Any, <:Any, Nothing}, X::MatNum;
dims::Int = 1, kwargs...)
partial_fit!(ce::PortfolioOptimisersCovariance{<:Any, <:Any, Nothing}, x::VecNum;
kwargs...)Folds observations into a PortfolioOptimisersCovariance through ce.ce.
The composite is exact by composition and keeps no state of its own. ce.ce folds the raw matrix. ce.mp is a step of cov(ce) with no data, and it runs on the matrix that it gets. The composite has no accumulator, and its cache stays nothing on this route. The nothing in cache selects this route. A composite with a SampleBufferState got that state from Online, and it takes the buffer fold.
The step costs what the step of the inner estimator costs, which is O(N²). The composite adds no cost.
ce.mp needs only a moment and a count, with one exception. An mp.alg of the caller gets the whole sample. assert_shape_only_matrix_processing refuses it at the fold. The refusal cannot wait until cov(ce) with no data, because the rows are gone at that time.
Algorithm
- Refuse an
mpthat does not run from the shape of the sample, withassert_shape_only_matrix_processing. - Rebuild
cewith the estimator thatpartial_fit!returns ince.ce, throughrebuild_struct.
Arguments
ce: Covariance estimator.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.x: One observation, with one entry for each asset.dims: Dimension along which the computation runs.kwargs...: More keyword arguments force.ce.
Validation
ce.mpruns from the shape of the sample, asassert_shape_only_matrix_processingstates. AnArgumentErroris thrown otherwise.
Returns
ce: The composite. Itsce.ceis the estimator that holds the state after the last observation.
Related
Statistics.cov — Method
Statistics.cov(ce::PortfolioOptimisersCovariance{<:Any, <:Any, Nothing}; kwargs...)
Statistics.cor(ce::PortfolioOptimisersCovariance{<:Any, <:Any, Nothing}; kwargs...)Reads the covariance or the correlation that a folded PortfolioOptimisersCovariance holds.
This method makes the estimate from the fold by composition. It is cov(ce) or cor(ce) with no data. ce.ce returns its own folded matrix. Then the method applies ce.mp to that matrix from the shape of the sample. effective_observation_count reads the effective number of folded observations from the inner state. That number is the same count that the batch fit reads from processing_observation_count, so the substitution is an identity and not an approximation. This method returns the same result as a batch fit over the same observations. A windowed ce.ce under a SeedWindow counts the rows that its window kept on both routes.
The method reads the count only when matrix_processing_reads_count answers true, which is when ce.mp denoises. So the default mp, which only repairs the matrix, works over an inner state that records no count.
If the matrix of the inner estimator is immutable, the method copies it, because the processing writes in place. The matrix methods of this file do the same.
Algorithm
- Read the folded matrix of the inner estimator with the one-argument
Statistics.covorStatistics.cor. - Copy it into a
Matrixwhen it is immutable. - Take
Tfromeffective_observation_countwhenmatrix_processing_reads_countanswerstrue, andnothingotherwise. - Apply
ce.mpin place through the shape method ofmatrix_processing_block!, withTand withNfrom the matrix. The method uses the block form, because a fold over a changing universe returnsNaNfor an asset outside the Coverage Universe. TheAssetPanelmethods of this file do the same. A complete matrix costs nothing more. - Return the matrix.
Arguments
ce: Covariance estimator.kwargs...: More keyword arguments force.ce.
Validation
ce.ceholds a partial-fit state. AnArgumentErroris thrown otherwise.ce.mpruns from the shape of the sample, asassert_shape_only_matrix_processingstates. AnArgumentErroris thrown otherwise.- When
ce.mpreads the count, the state ofce.cerecords one, aseffective_observation_countstates. AnArgumentErroris thrown otherwise.
Returns
sigma::MatNum: The processed covariance matrix, or the processed correlation matrixrho.
Related
Statistics.cov — Method
Statistics.cov(ce::PortfolioOptimisersCovariance{<:Any, <:Any, <:SampleBufferState})
Statistics.cor(ce::PortfolioOptimisersCovariance{<:Any, <:Any, <:SampleBufferState})Reads the covariance or the correlation of a buffered PortfolioOptimisersCovariance from a new fit over its buffer.
This method makes the estimate on the buffer route. Online seeds this route, and an mp.alg of the caller needs it. The method runs the batch verb over the observations that the buffer kept, so it returns the same result as a batch fit over those rows, with mp.alg.
Arguments
ce: Covariance estimator.
Validation
ceholds aSampleBufferState. AnArgumentErroris thrown otherwise.
Returns
sigma::MatNum: The processed covariance matrix, or the processed correlation matrixrho.
Related