PortfolioOptimisersCovariance

PortfolioOptimisers.PortfolioOptimisersCovariance — Type
struct PortfolioOptimisersCovariance{__T_ce, __T_mp, __T_cache} <: AbstractCovarianceEstimator

Runs a covariance estimator, then applies a matrix processing step to its result.

ce computes the raw matrix. mp repairs or filters it with the positive definite repair, the denoise step and the detone step. The library uses this composite as its default covariance estimator.

Warning

The variance of the composite is the diagonal of its processed covariance, so each fit of var runs mp. variance_series fits var on the observations 1 to t for each row t. The fit of row 1 reads one observation, so its covariance is NaN. Posdef, Denoise and Detone refuse a NaN matrix, so the series raises ArgumentError: matrix contains Infs or NaNs under the default mp. A CrossSectionalFactorPrior whose ve is the composite reads this series, and so does an EWVolatility or an EWResidualVolatility whose ce is the composite. So both raise too, and so does each one with a WindowedCovariance over the composite, as WindowedCovariance() is. For a variance series, set no dn or dt, and remove the positive definite repair with mp = MatrixProcessing(; pdm = nothing) or change pdm to a step of your own that returns a non-finite matrix without change. The series then has NaN in row 1. With no step in mp, the inner estimator ce gives the same variance to rounding, so it can be the variance estimator in place of the composite.

Fields

  • ce: Covariance estimator.
  • mp: Matrix processing estimator.
  • cache: Optional partial-fit state. It is nothing until partial_fit! writes one. When the caller gives no data matrix, the batch verb of the estimator reads it. Each propagation channel does one thing with it. factory keeps it unchanged, because a factory call resolves the configuration, not the sample. port_opt_view copies the entries of the selected assets into a new state, so the viewed estimator holds the state of those assets alone. obs_weights_view removes it, because a state has no slice on the observation axis. Only a type with an observation-weights field has that channel. A type with no such field falls through to the identity method, so it keeps its state. A family whose state has no exact asset slice removes it on both axes and names the reason.

Constructors

PortfolioOptimisersCovariance(;    ce::StatsBase.CovarianceEstimator = Covariance(),    mp::AbstractMatrixProcessingEstimator = MatrixProcessing(),    cache::Option{<:AbstractPartialFitState} = nothing) -> PortfolioOptimisersCovariance

Keywords correspond to the struct's fields.

Validation

The incremental fit

The composite folds by composition. The state in cache selects one of two routes.

If cache holds nothing, partial_fit! passes the observation to ce.ce and keeps no state of its own. This route works because mp needs only a moment and a count. pdm and dt read only sigma, and dn reads the effective sample ratio T / N. The one-argument Statistics.cov reads the folded matrix of the inner estimator. Then it applies mp. When mp denoises, it reads the count from effective_observation_count, which is the same count that the batch fit reads. For an unweighted ce that count is size(X, 1). For a weighted ce it is the count of its weights. An mp.alg of the caller is the one step that needs the sample itself. The fold refuses it with an error that names it.

If cache holds a SampleBufferState, which Online seeds, the composite takes the buffer route. Then cov(ce) with no data runs the batch verb over the observations that the buffer kept. An mp.alg needs this route.

Propagated parameters

factory passes its arguments on to these @fprop fields:

  • ce: Recursively updated through factory.
  • cache: Carried unchanged through factory.

View parameters

port_opt_view takes the view of these @vprop fields at the selected indices:

Examples

julia> PortfolioOptimisersCovariance()PortfolioOptimisersCovariance  ce ┼ Covariance     │    me ┼ SimpleExpectedReturns     │       │   w ┴ nothing     │    ce ┼ GeneralCovariance     │       │   ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true)     │       │    w ┴ nothing     │   alg ┼ FullMoment()     │     w ┴ nothing  mp ┼ MatrixProcessing     │     pdm ┼ Posdef     │         │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton     │         │   kwargs ┴ @NamedTuple{}: NamedTuple()     │      dn ┼ nothing     │      dt ┼ nothing     │     alg ┼ nothing     │   order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg)

Related

source
Statistics.cov — Method
Statistics.cov(ce::PortfolioOptimisersCovariance, X::MatNum; dims = 1,
               active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing, kwargs...)

Compute the covariance matrix with a PortfolioOptimisersCovariance estimator, and process the matrix.

The method computes the covariance matrix of X with the inner estimator ce.ce. Then it applies the matrix processing step ce.mp.

The composite does not change a gap. It passes X and active_mask to ce.ce without change, so its gap_fill_value is the gap_fill_value of ce.ce. A composite that wraps a gap-aware estimator does not refuse a sample that is not finite. It repairs the finite block and not the whole frame. The AssetPanel method of this file does the same.

Algorithm

  1. Check dims, and orient X and active_mask to observations × assets. Transpose them when dims == 2.
  2. Refuse a gapped sample with assert_finite_sample when gap_fill_value on ce is finite, because then ce.ce is a plain estimator that cannot use a gapped sample.
  3. Compute sigma with Statistics.cov(library_covariance_estimator(ce.ce), X; kwargs...). Pass active_mask too if it is given.
  4. When sigma is immutable, copy it into a Matrix, because step 5 writes in place.
  5. Take the effective number of observations T with processing_observation_count, with the keywords of step 3. A mean of the caller then centres the data of the count and of the fit.
  6. Apply sample_matrix_processing! with ce.mp, X and T to sigma in place. If ce.ce is gap-aware, apply matrix_processing_block! instead, so that an asset outside the Coverage Universe keeps its NaN row and column.
  7. Return sigma.

ce.ce runs before ce.mp. ce.mp.order sets the order of the steps inside the processing. Step 1 orients X one time, so the estimator and the processing read the same orientation. Neither of them takes its own dims.

A weighted ce.ce is worth fewer observations than X has rows. So the denoising step reads the ratio T / N of the count of its weights, from effective_observation_count, and not of size(X, 1). An unweighted ce.ce gives T = size(X, 1), so its result does not change.

Arguments

  • ce: The composite covariance estimator with a processing step.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.
  • active_mask: The optional boolean matrix with the same size as X. The method passes it to ce.ce if it is given.
  • kwargs...: More keyword arguments for the inner covariance estimator and for the matrix processing step.

Validation

  • dims in (1, 2).

Returns

  • sigma::Matrix{<:Number}: The processed covariance matrix.

Related

source
Statistics.cor — Method
Statistics.cor(ce::PortfolioOptimisersCovariance, X::MatNum; dims = 1,
               active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing, kwargs...)

Compute the correlation matrix with a PortfolioOptimisersCovariance estimator, and process the matrix.

The method computes the correlation matrix of X with the inner estimator ce.ce. Then it applies the matrix processing step ce.mp.

Algorithm

  1. Check dims and orient X to observations × assets. Transpose it when dims == 2.
  2. Compute rho with Statistics.cor(library_covariance_estimator(ce.ce), X; kwargs...).
  3. When rho is immutable, copy it into a Matrix, because step 4 writes in place.
  4. Apply matrix_processing_block! with ce.mp, X and the count T of processing_observation_count to rho, in place. When every diagonal entry of rho is finite, it applies sample_matrix_processing! to the whole matrix. A gap-aware ce.ce gives a NaN row for an asset outside the Coverage Universe, and an estimator that rescales a covariance, or that ranks, gives one for an asset with a zero variance, see cov2cor_nan!. The processing then runs on the other assets, and the NaN row stays for the distance reader to refuse, see assert_distance_reader_rows.
  5. Return rho.

ce.ce runs before ce.mp. ce.mp.order sets the order of the steps inside the processing. Step 1 orients X one time, so the estimator and the processing read the same orientation. Neither of them takes its own dims.

Arguments

  • ce: The composite covariance estimator with a processing step.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.
  • active_mask: The optional boolean matrix with the same size as X. The method passes it to ce.ce if it is given.
  • kwargs...: More keyword arguments for the inner covariance estimator and for the matrix processing step.

Validation

  • dims in (1, 2).

Returns

  • rho::Matrix{<:Number}: The processed correlation matrix.

Related

source
Statistics.cov — Method
Statistics.cov(ce::PortfolioOptimisersCovariance, X::MatNum,
               pnl::Option{<:AssetPanel}; dims = 1, kwargs...) -> MatNum
Statistics.cor(ce::PortfolioOptimisersCovariance, X::MatNum,
               pnl::Option{<:AssetPanel}; dims = 1, kwargs...) -> MatNum

Pass the Asset Panel to the inner estimator, then repair the finite block of the frame that it returns.

This method is the override of the composite for the reduce-and-expand root. The inner estimator does the reduction, because only the inner estimator knows if it is plain or mask-aware. This method does the repair. The repair runs on the finite block through matrix_processing_block!, so an asset outside the Coverage Universe keeps its NaN row and column, and no entry outside the finite block changes.

Algorithm

  1. Check dims and orient X to observations × assets.
  2. Compute the matrix with the inner estimator. Pass pnl as its third positional argument.
  3. When the matrix is immutable, copy it into a Matrix, because step 4 writes in place.
  4. Repair its finite block with matrix_processing_block!, with ce.mp, the columns of X that the block names, and the count T of processing_observation_count.
  5. Return the matrix.

Arguments

  • ce: The composite covariance estimator with a processing step.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • pnl: Optional AssetPanel. The Coverage Universe of the fit comes from its active mask. If it is nothing, the rule reads only the finiteness of the returns.
  • dims: Dimension along which the computation runs.
  • kwargs...: More keyword arguments for the inner estimator and for the matrix processing step.

Validation

  • dims in (1, 2).

Returns

  • sigma::MatNum: The processed covariance matrix or correlation matrix, on the full asset universe.

Related

source
Statistics.var — Method
Statistics.var(ce::PortfolioOptimisersCovariance, X::MatNum,
               pnl::Option{<:AssetPanel}; dims::Int = 1, kwargs...) -> MatNum
Statistics.std(ce::PortfolioOptimisersCovariance, X::MatNum,
               pnl::Option{<:AssetPanel}; dims::Int = 1, kwargs...) -> MatNum

Take the variance or the standard deviation of the composite over an Asset Panel from the diagonal of its processed covariance over the same panel.

The variance of the composite is the diagonal of its processed covariance, as the matrix method of var states. Over an Asset Panel, the covariance comes from the panel method of cov of the composite, which lets the inner estimator choose the plain route or the mask-aware route. So this method does the same, and mp runs on the fit. The reduce-and-expand root of var would reduce the window to its Coverage Universe whatever the inner estimator is. When the composite does not fold, by series_folds, variance_series over an Asset Panel fits this method at each row.

Algorithm

  1. Compute the processed covariance with the panel method of Statistics.cov of the composite, with dims and the keywords.
  2. Read its diagonal, and take its square root for std.
  3. Reshape it to a 1 × N row vector when dims == 1, and to an N × 1 column vector otherwise.

Arguments

  • ce: The composite covariance estimator with a processing step.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • pnl: Optional AssetPanel. The Coverage Universe of the fit comes from its active mask. If it is nothing, the rule reads only the finiteness of the returns.
  • dims: Dimension along which the computation runs.
  • kwargs...: More keyword arguments for the inner estimator and for the matrix processing step.

Validation

  • dims in (1, 2).

Returns

  • val::Matrix{<:Number}: The variance or the standard deviation of each asset, NaN outside the assets that the inner estimator fits. Its shape is (1, N) if dims == 1, or (N, 1) if dims == 2.

Related

source
PortfolioOptimisers.supports_partial_fit — Method
supports_partial_fit(
    ce::PortfolioOptimisersCovariance
) -> Any

Answers whether partial_fit! folds a PortfolioOptimisersCovariance.

The composite folds by composition, so it folds exactly when the inner estimator folds and mp runs from the shape of the sample, as supports_shape_processing states. A MatrixProcessing passes when it has no alg that reads the sample. A matrix processing estimator of the caller never passes. A composite that a wrapper gave a buffer folds in all cases. This is the default buffer route of supports_partial_fit, and it is the only route that lets an mp.alg run online.

Arguments

  • ce: Covariance estimator.

Returns

Related

source
PortfolioOptimisers.partial_fit! — Method
partial_fit!(ce::PortfolioOptimisersCovariance{<:Any, <:Any, Nothing}, X::MatNum;
             dims::Int = 1, kwargs...)
partial_fit!(ce::PortfolioOptimisersCovariance{<:Any, <:Any, Nothing}, x::VecNum;
             kwargs...)

Folds observations into a PortfolioOptimisersCovariance through ce.ce.

The composite is exact by composition and keeps no state of its own. ce.ce folds the raw matrix. ce.mp is a step of cov(ce) with no data, and it runs on the matrix that it gets. The composite has no accumulator, and its cache stays nothing on this route. The nothing in cache selects this route. A composite with a SampleBufferState got that state from Online, and it takes the buffer fold.

The step costs what the step of the inner estimator costs, which is O(N²). The composite adds no cost.

ce.mp needs only a moment and a count, with one exception. An mp.alg of the caller gets the whole sample. assert_shape_only_matrix_processing refuses it at the fold. The refusal cannot wait until cov(ce) with no data, because the rows are gone at that time.

Algorithm

  1. Refuse an mp that does not run from the shape of the sample, with assert_shape_only_matrix_processing.
  2. Rebuild ce with the estimator that partial_fit! returns in ce.ce, through rebuild_struct.

Arguments

  • ce: Covariance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • x: One observation, with one entry for each asset.
  • dims: Dimension along which the computation runs.
  • kwargs...: More keyword arguments for ce.ce.

Validation

Returns

  • ce: The composite. Its ce.ce is the estimator that holds the state after the last observation.

Related

source
Statistics.cov — Method
Statistics.cov(ce::PortfolioOptimisersCovariance{<:Any, <:Any, Nothing}; kwargs...)
Statistics.cor(ce::PortfolioOptimisersCovariance{<:Any, <:Any, Nothing}; kwargs...)

Reads the covariance or the correlation that a folded PortfolioOptimisersCovariance holds.

This method makes the estimate from the fold by composition. It is cov(ce) or cor(ce) with no data. ce.ce returns its own folded matrix. Then the method applies ce.mp to that matrix from the shape of the sample. effective_observation_count reads the effective number of folded observations from the inner state. That number is the same count that the batch fit reads from processing_observation_count, so the substitution is an identity and not an approximation. This method returns the same result as a batch fit over the same observations. A windowed ce.ce under a SeedWindow counts the rows that its window kept on both routes.

The method reads the count only when matrix_processing_reads_count answers true, which is when ce.mp denoises. So the default mp, which only repairs the matrix, works over an inner state that records no count.

If the matrix of the inner estimator is immutable, the method copies it, because the processing writes in place. The matrix methods of this file do the same.

Algorithm

  1. Read the folded matrix of the inner estimator with the one-argument Statistics.cov or Statistics.cor.
  2. Copy it into a Matrix when it is immutable.
  3. Take T from effective_observation_count when matrix_processing_reads_count answers true, and nothing otherwise.
  4. Apply ce.mp in place through the shape method of matrix_processing_block!, with T and with N from the matrix. The method uses the block form, because a fold over a changing universe returns NaN for an asset outside the Coverage Universe. The AssetPanel methods of this file do the same. A complete matrix costs nothing more.
  5. Return the matrix.

Arguments

  • ce: Covariance estimator.
  • kwargs...: More keyword arguments for ce.ce.

Validation

  • ce.ce holds a partial-fit state. An ArgumentError is thrown otherwise.
  • ce.mp runs from the shape of the sample, as assert_shape_only_matrix_processing states. An ArgumentError is thrown otherwise.
  • When ce.mp reads the count, the state of ce.ce records one, as effective_observation_count states. An ArgumentError is thrown otherwise.

Returns

  • sigma::MatNum: The processed covariance matrix, or the processed correlation matrix rho.

Related

source
Statistics.cov — Method
Statistics.cov(ce::PortfolioOptimisersCovariance{<:Any, <:Any, <:SampleBufferState})
Statistics.cor(ce::PortfolioOptimisersCovariance{<:Any, <:Any, <:SampleBufferState})

Reads the covariance or the correlation of a buffered PortfolioOptimisersCovariance from a new fit over its buffer.

This method makes the estimate on the buffer route. Online seeds this route, and an mp.alg of the caller needs it. The method runs the batch verb over the observations that the buffer kept, so it returns the same result as a batch fit over those rows, with mp.alg.

Arguments

  • ce: Covariance estimator.

Validation

Returns

  • sigma::MatNum: The processed covariance matrix, or the processed correlation matrix rho.

Related

source