Matrix processing

Co-moment matrices can be post-processed after being computed. These processes are often complementary but there is no set order.

At the base level, we have four possible post processing steps:

1. Positive definite projection.

2. Denoising.

3. Detoning.

4. Custom process.

The only set order is that positive definite projection should come first. This is because the other post-processing methods work best with positive definite matrices, and may use positive definite projection internally.

Aside from this, there is no set canonical order, the closest to a heuristic we can justify is to denoise before detoning. The order is configured as a tuple or vector of step symbols (:pdm, :dn, :dt, :alg), applied left to right.

PortfolioOptimisers.AbstractMatrixProcessingEstimator Type

abstract type AbstractMatrixProcessingEstimator <: AbstractEstimator

Abstract supertype for all matrix processing estimator types in PortfolioOptimisers.jl.

All concrete and/or abstract types that implement matrix processing routines–-such as covariance matrix cleaning, denoising, or detoning–-should be subtypes of AbstractMatrixProcessingEstimator.

Interfaces

In order to implement a new matrix processing estimator which will work seamlessly with the library, subtype AbstractMatrixProcessingEstimator with all necessary parameters as part of the struct, and implement the following methods:

• matrix_processing!(mp::AbstractMatrixProcessingEstimator, sigma::MatNum, X::MatNum, args...; kwargs...) -> MatNum: In-place processing of a covariance or correlation matrix.

• matrix_processing(mp::AbstractMatrixProcessingEstimator, sigma::MatNum, X::MatNum, args...; kwargs...) -> MatNum: Optional out-of-place processing of a covariance or correlation matrix.

Arguments

• mp: Matrix processing estimator.

• sigma: Covariance-like or correlation-like matrix features × features.

• X: Data matrix observations × features if the dims keyword does not exist or dims = 1, features × observations when dims = 2.

• args...: Additional positional arguments passed to custom algorithms.

• kwargs...: Additional keyword arguments passed to custom algorithms.

Returns

• sigma::MatNum: The processed input matrix sigma.

Examples

We can create a dummy matrix processing estimator as follows:

julia> struct MyMatrixProcessingEstimator <: PortfolioOptimisers.AbstractMatrixProcessingEstimator end

julia> function PortfolioOptimisers.matrix_processing!(est::MyMatrixProcessingEstimator,
                                                       sigma::PortfolioOptimisers.MatNum,
                                                       X::PortfolioOptimisers.MatNum)
           # Implement your in-place matrix processing logic here.
           println("Processing matrix in-place...")
           return sigma
       end

julia> function PortfolioOptimisers.matrix_processing(est::MyMatrixProcessingEstimator,
                                                      sigma::PortfolioOptimisers.MatNum,
                                                      X::PortfolioOptimisers.MatNum)
           sigma = copy(sigma)
           println("Copy sigma...")
           matrix_processing!(est, sigma, X)
           return sigma
       end

julia> matrix_processing!(MyMatrixProcessingEstimator(), [1.0 2.0; 2.0 1.0], rand(10, 2))
Processing matrix in-place...
2×2 Matrix{Float64}:
 1.0  2.0
 2.0  1.0

julia> matrix_processing(MyMatrixProcessingEstimator(), [1.0 2.0; 2.0 1.0], rand(10, 2))
Copy sigma...
Processing matrix in-place...
2×2 Matrix{Float64}:
 1.0  2.0
 2.0  1.0

Related

• AbstractEstimator

• MatrixProcessing

source

PortfolioOptimisers.AbstractMatrixProcessingAlgorithm Type

abstract type AbstractMatrixProcessingAlgorithm <: AbstractAlgorithm

Abstract supertype for all matrix processing algorithm types in PortfolioOptimisers.jl.

All concrete and/or abstract types that implement a specific matrix processing algorithm should be subtypes of AbstractMatrixProcessingAlgorithm.

Interfaces

In order to implement a new matrix processing algorithm that works with the current matrix processing estimator, subtype AbstractMatrixProcessingAlgorithm, with all necessary parameters as part of the struct, and implement the following methods:

• matrix_processing_algorithm!(mpa::AbstractMatrixProcessingAlgorithm, sigma::MatNum, args...; kwargs...) -> MatNum: In-place application of a custom matrix processing algorithm.

• matrix_processing_algorithm(mpa::AbstractMatrixProcessingAlgorithm, sigma::MatNum, args...; kwargs...) -> MatNum: Optional out-of-place application of a custom matrix processing algorithm.

Arguments

• mpa: Matrix processing algorithm.

• args...: Additional positional arguments.

• kwargs...: Additional keyword arguments.

Returns

• sigma::MatNum: The input matrix sigma after applying the algorithm.

Examples

We can create a dummy matrix processing algorithm as follows:

julia> struct MyMatrixProcessingAlgorithm <: PortfolioOptimisers.AbstractMatrixProcessingAlgorithm end

julia> function PortfolioOptimisers.matrix_processing_algorithm!(alg::MyMatrixProcessingAlgorithm,
                                                                 sigma::PortfolioOptimisers.MatNum,
                                                                 X::PortfolioOptimisers.MatNum;
                                                                 kwargs...)
           # Implement your in-place matrix processing algorithm logic here.
           println("Applying custom matrix processing algorithm in-place...")
           return sigma
       end

julia> function PortfolioOptimisers.matrix_processing_algorithm(alg::MyMatrixProcessingAlgorithm,
                                                                sigma::PortfolioOptimisers.MatNum,
                                                                X::PortfolioOptimisers.MatNum;
                                                                kwargs...)
           sigma = copy(sigma)
           println("Copy sigma...")
           return PortfolioOptimisers.matrix_processing_algorithm!(alg, sigma, X; kwargs...)
       end

julia> matrix_processing!(MatrixProcessing(; alg = MyMatrixProcessingAlgorithm()),
                          [1.0 2.0; 2.0 1.0], rand(10, 2))
Applying custom matrix processing algorithm in-place...
2×2 Matrix{Float64}:
 1.0  1.0
 1.0  1.0

julia> PortfolioOptimisers.matrix_processing_algorithm(MyMatrixProcessingAlgorithm(),
                                                       [1.0 2.0; 2.0 1.0], rand(10, 2))
Copy sigma...
Applying custom matrix processing algorithm in-place...
2×2 Matrix{Float64}:
 1.0  2.0
 2.0  1.0

Related

• AbstractAlgorithm

• MatrixProcessing

source

PortfolioOptimisers.MatrixProcessing Type

struct MatrixProcessing{__T_pdm, __T_dn, __T_dt, __T_alg, __T_order} <: AbstractMatrixProcessingEstimator

A flexible container type for configuring and applying matrix processing routines in PortfolioOptimisers.jl.

MatrixProcessing encapsulates all steps required for processing covariance or correlation matrices, including positive definiteness enforcement, denoising, detoning, and optional custom matrix processing algorithms via matrix_processing! and matrix_processing. This estimator allows users to build complex matrix processing pipelines tailored to their specific needs.

Fields

• pdm: Optional positive definite matrix estimator.

• dn: Optional matrix denoising estimator.

• dt: Optional matrix detoning estimator.

• alg: Optional custom matrix processing algorithm.

• order: A tuple or vector of symbols naming the processing steps in the order they are applied. Recognised steps are :pdm, :dn, :dt, and :alg; an unrecognised symbol errors at construction.

Constructors

MatrixProcessing(;
    pdm::Option{<:Posdef} = Posdef(),
    dn::Option{<:Denoise} = nothing,
    dt::Option{<:Detone} = nothing,
    alg::Option{<:AbstractMatrixProcessingAlgorithm} = nothing,
    order = (:pdm, :dn, :dt, :alg)
) -> MatrixProcessing

Keywords correspond to the struct's fields.

Examples

julia> MatrixProcessing()
MatrixProcessing
    pdm ┼ Posdef
        │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton
        │   kwargs ┴ @NamedTuple{}: NamedTuple()
     dn ┼ nothing
     dt ┼ nothing
    alg ┼ nothing
  order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg)

julia> MatrixProcessing(; dn = Denoise(), dt = Detone(; n = 2))
MatrixProcessing
    pdm ┼ Posdef
        │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton
        │   kwargs ┴ @NamedTuple{}: NamedTuple()
     dn ┼ Denoise
        │      pdm ┼ Posdef
        │          │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton
        │          │   kwargs ┴ @NamedTuple{}: NamedTuple()
        │      alg ┼ ShrunkDenoise
        │          │   alpha ┴ Float64: 0.0
        │     args ┼ Tuple{}: ()
        │   kwargs ┼ @NamedTuple{}: NamedTuple()
        │   kernel ┼ typeof(AverageShiftedHistograms.Kernels.gaussian): AverageShiftedHistograms.Kernels.gaussian
        │        m ┼ Int64: 10
        │        n ┴ Int64: 1000
     dt ┼ Detone
        │   pdm ┼ Posdef
        │       │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton
        │       │   kwargs ┴ @NamedTuple{}: NamedTuple()
        │     n ┴ Int64: 2
    alg ┼ nothing
  order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg)

Related

• AbstractMatrixProcessingEstimator

• matrix_processing!

• matrix_processing

• Option

• Posdef

• Denoise

• Detone

• AbstractMatrixProcessingAlgorithm

References

• [1] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.

• [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).

source

PortfolioOptimisers.matrix_processing! Function

matrix_processing!(
    mp::Option{<:MatrixProcessing},
    sigma::MatNum,
    X::MatNum,
    args...;
    kwargs...
) -> MatNum

In-place matrix processing pipeline.

This method applies a sequence of matrix processing steps to the input covariance or correlation matrix sigma, modifying it in-place. The steps and their order are given by mp.order–-a tuple or vector of symbols (:pdm, :dn, :dt, :alg)–-and each step is dispatched through matrix_processing_step!.

Arguments

• mp: Optional matrix processing estimator.

  • ::MatrixProcessing: The specified matrix processing estimator is applied to X in-place.

  • ::Nothing: No-op.

• sigma: Covariance-like or correlation-like matrix features × features.

• X: Data matrix observations × features if the dims keyword does not exist or dims = 1, features × observations when dims = 2.

• args...: Additional positional arguments passed to custom algorithms.

• kwargs...: Additional keyword arguments passed to custom algorithms.

Returns

• sigma::MatNum: The input matrix sigma is modified in-place.

Details

• If mp is nothing, the function returns sigma without modification.

• Iterates over mp.order and applies each named step via matrix_processing_step!: :pdm (using mp.pdm), :dn (using mp.dn and the ratio T / N from X), :dt (using mp.dt), and :alg (using mp.alg).

• An unrecognised step symbol errors at construction.

Examples

julia> using StableRNGs, Statistics

julia> rng = StableRNG(123456789);

julia> X = rand(rng, 10, 5);

julia> sigma = cov(X)
5×5 Matrix{Float64}:
  0.132026     0.0022567   0.0198243    0.00359832  -0.00743829
  0.0022567    0.0514194  -0.0131242    0.004123     0.0312379
  0.0198243   -0.0131242   0.0843837   -0.0325342   -0.00609624
  0.00359832   0.004123   -0.0325342    0.0424332    0.0152574
 -0.00743829   0.0312379  -0.00609624   0.0152574    0.0926441

julia> matrix_processing!(MatrixProcessing(; dn = Denoise()), sigma, X)
5×5 Matrix{Float64}:
 0.132026  0.0        0.0        0.0        0.0
 0.0       0.0514194  0.0        0.0        0.0
 0.0       0.0        0.0843837  0.0        0.0
 0.0       0.0        0.0        0.0424332  0.0
 0.0       0.0        0.0        0.0        0.0926441

julia> sigma = cov(X)
5×5 Matrix{Float64}:
  0.132026     0.0022567   0.0198243    0.00359832  -0.00743829
  0.0022567    0.0514194  -0.0131242    0.004123     0.0312379
  0.0198243   -0.0131242   0.0843837   -0.0325342   -0.00609624
  0.00359832   0.004123   -0.0325342    0.0424332    0.0152574
 -0.00743829   0.0312379  -0.00609624   0.0152574    0.0926441

julia> matrix_processing!(MatrixProcessing(; dt = Detone()), sigma, X)
5×5 Matrix{Float64}:
 0.132026    0.0124802   0.0117303    0.0176194    0.0042142
 0.0124802   0.0514194   0.0273105   -0.0290864    0.0088165
 0.0117303   0.0273105   0.0843837   -0.00279296   0.0619156
 0.0176194  -0.0290864  -0.00279296   0.0424332   -0.0242252
 0.0042142   0.0088165   0.0619156   -0.0242252    0.0926441

Related

• AbstractMatrixProcessingEstimator

• MatrixProcessing

• matrix_processing_step!

• matrix_processing

• posdef!

• denoise!

• detone!

• matrix_processing_algorithm!

• MatNum

References

• [1] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.

• [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).

source

PortfolioOptimisers.matrix_processing_step! Function

matrix_processing_step!(::Val{step}, mp::MatrixProcessing, sigma::MatNum, X::MatNum; kwargs...) -> MatNum

Apply a single named matrix processing step to sigma in-place, dispatching on the step symbol step.

This is the per-step worker that matrix_processing! calls while iterating over mp.order. Each recognised symbol maps to one of the estimator fields of mp; the override-or-skip behaviour is inherited from the underlying primitives (a nothing estimator is a no-op).

Arguments

• ::Val{step}: The processing step to apply, named by a symbol:

  • :pdm: positive definiteness enforcement using mp.pdm.

  • :dn: denoising using mp.dn and the ratio T / N derived from X.

  • :dt: detoning using mp.dt.

  • :alg: optional custom algorithm using mp.alg.

  • Any other symbol: MethodError.

• mp: Matrix processing estimator holding the per-step estimators.

• sigma: Covariance-like or correlation-like matrix features × features.

• X: Data matrix observations × features if the dims keyword does not exist or dims = 1, features × observations when dims = 2.

• kwargs...: Additional keyword arguments passed to custom algorithms.

Returns

• sigma::MatNum: The input matrix sigma, modified in-place.

Related

• matrix_processing!

• MatrixProcessing

source

PortfolioOptimisers.matrix_processing Function

matrix_processing(
    mp::Option{<:AbstractMatrixProcessingEstimator},
    sigma::MatNum,
    X::MatNum,
    args...;
    kwargs...
) -> MatNum

Out-of-place version of matrix_processing!.

Arguments

• mp: Optional matrix processing estimator.

  • ::AbstractMatrixProcessingEstimator: The specified processing pipeline is applied to a copy of sigma.

  • ::Nothing: No-op, returns sigma unchanged.

• sigma: Covariance-like or correlation-like matrix features × features.

• X: Data matrix observations × features if the dims keyword does not exist or dims = 1, features × observations when dims = 2.

• args...: Additional positional arguments passed to custom algorithms.

• kwargs...: Additional keyword arguments passed to custom algorithms.

Returns

• sigma::MatNum: A new matrix equal to the processed version of the input.

Examples

julia> using StableRNGs, Statistics

julia> rng = StableRNG(123456789);

julia> X = rand(rng, 10, 5);

julia> sigma = cov(X);

julia> Xs = matrix_processing(MatrixProcessing(; dn = Denoise()), sigma, X);

julia> size(Xs)
(5, 5)

Related

• matrix_processing!

• MatrixProcessing

• posdef!

• denoise!

• detone!

• matrix_processing_algorithm!

• AbstractMatrixProcessingEstimator

• MatNum

source

PortfolioOptimisers.matrix_processing_algorithm! Method

matrix_processing_algorithm!(::Nothing, sigma::MatNum, args...; kwargs...)

No-op fallback for matrix processing algorithm routines.

These methods are called internally when no matrix processing algorithm is specified (i.e., when the algorithm argument is nothing). They perform no operation and return nothing, ensuring that the matrix processing pipeline can safely skip optional algorithmic steps.

Arguments

• ::Nothing: Indicates that no matrix processing algorithm is specified.

• args...: Additional positional arguments (ignored).

• kwargs...: Additional keyword arguments (ignored).

Returns

• sigma::MatNum: The input matrix sigma is returned unchanged.

Related

• matrix_processing_algorithm

• MatrixProcessing

source

PortfolioOptimisers.matrix_processing_algorithm Method

matrix_processing_algorithm(::Nothing, sigma::MatNum, args...; kwargs...)

Same as matrix_processing_algorithm!, but meant for returning a new matrix instead of modifying it in-place.

Related

• matrix_processing_algorithm!

• MatrixProcessing

source