Denoise

Real world data in general is often noisy. Financial data, is subject to myriad sources of noise, from latency, to arbitrage, to price uncertainty, to inherent randomness.

Denoising is about reducing or removing noise from signal. This can be done by modifying the small eigenvalues associated with noise to reduce their impact on the result [1, 2]. Denoising also reduces the condition number, thus improving the numerical behaviour of the final matrix.

PortfolioOptimisers.AbstractDenoiseEstimator Type

abstract type AbstractDenoiseEstimator <: AbstractEstimator end

Abstract supertype for all denoising estimator types in PortfolioOptimisers.jl.

All concrete and/or abstract types that implement denoising of covariance-like or correlation-like matrices should be subtypes of AbstractDenoiseEstimator.

Interfaces

In order to implement a new denoising estimator which will work seamlessly with the library, subtype AbstractDenoiseEstimator with all necessary parameters as part of the struct, and implement the following methods:

• denoise!(dn::AbstractDenoiseEstimator, X::MatNum, q::Number): In-place denoising.

• denoise(dn::AbstractDenoiseEstimator, X::MatNum, q::Number): Optional out-of-place denoising.

Arguments

• dn: Matrix denoising estimator.

• X: Covariance-like or correlation-like matrix.

• q: The effective sample ratio observations / assets, used for spectral thresholding.

Returns

• X::MatNum: The denoised input matrix X.

Examples

We can create a dummy denoising estimator as follows:

julia> struct MyDenoiseEstimator <: PortfolioOptimisers.AbstractDenoiseEstimator end

julia> function PortfolioOptimisers.denoise!(dn::MyDenoiseEstimator, X::PortfolioOptimisers.MatNum)
           # Implement your in-place denoising estimator here.
           println("Denoising matrix in-place...")
           return X
       end

julia> function PortfolioOptimisers.denoise(dn::MyDenoiseEstimator, X::PortfolioOptimisers.MatNum)
           X = copy(X)
           println("Copy X...")
           denoise!(dn, X)
           return X
       end

julia> denoise!(MyDenoiseEstimator(), [1.0 2.0; 2.0 1.0])
Denoising matrix in-place...
2×2 Matrix{Float64}:
 1.0  2.0
 2.0  1.0

julia> denoise(MyDenoiseEstimator(), [1.0 2.0; 2.0 1.0])
Copy X...
Denoising matrix in-place...
2×2 Matrix{Float64}:
 1.0  2.0
 2.0  1.0

Related

• AbstractEstimator

• Denoise

• denoise!

• denoise

source

PortfolioOptimisers.AbstractDenoiseAlgorithm Type

abstract type AbstractDenoiseAlgorithm <: AbstractAlgorithm end

Abstract supertype for all denoising algorithm types in PortfolioOptimisers.jl.

All concrete and/or abstract types that implement a specific denoising algorithm should be subtypes of AbstractDenoiseAlgorithm.

Interfaces

If you wish to implement a new denoising algorithm that works with an existing denoising estimator, subtype AbstractDenoiseAlgorithm, with all necessary parameters as part of the struct, and implement the following method:

• _denoise!(alg::AbstractDenoiseAlgorithm, X::MatNum, vals::VecNum, vecs::MatNum, num_factors::Integer): In-place denoising of a covariance or correlation matrix using the specific algorithm.

Arguments

• alg: Denoising algorithm.

• X: Covariance-like or correlation-like matrix.

• vals: Eigenvalues of X, sorted in ascending order.

• vecs: Corresponding eigenvectors of X.

• num_factors: Number of eigenvalues to treat as noise.

Returns

• X::MatNum: The input matrix X is modified in-place.

Examples

We can create a dummy denoising algorithm as follows:

julia> struct MyDenoiseAlgorithm <: PortfolioOptimisers.AbstractDenoiseAlgorithm end

julia> function PortfolioOptimisers._denoise!(dn::MyDenoiseAlgorithm,
                                              X::PortfolioOptimisers.MatNum,
                                              vals::PortfolioOptimisers.VecNum,
                                              vecs::PortfolioOptimisers.MatNum,
                                              num_factors::Integer)
           # Implement your in-place denoising logic here.
           println("Denoising matrix using custom algorithm...")
           return X
       end

julia> denoise!(Denoise(; alg = MyDenoiseAlgorithm()), [2.0 1.0; 1.0 2.0], 1 / 100)
Denoising matrix using custom algorithm...
2×2 Matrix{Float64}:
 2.0  1.0
 1.0  2.0

julia> denoise(Denoise(; alg = MyDenoiseAlgorithm()), [2.0 1.0; 1.0 2.0], 1 / 100)
Denoising matrix using custom algorithm...
2×2 Matrix{Float64}:
 2.0  1.0
 1.0  2.0

Related

• AbstractAlgorithm

• SpectralDenoise

• FixedDenoise

• ShrunkDenoise

source

PortfolioOptimisers.SpectralDenoise Type

struct SpectralDenoise <: AbstractDenoiseAlgorithm end

A denoising algorithm that sets the smallest num_factors eigenvalues of a covariance or correlation matrix to zero, effectively removing the principal components relating to random noise according to random matrix theory-based approaches.

Examples

julia> SpectralDenoise()
SpectralDenoise()

Related

• AbstractDenoiseAlgorithm

• denoise!

• Denoise

References

• [1] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.

• [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).

source

PortfolioOptimisers.FixedDenoise Type

struct FixedDenoise <: AbstractDenoiseAlgorithm end

A denoising algorithm that replaces the smallest num_factors eigenvalues of a covariance or correlation matrix with their average, effectively averaging the principal components relating to random noise according to random matrix theory-based approaches.

Examples

julia> FixedDenoise()
FixedDenoise()

Related

• AbstractDenoiseAlgorithm

• denoise!

• Denoise

References

• [1] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.

• [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).

source

PortfolioOptimisers.ShrunkDenoise Type

struct ShrunkDenoise{T1} <: AbstractDenoiseAlgorithm
    alpha::T1
end

A denoising algorithm that shrinks the smallest num_factors eigenvalues of a covariance or correlation matrix towards their diagonal, controlled by the shrinkage parameter alpha. This approach interpolates between no shrinkage (alpha = 0) and full shrinkage (alpha = 1), providing a flexible way to regularize noisy eigenvalues.

Fields

• alpha: The shrinkage parameter controlling the degree of shrinkage applied to the smallest eigenvalues.

Constructor

ShrunkDenoise(; alpha::Number = 0.0)

Keyword arguments correspond to the fields above.

Validation

• 0 <= alpha <= 1.

Examples

julia> ShrunkDenoise(; alpha = 0.5)
ShrunkDenoise
  alpha ┴ Float64: 0.5

Related

• AbstractDenoiseAlgorithm

• denoise!

• Denoise

References

• [1] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.

• [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).

source

PortfolioOptimisers.Denoise Type

struct Denoise{T1, T2, T3, T4, T5, T6, T7} <: AbstractDenoiseEstimator
    alg::T1
    args::T2
    kwargs::T3
    kernel::T4
    m::T5
    n::T6
    pdm::T7
end

A flexible container type for configuring and applying denoising algorithms to covariance or correlation matrices in PortfolioOptimisers.jl.

Denoise encapsulates all parameters required for matrix denoising in denoise! and denoise, allowing users to specify the denoising algorithm, optimization parameters, kernel settings for density estimation, and optional positive definite matrix projection.

Fields

• alg: Denoising algorithm.

• args: Positional arguments for the univariate Optim.optimize.

• kwargs: Keyword arguments for the univariate Optim.optimize.

• kernel: Kernel function for AverageShiftedHistograms.ash.

• m: Number of adjacent histograms to smooth over in AverageShiftedHistograms.ash.

• n: Number of points in the range of eigenvalues used in the average shifted histogram density estimation.

• pdm: Optional positive definite matrix estimator.

Constructor

Denoise(; alg::AbstractDenoiseAlgorithm = ShrunkDenoise(), m::Integer = 10,
        n::Integer = 1000, kernel::Any = AverageShiftedHistograms.Kernels.gaussian,
        args::Tuple = (), kwargs::NamedTuple = (;), pdm::Option{<:Posdef} = Posdef())

Keyword arguments correspond to the fields above.

Examples

julia> Denoise()
Denoise
     alg ┼ ShrunkDenoise
         │   alpha ┴ Float64: 0.0
    args ┼ Tuple{}: ()
  kwargs ┼ @NamedTuple{}: NamedTuple()
  kernel ┼ typeof(AverageShiftedHistograms.Kernels.gaussian): AverageShiftedHistograms.Kernels.gaussian
       m ┼ Int64: 10
       n ┼ Int64: 1000
     pdm ┼ Posdef
         │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton
         │   kwargs ┴ @NamedTuple{}: NamedTuple()

julia> Denoise(; alg = SpectralDenoise(), m = 20, n = 500)
Denoise
     alg ┼ SpectralDenoise()
    args ┼ Tuple{}: ()
  kwargs ┼ @NamedTuple{}: NamedTuple()
  kernel ┼ typeof(AverageShiftedHistograms.Kernels.gaussian): AverageShiftedHistograms.Kernels.gaussian
       m ┼ Int64: 20
       n ┼ Int64: 500
     pdm ┼ Posdef
         │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton
         │   kwargs ┴ @NamedTuple{}: NamedTuple()

Related

• AbstractDenoiseEstimator

• SpectralDenoise

• FixedDenoise

• ShrunkDenoise

• denoise!

• denoise

• AverageShiftedHistograms.Kernels

References

• [1] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.

• [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).

source

PortfolioOptimisers.errPDF Function

errPDF(x::Number, vals::VecNum, q::Number,
       kernel::Any = AverageShiftedHistograms.Kernels.gaussian, m::Integer = 10,
       n::Integer = 1000)

Compute the sum of squared errors (SSE) between the theoretical Marčenko–Pastur (MP) eigenvalue density and the empirical eigenvalue density estimated from observed eigenvalues.

This function is used internally to fit the MP distribution to the observed spectrum, as part of the denoising procedure.

Arguments

• x: Scale parameter for the MP distribution [0, 1].

• vals: Observed eigenvalues.

• q: Effective sample ratio (e.g., n_obs / n_assets).

• kernel: Kernel function for AverageShiftedHistograms.ash.

• m: Number of adjacent histograms to smooth over.

• n: Number of points in the range of eigenvalues for density estimation.

Returns

• sse::Number: The sum of squared errors between the empirical and theoretical densities.

Related

• find_max_eval

• Denoise

• VecNum

• AverageShiftedHistograms.Kernels

References

• [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).

source

PortfolioOptimisers.find_max_eval Function

find_max_eval(vals::VecNum, q::Number,
              kernel::Any = AverageShiftedHistograms.Kernels.gaussian, m::Integer = 10,
              n::Integer = 1000, args::Tuple = (), kwargs::NamedTuple = (;))

Estimate the upper edge of the Marčenko–Pastur (MP) distribution for a set of eigenvalues, used to separate signal from noise in random matrix denoising.

This function fits the MP distribution to the observed spectrum by minimizing the sum of squared errors between the empirical and theoretical densities, and returns the estimated maximum eigenvalue for noise.

Arguments

• vals: Observed eigenvalues (typically sorted in ascending order).

• q: Effective sample ratio (e.g., n_obs / n_assets).

• kernel: Kernel function for AverageShiftedHistograms.ash.

• m: Number of adjacent histograms to smooth over.

• n: Number of points in the range of eigenvalues for density estimation.

• args: Additional positional arguments for Optim.optimize.

• kwargs: Additional keyword arguments for Optim.optimize.

Returns

• e_max::Number: Estimated upper edge of the noise eigenvalue spectrum.

• x::Number: Fitted scale parameter.

Related

• errPDF

• Denoise

• VecNum

• AverageShiftedHistograms.Kernels

References

• [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).

source

PortfolioOptimisers._denoise! Function

_denoise!(alg::AbstractDenoiseAlgorithm, X::MatNum, vals::VecNum, vecs::MatNum,
          num_factors::Integer)

In-place denoising of a covariance or correlation matrix using a specific denoising algorithm.

These methods are called internally by denoise! and denoise when a Denoise estimator is used, and should not typically be called directly.

Arguments

• alg: Denoising algorithm.

• X: Covariance-like or correlation-like matrix.

• vals: Eigenvalues of X, sorted in ascending order.

• vecs: Corresponding eigenvectors of X.

• num_factors: Number of eigenvalues to treat as noise.

Returns

• X::MatNum: The input matrix X is modified in-place.

Related

• denoise!

• Denoise

• SpectralDenoise

• FixedDenoise

• ShrunkDenoise

• MatNum

• VecNum

References

• [1] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.

• [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).

source

PortfolioOptimisers.denoise! Function

denoise!(dn::Denoise, X::MatNum, q::Number)
denoise!(::Nothing, X::MatNum, args...)

In-place denoising of a covariance or correlation matrix using a Denoise estimator.

For matrices without unit diagonal, the function converts them into correlation matrices i.e. matrices with unit diagonal, applies the algorithm, and rescales them back.

Arguments

• dn: Optional matrix denoising estimator.

  • ::Denoise: The specified denoising algorithm is applied to X in-place.

  • ::Nothing: No-op.

• X: Covariance-like or correlation-like matrix.

• q: The effective sample ratio observations / assets, used for spectral thresholding.

Returns

• X::MatNum: The input matrix X is modified in-place.

Examples

julia> using StableRNGs

julia> rng = StableRNG(123456789);

julia> X = rand(rng, 10, 5);

julia> X = X' * X
5×5 Matrix{Float64}:
 3.29494  2.0765   1.73334  2.01524  1.77493
 2.0765   2.46967  1.39953  1.97242  2.07886
 1.73334  1.39953  1.90712  1.17071  1.30459
 2.01524  1.97242  1.17071  2.24818  1.87091
 1.77493  2.07886  1.30459  1.87091  2.44414

julia> denoise!(Denoise(), X, 10 / 5)
5×5 Matrix{Float64}:
 3.29494  2.28883  1.70633  2.12343  2.17377
 2.28883  2.46967  1.59575  1.98583  2.0329
 1.70633  1.59575  1.90712  1.48044  1.51553
 2.12343  1.98583  1.48044  2.24818  1.886
 2.17377  2.0329   1.51553  1.886    2.44414

Related

• denoise

• Denoise

• SpectralDenoise

• FixedDenoise

• ShrunkDenoise

• posdef!

• MatNum

• Option

• Posdef

References

• [1] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.

• [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).

source

PortfolioOptimisers.denoise Function

denoise(dn::Denoise, X::MatNum, q::Number)
denoise(::Nothing, X::MatNum, args...)

Out-of-place version of denoise!.

Related

• denoise!

• Denoise

• SpectralDenoise

• FixedDenoise

• ShrunkDenoise

• posdef

• MatNum

• Option

• Posdef

References

• [1] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.

• [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).

source