PortfolioOptimisers.jl

Appearance

6

Denoise ​

Real world data in general is often noisy. Financial data, is subject to myriad sources of noise, from latency, to arbitrage, to price uncertainty, to inherent randomness.

Denoising is about reducing or removing noise from signal. This can be done by modifying the small eigenvalues associated with noise to reduce their impact on the result [1, 2]. Denoising also reduces the condition number, thus improving the numerical behaviour of the final matrix.

PortfolioOptimisers.AbstractDenoiseEstimator Type
julia
abstract type AbstractDenoiseEstimator <: AbstractEstimator end

Abstract supertype for all denoising estimator types in PortfolioOptimisers.jl.

All concrete types that implement denoising of covariance or correlation matrices should subtype AbstractDenoiseEstimator. This enables a consistent interface for denoising routines throughout the package.

Related

source
PortfolioOptimisers.AbstractDenoiseAlgorithm Type
julia
abstract type AbstractDenoiseAlgorithm <: AbstractAlgorithm end

Abstract supertype for all denoising algorithm types in PortfolioOptimisers.jl.

All concrete types that implement a specific denoising algorithm should subtype AbstractDenoiseAlgorithm. This enables flexible extension and dispatch of denoising routines.

Related

source
PortfolioOptimisers.SpectralDenoise Type
julia
struct SpectralDenoise <: AbstractDenoiseAlgorithm end

A denoising algorithm that sets the smallest num_factors eigenvalues of a covariance or correlation matrix to zero, effectively removing the principal components relating to random noise according to random matrix theory-based approaches.

Examples

julia
julia> SpectralDenoise()
SpectralDenoise()

Related

References

  • [1] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.

  • [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).

source
PortfolioOptimisers.FixedDenoise Type
julia
struct FixedDenoise <: AbstractDenoiseAlgorithm end

A denoising algorithm that replaces the smallest num_factors eigenvalues of a covariance or correlation matrix with their average, effectively averaging the principal components relating to random noise according to random matrix theory-based approaches.

Examples

julia
julia> FixedDenoise()
FixedDenoise()

Related

References

  • [1] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.

  • [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).

source
PortfolioOptimisers.ShrunkDenoise Type
julia
struct ShrunkDenoise{T1} <: AbstractDenoiseAlgorithm
    alpha::T1
end

A denoising algorithm that shrinks the smallest num_factors eigenvalues of a covariance or correlation matrix towards their diagonal, controlled by the shrinkage parameter alpha. This approach interpolates between no shrinkage (alpha = 0) and full shrinkage (alpha = 1), providing a flexible way to regularize noisy eigenvalues.

Fields

  • alpha: The shrinkage parameter controlling the degree of shrinkage applied to the smallest eigenvalues.

Constructor

julia
ShrunkDenoise(; alpha::Number = 0.0)

Keyword arguments correspond to the fields above.

Validation

  • 0 <= alpha <= 1.

Examples

julia
julia> ShrunkDenoise(; alpha = 0.5)
ShrunkDenoise
  alpha ┴ Float64: 0.5

Related

References

  • [1] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.

  • [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).

source
PortfolioOptimisers.Denoise Type
julia
struct Denoise{T1, T2, T3, T4, T5, T6, T7} <: AbstractDenoiseEstimator
    alg::T1
    args::T2
    kwargs::T3
    kernel::T4
    m::T5
    n::T6
    pdm::T7
end

A flexible container type for configuring and applying denoising algorithms to covariance or correlation matrices in PortfolioOptimisers.jl.

Denoise encapsulates all parameters required for matrix denoising in denoise! and denoise, allowing users to specify the denoising algorithm, optimization parameters, kernel settings for density estimation, and optional positive definite matrix projection.

Fields

  • alg: Denoising algorithm.

  • args: Positional arguments for the univariate Optim.optimize.

  • kwargs: Keyword arguments for the univariate Optim.optimize.

  • kernel: Kernel function for AverageShiftedHistograms.ash.

  • m: Number of adjacent histograms to smooth over in AverageShiftedHistograms.ash.

  • n: Number of points in the range of eigenvalues used in the average shifted histogram density estimation.

  • pdm: Optional Positive definite matrix estimator. If provided, ensures the output is positive definite.

Constructor

julia
Denoise(; alg::AbstractDenoiseAlgorithm = ShrunkDenoise(), m::Integer = 10,
        n::Integer = 1000, kernel::Any = AverageShiftedHistograms.Kernels.gaussian,
        args::Tuple = (), kwargs::NamedTuple = (;), pdm::Option{<:Posdef} = Posdef())

Keyword arguments correspond to the fields above.

Examples

julia
julia> Denoise()
Denoise
     alg ┼ ShrunkDenoise
         │   alpha ┴ Float64: 0.0
    args ┼ Tuple{}: ()
  kwargs ┼ @NamedTuple{}: NamedTuple()
  kernel ┼ typeof(AverageShiftedHistograms.Kernels.gaussian): AverageShiftedHistograms.Kernels.gaussian
       m ┼ Int64: 10
       n ┼ Int64: 1000
     pdm ┼ Posdef
         │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton
         │   kwargs ┴ @NamedTuple{}: NamedTuple()

julia> Denoise(; alg = SpectralDenoise(), m = 20, n = 500)
Denoise
     alg ┼ SpectralDenoise()
    args ┼ Tuple{}: ()
  kwargs ┼ @NamedTuple{}: NamedTuple()
  kernel ┼ typeof(AverageShiftedHistograms.Kernels.gaussian): AverageShiftedHistograms.Kernels.gaussian
       m ┼ Int64: 20
       n ┼ Int64: 500
     pdm ┼ Posdef
         │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton
         │   kwargs ┴ @NamedTuple{}: NamedTuple()

Related

References

  • [1] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.

  • [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).

source
PortfolioOptimisers.errPDF Function
julia
errPDF(x::Number, vals::VecNum, q::Number,
       kernel::Any = AverageShiftedHistograms.Kernels.gaussian, m::Integer = 10,
       n::Integer = 1000)

Compute the sum of squared errors (SSE) between the theoretical Marčenko–Pastur (MP) eigenvalue density and the empirical eigenvalue density estimated from observed eigenvalues.

This function is used internally to fit the MP distribution to the observed spectrum, as part of the denoising procedure.

Arguments

  • x: Scale parameter for the MP distribution [0, 1].

  • vals: Observed eigenvalues.

  • q: Effective sample ratio (e.g., n_obs / n_assets).

  • kernel: Kernel function for AverageShiftedHistograms.ash.

  • m: Number of adjacent histograms to smooth over.

  • n: Number of points in the range of eigenvalues for density estimation.

Returns

  • sse::Number: The sum of squared errors between the empirical and theoretical densities.

Related

References

  • [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).
source
PortfolioOptimisers.find_max_eval Function
julia
find_max_eval(vals::VecNum, q::Number,
              kernel::Any = AverageShiftedHistograms.Kernels.gaussian, m::Integer = 10,
              n::Integer = 1000, args::Tuple = (), kwargs::NamedTuple = (;))

Estimate the upper edge of the Marčenko–Pastur (MP) distribution for a set of eigenvalues, used to separate signal from noise in random matrix denoising.

This function fits the MP distribution to the observed spectrum by minimizing the sum of squared errors between the empirical and theoretical densities, and returns the estimated maximum eigenvalue for noise.

Arguments

  • vals: Observed eigenvalues (typically sorted in ascending order).

  • q: Effective sample ratio (e.g., n_obs / n_assets).

  • kernel: Kernel function for AverageShiftedHistograms.ash.

  • m: Number of adjacent histograms to smooth over.

  • n: Number of points in the range of eigenvalues for density estimation.

  • args: Additional positional arguments for Optim.optimize.

  • kwargs: Additional keyword arguments for Optim.optimize.

Returns

  • e_max::Number: Estimated upper edge of the noise eigenvalue spectrum.

  • x::Number: Fitted scale parameter.

Related

References

  • [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).
source
PortfolioOptimisers._denoise! Function
julia
_denoise!(de::AbstractDenoiseAlgorithm, X::MatNum, vals::VecNum, vecs::MatNum,
          num_factors::Integer)

In-place denoising of a covariance or correlation matrix using a specific denoising algorithm.

These methods are called internally by denoise! and denoise when a Denoise estimator is used, and should not typically be called directly.

Arguments

  • alg: The denoising algorithm to apply.

  • X: The matrix to be denoised (modified in-place).

  • vals: Eigenvalues of X, sorted in ascending order.

  • vecs: Corresponding eigenvectors of X.

  • num_factors: Number of eigenvalues to treat as noise.

  • pdm: Positive definite matrix estimator.

Returns

  • nothing. The input matrix X is modified in-place.

Related

References

  • [1] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.

  • [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).

source
PortfolioOptimisers.denoise! Function
julia
denoise!(de::Denoise, X::MatNum, q::Number)
denoise!(::Nothing, args...)

In-place denoising of a covariance or correlation matrix using a Denoise estimator.

For matrices without unit diagonal, the function converts them into correlation matrices i.e. matrices with unit diagonal, applies the algorithm, and rescales them back.

Arguments

  • de: The estimator specifying the denoising algorithm.

    • de::Denoise: The specified denoising algorithm is applied to X in-place.

    • de::Nothing: No-op.

  • X: The covariance or correlation matrix to be denoised (modified in-place).

  • q: The effective sample ratio (e.g., n_obs / n_assets), used for spectral thresholding.

Returns

  • nothing. The input matrix X is modified in-place.

Examples

julia
julia> using StableRNGs

julia> rng = StableRNG(123456789);

julia> X = rand(rng, 10, 5);

julia> X = X' * X
5×5 Matrix{Float64}:
 3.29494  2.0765   1.73334  2.01524  1.77493
 2.0765   2.46967  1.39953  1.97242  2.07886
 1.73334  1.39953  1.90712  1.17071  1.30459
 2.01524  1.97242  1.17071  2.24818  1.87091
 1.77493  2.07886  1.30459  1.87091  2.44414

julia> denoise!(Denoise(), X, 10 / 5)

julia> X
5×5 Matrix{Float64}:
 3.29494  2.28883  1.70633  2.12343  2.17377
 2.28883  2.46967  1.59575  1.98583  2.0329
 1.70633  1.59575  1.90712  1.48044  1.51553
 2.12343  1.98583  1.48044  2.24818  1.886
 2.17377  2.0329   1.51553  1.886    2.44414

Related

References

  • [1] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.

  • [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).

source
PortfolioOptimisers.denoise Function
julia
denoise(de::Denoise, X::MatNum, q::Number)
denoise(::Nothing, args...)

Out-of-place version of denoise!.

Related

References

  • [1] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.

  • [2] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).

source