GitHub

Denoise

Real world data in general is often noisy. Financial data, is subject to myriad sources of noise, from latency, to arbitrage, to price uncertainty, to inherent randomness.

Denoising is about reducing or removing noise from signal. This can be done by modifying the small eigenvalues associated with noise to reduce their impact on the result [9, 10]. Denoising also reduces the condition number, thus improving the numerical behaviour of the final matrix.

PortfolioOptimisers.AbstractDenoiseEstimator — Type
abstract type AbstractDenoiseEstimator <: AbstractEstimator

Abstract supertype for all denoising estimator types.

All concrete and/or abstract types that implement denoising of covariance-like or correlation-like matrices should be subtypes of AbstractDenoiseEstimator.

Interfaces

In order to implement a new denoising estimator which will work seamlessly with the library, subtype AbstractDenoiseEstimator with all necessary parameters as part of the struct, and implement the following methods:

  • denoise!(dn::AbstractDenoiseEstimator, X::MatNum, q::Number) -> MatNum: In-place denoising.
  • denoise(dn::AbstractDenoiseEstimator, X::MatNum, q::Number) -> MatNum: Optional out-of-place denoising. A fallback method copies X and calls denoise!, so it is only needed if the copy can be avoided.

Arguments

  • dn: Matrix denoising estimator.
  • X: Covariance-like or correlation-like matrix assets × assets.
  • q: The effective sample ratio observations / assets, used for spectral thresholding.

Returns

  • X::MatNum: The denoised input matrix X.

Examples

We can create a dummy denoising estimator as follows:

julia> struct MyDenoiseEstimator <: PortfolioOptimisers.AbstractDenoiseEstimator endjulia> function PortfolioOptimisers.denoise!(dn::MyDenoiseEstimator, X::PortfolioOptimisers.MatNum,                                             q::Number)           # Implement your in-place denoising estimator here.           println("Denoising matrix in-place...")           return X       endjulia> function PortfolioOptimisers.denoise(dn::MyDenoiseEstimator, X::PortfolioOptimisers.MatNum,                                            q::Number)           X = copy(X)           println("Copy X...")           denoise!(dn, X, q)           return X       endjulia> denoise!(MyDenoiseEstimator(), [1.0 2.0; 2.0 1.0], 2.0)Denoising matrix in-place...2×2 Matrix{Float64}: 1.0  2.0 2.0  1.0julia> denoise(MyDenoiseEstimator(), [1.0 2.0; 2.0 1.0], 2.0)Copy X...Denoising matrix in-place...2×2 Matrix{Float64}: 1.0  2.0 2.0  1.0

Related

source
PortfolioOptimisers.AbstractDenoiseAlgorithm — Type
abstract type AbstractDenoiseAlgorithm <: AbstractAlgorithm

Abstract supertype for all denoising algorithm types.

All concrete and/or abstract types that implement a specific denoising algorithm should be subtypes of AbstractDenoiseAlgorithm.

Interfaces

If you wish to implement a new denoising algorithm that works with an existing denoising estimator, subtype AbstractDenoiseAlgorithm, with all necessary parameters as part of the struct, and implement _denoise!, Denoise's internal in-place dispatch hook: _denoise!(alg::AbstractDenoiseAlgorithm, X::MatNum, vals::VecNum, vecs::MatNum, num_factors::Integer) -> MatNum, denoising a covariance or correlation matrix using the specific algorithm. The leading underscore marks it as reached only through denoise!/denoise's dispatch, never called directly by a user of the library.

Arguments

  • alg: Denoising algorithm.
  • X: Covariance-like or correlation-like matrix assets × assets.
  • vals: Eigenvalues of X, sorted in ascending order.
  • vecs: Corresponding eigenvectors of X.
  • num_factors: Number of eigenvalues to treat as noise.

Returns

  • X::MatNum: The input matrix X is modified in-place.

Examples

We can create a dummy denoising algorithm as follows:

julia> struct MyDenoiseAlgorithm <: PortfolioOptimisers.AbstractDenoiseAlgorithm endjulia> function PortfolioOptimisers._denoise!(dn::MyDenoiseAlgorithm,                                              X::PortfolioOptimisers.MatNum,                                              vals::PortfolioOptimisers.VecNum,                                              vecs::PortfolioOptimisers.MatNum,                                              num_factors::Integer)           # Implement your in-place denoising logic here.           println("Denoising matrix using custom algorithm...")           return X       endjulia> denoise!(Denoise(; alg = MyDenoiseAlgorithm()), [2.0 1.0; 1.0 2.0], 1 / 100)Denoising matrix using custom algorithm...2×2 Matrix{Float64}: 2.0  1.0 1.0  2.0julia> denoise(Denoise(; alg = MyDenoiseAlgorithm()), [2.0 1.0; 1.0 2.0], 1 / 100)Denoising matrix using custom algorithm...2×2 Matrix{Float64}: 2.0  1.0 1.0  2.0

Related

source
PortfolioOptimisers.SpectralDenoise — Type
struct SpectralDenoise <: AbstractDenoiseAlgorithm

Denoises by setting the noise eigenvalues to zero. This removes the principal components that random matrix theory attributes to noise, then rescales the reconstruction back to unit diagonal.

Mathematical definition

The noise eigenvalues are set to zero, the matrix is rebuilt from the resulting spectrum, and the result is rescaled to unit diagonal:

\[\begin{align} \tilde{\lambda}_i &= \begin{cases} 0 & \lambda_i \leq \lambda_+ \\ \lambda_i & \lambda_i > \lambda_+ \end{cases}\,, \\ \mathbf{C}_{\mathrm{signal}} &= \mathbf{V} \, \mathrm{Diag}(\tilde{\boldsymbol{\lambda}}) \, \mathbf{V}^\intercal\,, \\ \tilde{X}_{ij} &= \frac{(C_{\mathrm{signal}})_{ij}}{\sqrt{(C_{\mathrm{signal}})_{ii} \, (C_{\mathrm{signal}})_{jj}}}\,. \end{align}\]

Where:

  • $\tilde{\lambda}_i$: Denoised $i$-th eigenvalue.
  • $\lambda_i$: $i$-th eigenvalue of the input matrix.
  • $\lambda_+$: Marčenko-Pastur upper bound of the noise band. An eigenvalue is noise when $\lambda_i \leq \lambda_+$, and signal when $\lambda_i > \lambda_+$.
  • $\mathbf{V}$: Eigenvector matrix of the input.
  • $\mathbf{C}_{\mathrm{signal}}$: Reconstruction from the signal eigenpairs alone.
  • $\tilde{\mathbf{X}}$: Denoised matrix.

Zeroing the noise eigenvalues takes the diagonal of $\mathbf{C}_{\mathrm{signal}}$ below one, so the rescaling is not cosmetic: it changes every entry.

When every eigenvalue is at or below $\lambda_+$, $\mathbf{C}_{\mathrm{signal}}$ is the zero matrix and the rescaling is undefined. The denoised matrix is the identity in that case:

\[\tilde{\mathbf{X}} = \mathbf{I} \quad \textrm{if} \quad \lambda_i \leq \lambda_+ \quad \forall \, i\,.\]

No signal survives, so every asset keeps its own variance and no pair keeps a correlation. FixedDenoise returns the identity on the same input, so the two tags agree on this case.

Algorithm

The branch of _denoise! that this tag selects runs these steps.

  1. When num_factors equals length(vals), write the identity into X and return it. Every eigenvalue is noise, so steps 2 to 4 would divide zero by zero.
  2. Set the num_factors smallest entries of vals to zero. vals is sorted ascending, so those are the noise eigenvalues.
  3. Rebuild the matrix as vecs * Diagonal(vals) * transpose(vecs), which is the signal-only reconstruction $\mathbf{C}_{\mathrm{signal}}$.
  4. Rescale the reconstruction to unit diagonal with StatsBase.cov2cor, and write the result into X. The rescaling also sheds the round-off of the eigendecomposition, so this branch never pins the diagonal by hand.

Constructors

SpectralDenoise() -> SpectralDenoise

Examples

julia> SpectralDenoise()SpectralDenoise()

Related

References

  • [9] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.
  • [10] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).
  • [5] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 3.5.2.2, Equations 3.54 and 3.55.
source
PortfolioOptimisers.FixedDenoise — Type
struct FixedDenoise <: AbstractDenoiseAlgorithm

Denoises by replacing the noise eigenvalues with their own mean. This flattens the principal components that random matrix theory attributes to noise, rather than discarding them, then rescales the reconstruction back to unit diagonal.

Mathematical definition

The noise eigenvalues are replaced by their own mean, the matrix is rebuilt from the flattened spectrum, and the result is rescaled to unit diagonal:

\[\begin{align} \tilde{\lambda}_i &= \begin{cases} \bar{\lambda}_\text{noise} & \lambda_i \leq \lambda_+ \\ \lambda_i & \lambda_i > \lambda_+ \end{cases}\,, \\ \mathbf{C} &= \mathbf{V} \, \mathrm{Diag}(\tilde{\boldsymbol{\lambda}}) \, \mathbf{V}^\intercal\,, \\ \tilde{X}_{ij} &= \frac{C_{ij}}{\sqrt{C_{ii} C_{jj}}}\,. \end{align}\]

Where:

  • $\tilde{\lambda}_i$: Denoised $i$-th eigenvalue.
  • $\lambda_i$: $i$-th eigenvalue of the input matrix.
  • $\lambda_+$: Marčenko-Pastur upper bound of the noise band. An eigenvalue is noise when $\lambda_i \leq \lambda_+$, and signal when $\lambda_i > \lambda_+$.
  • $\bar{\lambda}_\text{noise}$: Mean of the noise eigenvalues.
  • $\mathbf{V}$: Eigenvector matrix of the input.
  • $\mathbf{C}$: Reconstruction from the flattened spectrum.
  • $\tilde{\mathbf{X}}$: Denoised matrix.

Flattening the noise eigenvalues preserves the trace but not the diagonal, so the rescaling is not cosmetic: it changes every entry.

Algorithm

The branch of _denoise! that this tag selects runs these steps.

  1. Replace the num_factors smallest entries of vals by their own mean. vals is sorted ascending, so those are the noise eigenvalues.
  2. Rebuild the matrix as vecs * Diagonal(vals) * transpose(vecs), which is the reconstruction $\mathbf{C}$ from the flattened spectrum.
  3. Rescale the reconstruction to unit diagonal with StatsBase.cov2cor, and write the result into X. The rescaling also sheds the round-off of the eigendecomposition, so this branch never pins the diagonal by hand.

Constructors

FixedDenoise() -> FixedDenoise

Examples

julia> FixedDenoise()FixedDenoise()

Related

References

  • [9] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.
  • [10] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).
  • [5] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 3.5.2.1, Equations 3.52 and 3.53.
source
PortfolioOptimisers.ShrunkDenoise — Type
struct ShrunkDenoise{__T_alpha} <: AbstractDenoiseAlgorithm

Denoises by shrinking the off-diagonal part of the noise block towards zero, keeping its diagonal whole. The polarity of alpha is the reverse of the reading its name suggests: alpha is the weight kept on that off-diagonal part, so alpha = 0 is total shrinkage and alpha = 1 returns the input unchanged. The default alpha = 0.0 is therefore total shrinkage.

Mathematical definition

The spectrum is split at the Marčenko-Pastur upper bound. The signal block is rebuilt whole, and only the off-diagonal part of the noise block is scaled by $\alpha$:

\[\begin{align} \mathbf{C}_{\mathrm{signal}} &= \mathbf{V}_{\mathrm{signal}} \, \mathrm{Diag}(\boldsymbol{\lambda}_{\mathrm{signal}}) \, \mathbf{V}_{\mathrm{signal}}^\intercal\,, \\ \mathbf{C}_{\mathrm{noise}} &= \mathbf{V}_{\mathrm{noise}} \, \mathrm{Diag}(\boldsymbol{\lambda}_{\mathrm{noise}}) \, \mathbf{V}_{\mathrm{noise}}^\intercal\,, \\ \tilde{\mathbf{X}} &= \mathbf{C}_{\mathrm{signal}} + \alpha \mathbf{C}_{\mathrm{noise}} + (1 - \alpha) \, \mathrm{Diag}(\mathbf{C}_{\mathrm{noise}})\,. \end{align}\]

Where:

  • $\lambda_i$: $i$-th eigenvalue of the input matrix.
  • $\lambda_+$: Marčenko-Pastur upper bound of the noise band. An eigenvalue is noise when $\lambda_i \leq \lambda_+$, and signal when $\lambda_i > \lambda_+$.
  • $\mathbf{V}_{\mathrm{signal}}$: Eigenvector block of the signal eigenpairs.
  • $\mathbf{V}_{\mathrm{noise}}$: Eigenvector block of the noise eigenpairs.
  • $\boldsymbol{\lambda}_{\mathrm{signal}}$: Signal eigenvalues.
  • $\boldsymbol{\lambda}_{\mathrm{noise}}$: Noise eigenvalues.
  • $\mathbf{C}_{\mathrm{signal}}$: Reconstruction from the signal eigenpairs alone.
  • $\mathbf{C}_{\mathrm{noise}}$: Reconstruction from the noise eigenpairs alone.
  • $\alpha \in [0, 1]$: Weight kept on the off-diagonal part of the noise block. $\alpha = 0$ keeps only its diagonal, which is total shrinkage. $\alpha = 1$ keeps the block whole, so $\tilde{\mathbf{X}} = \mathbf{X}$.
  • $\tilde{\mathbf{X}}$: Denoised matrix.

The two $\alpha$ weights sum to one on the diagonal, so the reconstruction preserves it in exact arithmetic.

Algorithm

The branch of _denoise! that this tag selects runs these steps.

  1. Split vals and vecs at num_factors. The first num_factors entries are the noise block vals_l and vecs_l, and the rest are the signal block vals_r and vecs_r.
  2. Build corr0 from the signal block, which is $\mathbf{C}_{\mathrm{signal}}$.
  3. Build corr1 from the noise block, which is $\mathbf{C}_{\mathrm{noise}}$.
  4. Write corr0 + alpha * corr1 + (1 - alpha) * Diagonal(corr1) into X.
  5. Set the diagonal of X to one. The reconstruction preserves the diagonal in exact arithmetic, so this step sheds the round-off of the eigendecomposition. This branch reconstructs directly rather than through StatsBase.cov2cor, so it is the only branch that must pin its own diagonal.

Fields

  • alpha: Weight kept on the off-diagonal part of the noise block, $\alpha \in [0, 1]$. It is the weight kept, not the weight removed: 0 keeps only the diagonal of that block, which is total shrinkage, and 1 keeps the block whole, which returns the input unchanged.

Constructors

ShrunkDenoise(;    alpha::Number = 0.0,) -> ShrunkDenoise

Keywords correspond to the struct's fields.

Validation

  • 0 <= alpha <= 1.

Examples

julia> ShrunkDenoise(; alpha = 0.5)ShrunkDenoise  alpha ┴ Float64: 0.5

Related

References

  • [9] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.
  • [10] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).
  • [5] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 3.5.2.3, Equation 3.56.
source
PortfolioOptimisers.Denoise — Type
struct Denoise{__T_pdm, __T_alg, __T_args, __T_kwargs, __T_kernel, __T_m, __T_n} <: AbstractDenoiseEstimator

Configures and applies denoising algorithms to covariance or correlation matrices.

Denoise encapsulates all parameters required for matrix denoising in denoise! and denoise, allowing users to specify the denoising algorithm, optimization parameters, kernel settings for density estimation, and optional positive definite matrix projection.

Fields

  • pdm: Optional positive definite matrix estimator.
  • alg: Denoising algorithm.

Constructors

Denoise(;    pdm::Option{<:AbstractPosdefEstimator} = Posdef(),    alg::AbstractDenoiseAlgorithm = ShrunkDenoise(),    args::Tuple = (),    kwargs::NamedTuple = (;),    kernel = AverageShiftedHistograms.Kernels.gaussian,    m::Integer = 10,    n::Integer = 1000) -> Denoise

Keywords correspond to the struct's fields.

Validation

  • m > 1.
  • n > 1.

Examples

julia> Denoise()Denoise     pdm ┼ Posdef         │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton         │   kwargs ┴ @NamedTuple{}: NamedTuple()     alg ┼ ShrunkDenoise         │   alpha ┴ Float64: 0.0    args ┼ Tuple{}: ()  kwargs ┼ @NamedTuple{}: NamedTuple()  kernel ┼ typeof(AverageShiftedHistograms.Kernels.gaussian): AverageShiftedHistograms.Kernels.gaussian       m ┼ Int64: 10       n ┴ Int64: 1000julia> Denoise(; alg = SpectralDenoise(), m = 20, n = 500)Denoise     pdm ┼ Posdef         │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton         │   kwargs ┴ @NamedTuple{}: NamedTuple()     alg ┼ SpectralDenoise()    args ┼ Tuple{}: ()  kwargs ┼ @NamedTuple{}: NamedTuple()  kernel ┼ typeof(AverageShiftedHistograms.Kernels.gaussian): AverageShiftedHistograms.Kernels.gaussian       m ┼ Int64: 20       n ┴ Int64: 500

Related

References

  • [9] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.
  • [10] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).
  • [5] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 3.5.2.
source
PortfolioOptimisers.denoise! — Function
denoise!(dn::Option{<:AbstractDenoiseEstimator}, X::MatNum, q::Number) -> MatNum

In-place denoising of a covariance or correlation matrix using a Denoise estimator.

For matrices without unit diagonal, the function converts them into correlation matrices i.e. matrices with unit diagonal, applies the algorithm, and rescales them back.

Mathematical definition

The spectrum of $\mathbf{X}$ is split at the Marčenko-Pastur upper bound:

\[\begin{align} \lambda_{+} &= \sigma^2 \left(1 + \sqrt{\frac{1}{q}}\right)^2\,. \end{align}\]

Where:

  • $\lambda_+$: Marčenko-Pastur upper bound of the noise band. An eigenvalue is noise when $\lambda_i \leq \lambda_+$, and signal when $\lambda_i > \lambda_+$.
  • $\sigma^2$: Variance attributed to noise. A correlation matrix has $\sigma^2 = 1$.
  • $q = T/N$: Effective sample ratio, observations to assets.

The split is the whole of the mathematics that this function contributes. What happens to each side of it is the closed form of dn.alg.

Algorithm

  1. Check that X is square.
  2. Read the diagonal of X into s. When any entry of s is not one, X is a covariance matrix: replace s with its square roots and convert X to a correlation matrix with StatsBase.cov2cor!. The test is any(!isone, s), so it is the value of the diagonal that decides, never the type of X.
  3. Eigendecompose X, giving the ascending eigenvalues vals and the eigenvectors vecs.
  4. Fit the Marčenko-Pastur density to vals with find_max_eval, giving max_val, the upper edge of the noise band.
  5. Count the eigenvalues that do not exceed max_val, giving num_factors, the number of noise eigenvalues.
  6. Rebuild X from the split spectrum with _denoise!, through the branch that dn.alg selects.
  7. Repair the rebuilt matrix with posdef!, under dn.pdm.
  8. When step 2 converted a covariance matrix, convert X back with StatsBase.cor2cov!. The standard deviations are the ones read in step 2, so the original diagonal returns exactly.

Arguments

  • dn: Optional matrix denoising estimator.
    • ::Denoise: The specified denoising algorithm is applied to X in-place.
    • ::Nothing: No-op.
  • X: Covariance-like or correlation-like matrix assets × assets.
  • q: The effective sample ratio observations / assets, used for spectral thresholding.

Validation

  • X is square, checked with assert_matrix_issquare. The ::Nothing method returns before the check, so a dn of nothing accepts any X.

Returns

  • X::MatNum: The input matrix X is modified in-place.

Examples

julia> using StableRNGsjulia> rng = StableRNG(123456789);julia> X = rand(rng, 10, 5);julia> X = X' * X5×5 Matrix{Float64}: 3.29494  2.0765   1.73334  2.01524  1.77493 2.0765   2.46967  1.39953  1.97242  2.07886 1.73334  1.39953  1.90712  1.17071  1.30459 2.01524  1.97242  1.17071  2.24818  1.87091 1.77493  2.07886  1.30459  1.87091  2.44414julia> denoise!(Denoise(), X, 10 / 5)5×5 Matrix{Float64}: 3.29494  2.28883  1.70633  2.12343  2.17377 2.28883  2.46967  1.59575  1.98583  2.0329 1.70633  1.59575  1.90712  1.48044  1.51553 2.12343  1.98583  1.48044  2.24818  1.886 2.17377  2.0329   1.51553  1.886    2.44414

Related

References

  • [9] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.
  • [10] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).
  • [5] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 3.5.2.
source
PortfolioOptimisers.denoise — Function
denoise(dn::Option{<:AbstractDenoiseEstimator}, X::MatNum, q::Number) -> MatNum

Out-of-place version of denoise!.

Algorithm

  1. Copy X.
  2. Apply denoise! to the copy, and return it. The input is never modified.

Arguments

  • dn: Optional matrix denoising estimator.
    • ::Denoise: The specified denoising algorithm is applied to a copy of X.
    • ::Nothing: No-op, returns X unchanged.
  • X: Covariance-like or correlation-like matrix assets × assets.
  • q: The effective sample ratio observations / assets, used for spectral thresholding.

Returns

  • X::MatNum: A new matrix equal to the denoised version of the input.

Examples

julia> using StableRNGsjulia> rng = StableRNG(123456789);julia> X = rand(rng, 10, 5);       X = X' * X;julia> Xd = denoise(Denoise(), X, 10 / 5);julia> size(Xd)(5, 5)

Related

References

  • [9] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.
  • [10] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).
  • [5] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 3.5.2.
source

References

[5]
D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025).
[9]
M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020).
[10]
V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).