Denoise: private API

denoise! and denoise call the functions below. find_max_eval estimates the upper edge of the Marčenko-Pastur distribution, which separates the eigenvalues of noise from those of signal. A new denoising algorithm adds a method to _denoise!.

PortfolioOptimisers.find_max_eval — Function
find_max_eval(
    vals::VecNum,
    q::Number,
    kernel::Any = AverageShiftedHistograms.Kernels.gaussian,
    m::Integer = 10,
    n::Integer = 1000,
    args::Tuple = (),
    kwargs::NamedTuple = (;)
) -> Number

Estimates the upper edge of the Marčenko-Pastur distribution of a set of eigenvalues. The edge separates the signal eigenvalues from the noise eigenvalues.

The function fits the Marčenko-Pastur density to the observed spectrum. It minimises the sum of squared errors between the empirical and the theoretical densities, and returns the largest eigenvalue of the noise.

The fit has these limits:

  • The fitted variance is at most one. A spectrum whose noise variance is larger than one fits at the boundary, and the function returns the unit-variance edge. A correlation matrix has unit variance by construction, and the bound is for that case.
  • A spectrum whose eigenvalues are all equal has no fit, because its range is a single point. Such a matrix is a multiple of the identity, so it holds no signal to separate.
  • When the search does not converge, the function warns and uses a unit variance, so a caller can tell a fitted edge from a fallback edge. Only args and kwargs can make the search fail, and the defaults converge.

Mathematical definition

For an effective sample ratio $q = T/N$ and a noise variance $\sigma^2$, the Marčenko-Pastur density and its support are

\[\begin{align} f(\lambda) &= \begin{cases} \dfrac{q \sqrt{(\lambda_+ - \lambda)(\lambda - \lambda_-)}}{2 \pi \lambda \sigma^2} & \lambda \in [\lambda_-, \lambda_+] \\ 0 & \text{otherwise} \end{cases}\,, \\ \lambda_{\pm} &= \sigma^2 \left(1 \pm \sqrt{\frac{1}{q}}\right)^2\,. \end{align}\]

The noise variance is the minimiser of the sum of squared errors between that density and an average shifted histogram estimate of the density of the observed eigenvalues,

\[\begin{align} \hat{\sigma}^2 &= \underset{\sigma^2 \in [0, 1]}{\arg\min} \sum_{i=1}^{n} \left(\hat{f}(\lambda_i) - f(\lambda_i)\right)^2\,, \\ \hat{\lambda}_{+} &= \hat{\sigma}^2 \left(1 + \sqrt{\frac{1}{q}}\right)^2\,. \end{align}\]

Where:

  • $f$: The theoretical Marčenko-Pastur density.
  • $\hat{f}$: The average shifted histogram estimate of the density of the observed eigenvalues.
  • $\lambda_{\pm}$: The upper and the lower edges of the support of $f$.
  • $\hat{\lambda}_{+}$: The fitted upper edge, which the function returns.
  • $\sigma^2$: Variance attributed to noise. A correlation matrix has $\sigma^2 = 1$.
  • $q = T/N$: Effective sample ratio, observations to assets.
  • $n$: The number of grid points, which is the argument n.
  • $T$: Number of observations.
  • $N$: Number of assets.

The density is the unnumbered display of Example 1 of Section 1 of [27], with their concentration $c = q$ and their scale $\tau = \sigma^2 / q$. Equation 3 of [29] states it in the form above. That work treats $\sigma^2$ as a parameter to fit, and it names no criterion for the fit. Snippet 2 of Section 4.2 of [30] first states the fit by the sum of squared errors, and Section 2.4 of [26] restates it. That fit reads a Gaussian kernel density estimate of the eigenvalues on the grid of the theoretical density, searches $\sigma^2 \in [10^{-5}, 1 - 10^{-5}]$, and takes $\sigma^2 = 1$ when the search fails. This function reads an average shifted histogram instead of a kernel density estimate, and it searches $[0, 1]$.

Algorithm

  1. Compute the two edge factors of a unit variance, op_sqrt_iq_sq for $\lambda_+$ and om_sqrt_iq_sq for $\lambda_-$.
  2. Estimate the density of vals with AverageShiftedHistograms.ash, under kernel and m, over the range of vals itself. When that range is a single point, span [v, v + 1] instead, because the estimator needs a range it can bin. The function builds the estimate once, so its support is fixed and does not depend on the trial variance. A support that moves with the trial variance makes the estimate renormalise over a smaller window, and the objective then gets a false local minimum far below the true variance.
  3. Define the objective on a trial variance x. Steps 4 to 7 are its body.
  4. Scale the two edge factors by x to get e_min and e_max. Put n equally spaced points over [e_min, e_max] into rg.
  5. Evaluate the theoretical density on rg into column 1 of pdf. The function clamps the product under the square root at zero, so a round-off outside the support gives zero and not a domain error.
  6. Read the estimate of step 2 on rg, the same abscissa as column 1, into column 2 of pdf. Replace a non-finite entry by zero.
  7. Return the sum of the squared differences of the two columns.
  8. Minimise the objective over x in [0, 1] with Optim.optimize, under args and kwargs.
  9. Take x as the minimiser when the search converged. When it did not, warn and substitute x = 1, the variance of a correlation matrix, so the returned edge is the unit-variance edge exactly.
  10. Return x * op_sqrt_iq_sq, the fitted upper edge.

Arguments

  • vals: The observed eigenvalues. denoise! passes them in ascending order.
  • q: The effective sample ratio observations / assets.
  • kernel: The kernel function of AverageShiftedHistograms.ash.
  • m: The number of adjacent histograms to smooth over.
  • n: The number of points in the range of eigenvalues for the density estimate.
  • args: The positional arguments of Optim.optimize.
  • kwargs: The keyword arguments of Optim.optimize.

Returns

  • e_max::Number: The estimated upper edge of the spectrum of the noise eigenvalues.

Related

References

  • [27] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967). Section 1, Example 1.
  • [29] L. Laloux, P. Cizeau, J.-P. Bouchaud and M. Potters. Noise dressing of financial correlation matrices. Phys. Rev. Lett. 83, 1467–1470 (1999). Equation 3.
  • [30] M. López de Prado. A robust estimator of the efficient frontier. SSRN Electronic Journal (2019). Section 4.2, Snippet 2.
  • [26] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Section 2.4, Snippet 2.4.
source
PortfolioOptimisers._denoise! — Function
_denoise!(
    alg::AbstractDenoiseAlgorithm,
    X::MatNum,
    vals::VecNum,
    vecs::MatNum,
    num_factors::Integer
) -> MatNum

Denoises a correlation matrix in place under one denoising algorithm.

denoise! and denoise call these methods when they run a Denoise estimator. A user does not call them directly.

Algorithm

The method that Julia selects is the algorithm. vals is sorted ascending, so the first num_factors entries are the noise eigenvalues and the rest are the signal eigenvalues.

  1. alg is a SpectralDenoise: zero the noise eigenvalues, rebuild from the signal components alone, and rescale to unit diagonal with StatsBase.cov2cor. When every eigenvalue is noise, the reconstruction is the zero matrix and the rescaling is undefined, so this branch writes the identity instead.
  2. alg is a FixedDenoise: replace the noise eigenvalues by their own mean, rebuild from the flattened spectrum, and rescale to unit diagonal with StatsBase.cov2cor.
  3. alg is a ShrunkDenoise: rebuild the two blocks separately, combine them under alg.alpha, and pin the diagonal to one. This branch does not route through StatsBase.cov2cor, so it is the only branch that pins its own diagonal.

Every branch writes into X and returns it.

Arguments

  • alg: The denoising algorithm.
  • X: Covariance-like or correlation-like matrix assets × assets.
  • vals: The eigenvalues of X, in ascending order.
  • vecs: The eigenvectors of X, in the order of vals.
  • num_factors: The number of eigenvalues that are noise.

Returns

  • X::MatNum: The input matrix X, modified in place.

Related

References

  • [28] V. Plerou, P. Gopikrishnan, B. Rosenow, L. A. Amaral, T. Guhr and H. E. Stanley. Random matrix approach to cross correlations in financial data. Phys. Rev. E 65, 066126 (2002). Section VIII.
  • [29] L. Laloux, P. Cizeau, J.-P. Bouchaud and M. Potters. Noise dressing of financial correlation matrices. Phys. Rev. Lett. 83, 1467–1470 (1999). Page 1469.
  • [30] M. López de Prado. A robust estimator of the efficient frontier. SSRN Electronic Journal (2019). Section 4.2, Snippet 3.
  • [26] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Sections 2.5.1 and 2.5.2.
  • [12] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 3.5.2.2, Equation 3.55. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
source

References

[12]
D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025).
[26]
M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020).
[27]
V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).
[28]
V. Plerou, P. Gopikrishnan, B. Rosenow, L. A. Amaral, T. Guhr and H. E. Stanley. Random matrix approach to cross correlations in financial data. Phys. Rev. E 65, 066126 (2002).
[29]
L. Laloux, P. Cizeau, J.-P. Bouchaud and M. Potters. Noise dressing of financial correlation matrices. Phys. Rev. Lett. 83, 1467–1470 (1999).
[30]
M. López de Prado. A robust estimator of the efficient frontier. SSRN Electronic Journal (2019).