Denoise: private API

find_max_eval(
    vals::VecNum,
    q::Number,
    kernel::Any = AverageShiftedHistograms.Kernels.gaussian,
    m::Integer = 10,
    n::Integer = 1000,
    args::Tuple = (),
    kwargs::NamedTuple = (;)
) -> Number

Estimate the upper edge of the Marčenko–Pastur (MP) distribution for a set of eigenvalues, used to separate signal from noise in random matrix denoising.

This function fits the MP distribution to the observed spectrum by minimizing the sum of squared errors between the empirical and theoretical densities, and returns the estimated maximum eigenvalue for noise.

Three traps follow from the shape of that fit. The fitted variance is bounded above by one, so a spectrum whose noise variance exceeds one fits at the boundary and the returned edge is the unit-variance edge; a correlation matrix has unit variance by construction, which is the case this bound is written for. A spectrum whose eigenvalues are all equal carries no fit, because its range is a single point; such a matrix is a multiple of the identity, so it holds no signal to separate. A search that does not converge substitutes a unit variance and warns, so a caller can tell a fitted edge from a fallback edge; only args and kwargs can make the search fail, and the defaults converge.

Mathematical definition

For an effective sample ratio $q = T/N$ and a noise variance $\sigma^2$, the Marčenko-Pastur density and its support are

\[\begin{align} f(\lambda) &= \begin{cases} \dfrac{q \sqrt{(\lambda_+ - \lambda)(\lambda - \lambda_-)}}{2 \pi \lambda \sigma^2} & \lambda \in [\lambda_-, \lambda_+] \\ 0 & \text{otherwise} \end{cases}\,, \\ \lambda_{\pm} &= \sigma^2 \left(1 \pm \sqrt{\frac{1}{q}}\right)^2\,. \end{align}\]

The noise variance is the minimiser of the sum of squared errors between that density and an average shifted histogram estimate of the density of the observed eigenvalues,

\[\begin{align} \hat{\sigma}^2 &= \underset{\sigma^2 \in [0, 1]}{\arg\min} \sum_{i=1}^{n} \left(\hat{f}(\lambda_i) - f(\lambda_i)\right)^2\,, \\ \hat{\lambda}_{+} &= \hat{\sigma}^2 \left(1 + \sqrt{\frac{1}{q}}\right)^2\,. \end{align}\]

Where:

• $f$: Theoretical Marčenko-Pastur density.
• $\hat{f}$: Average shifted histogram estimate of the density of the observed eigenvalues.
• $\lambda_{\pm}$: Upper and lower edges of the support of $f$.
• $\hat{\lambda}_{+}$: Fitted upper edge, which is the value returned.
• $\sigma^2$: Variance attributed to noise. A correlation matrix has $\sigma^2 = 1$.
• $q = T/N$: Effective sample ratio, observations to assets.
• $n$: Number of grid points, which is the argument n.
• $T$: Number of observations.
• $N$: Number of assets.

Algorithm

1. Compute the two edge factors of a unit variance, op_sqrt_iq_sq for $\lambda_+$ and om_sqrt_iq_sq for $\lambda_-$.
2. Estimate the density of vals with AverageShiftedHistograms.ash, under kernel and m, over the range of vals itself. When that range is a single point, span [v, v + 1] instead, because the estimator needs a range it can bin. The estimate is built once, so its support is fixed and it does not depend on the trial variance. An estimate whose support followed the trial variance would renormalise over a shrinking window, which gives the objective a spurious local minimum well below the true variance.
3. Define the objective on a trial variance x. Steps 4 to 7 are its body.
4. Scale the two edge factors by x, giving e_min and e_max, and place n equally spaced points over [e_min, e_max], giving rg.
5. Evaluate the theoretical density on rg, giving column 1 of pdf. The product under the square root is clamped at zero, so a round-off outside the support gives zero rather than a domain error.
6. Read the estimate of step 2 on rg, the same abscissa as column 1, giving column 2 of pdf. Replace a non-finite entry by zero.
7. Return the sum of the squared differences of the two columns.
8. Minimise the objective over x in [0, 1] with Optim.optimize, under args and kwargs.
9. Take x as the minimiser when the search converged. When it did not, warn and substitute x = 1, the variance of a correlation matrix, so the returned edge is the unit-variance edge exactly.
10. Return x * op_sqrt_iq_sq, the fitted upper edge.

Arguments

• vals: Observed eigenvalues (typically sorted in ascending order).
• q: Effective sample ratio (e.g., n_obs / n_assets).
• kernel: Kernel function for AverageShiftedHistograms.ash.
• m: Number of adjacent histograms to smooth over.
• n: Number of points in the range of eigenvalues for density estimation.
• args: Additional positional arguments for Optim.optimize.
• kwargs: Additional keyword arguments for Optim.optimize.

Returns

• e_max::Number: Estimated upper edge of the noise eigenvalue spectrum.

Related

• Denoise
• VecNum
• AverageShiftedHistograms.Kernels

References

• [10] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).
• [9] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.
• [5] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 3.5.1, Equation 3.51.

_denoise!(
    alg::AbstractDenoiseAlgorithm,
    X::MatNum,
    vals::VecNum,
    vecs::MatNum,
    num_factors::Integer
) -> MatNum

In-place denoising of a correlation matrix using a specific denoising algorithm.

These methods are called internally by denoise! and denoise when a Denoise estimator is used, and should not typically be called directly.

Algorithm

The method that Julia selects is the algorithm. vals is sorted ascending, so the first num_factors entries are the noise eigenvalues and the rest are the signal eigenvalues.

1. alg is a SpectralDenoise: zero the noise eigenvalues, rebuild from the signal components alone, and rescale to unit diagonal with StatsBase.cov2cor. When every eigenvalue is noise, the reconstruction is the zero matrix and the rescaling is undefined, so this branch writes the identity instead.
2. alg is a FixedDenoise: replace the noise eigenvalues by their own mean, rebuild from the flattened spectrum, and rescale to unit diagonal with StatsBase.cov2cor.
3. alg is a ShrunkDenoise: rebuild the two blocks separately, combine them under alg.alpha, and pin the diagonal to one. This branch does not route through StatsBase.cov2cor, so it is the only branch that pins its own diagonal.

Every branch writes into X and returns it.

Arguments

• alg: Denoising algorithm.
• X: Covariance-like or correlation-like matrix assets × assets.
• vals: Eigenvalues of X, sorted in ascending order.
• vecs: Corresponding eigenvectors of X.
• num_factors: Number of eigenvalues to treat as noise.

Returns

• X::MatNum: The input matrix X is modified in-place.

Related

• denoise!
• Denoise
• SpectralDenoise: the closed form of the branch of step 1.
• FixedDenoise: the closed form of the branch of step 2.
• ShrunkDenoise: the closed form of the branch of step 3.
• MatNum
• VecNum

References

• [9] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Chapter 2.
• [10] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).
• [5] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 3.5.2.