Denoise: private API
denoise! and denoise call the functions below. find_max_eval estimates the upper edge of the Marčenko-Pastur distribution, which separates the eigenvalues of noise from those of signal. A new denoising algorithm adds a method to _denoise!.
PortfolioOptimisers.find_max_eval — Function
find_max_eval(
vals::VecNum,
q::Number,
kernel::Any = AverageShiftedHistograms.Kernels.gaussian,
m::Integer = 10,
n::Integer = 1000,
args::Tuple = (),
kwargs::NamedTuple = (;)
) -> NumberEstimates the upper edge of the Marčenko-Pastur distribution of a set of eigenvalues. The edge separates the signal eigenvalues from the noise eigenvalues.
The function fits the Marčenko-Pastur density to the observed spectrum. It minimises the sum of squared errors between the empirical and the theoretical densities, and returns the largest eigenvalue of the noise.
The fit has these limits:
- The fitted variance is at most one. A spectrum whose noise variance is larger than one fits at the boundary, and the function returns the unit-variance edge. A correlation matrix has unit variance by construction, and the bound is for that case.
- A spectrum whose eigenvalues are all equal has no fit, because its range is a single point. Such a matrix is a multiple of the identity, so it holds no signal to separate.
- When the search does not converge, the function warns and uses a unit variance, so a caller can tell a fitted edge from a fallback edge. Only
argsandkwargscan make the search fail, and the defaults converge.
Mathematical definition
For an effective sample ratio $q = T/N$ and a noise variance $\sigma^2$, the Marčenko-Pastur density and its support are
\[\begin{align} f(\lambda) &= \begin{cases} \dfrac{q \sqrt{(\lambda_+ - \lambda)(\lambda - \lambda_-)}}{2 \pi \lambda \sigma^2} & \lambda \in [\lambda_-, \lambda_+] \\ 0 & \text{otherwise} \end{cases}\,, \\ \lambda_{\pm} &= \sigma^2 \left(1 \pm \sqrt{\frac{1}{q}}\right)^2\,. \end{align}\]
The noise variance is the minimiser of the sum of squared errors between that density and an average shifted histogram estimate of the density of the observed eigenvalues,
\[\begin{align} \hat{\sigma}^2 &= \underset{\sigma^2 \in [0, 1]}{\arg\min} \sum_{i=1}^{n} \left(\hat{f}(\lambda_i) - f(\lambda_i)\right)^2\,, \\ \hat{\lambda}_{+} &= \hat{\sigma}^2 \left(1 + \sqrt{\frac{1}{q}}\right)^2\,. \end{align}\]
Where:
- $f$: The theoretical Marčenko-Pastur density.
- $\hat{f}$: The average shifted histogram estimate of the density of the observed eigenvalues.
- $\lambda_{\pm}$: The upper and the lower edges of the support of $f$.
- $\hat{\lambda}_{+}$: The fitted upper edge, which the function returns.
- $\sigma^2$: Variance attributed to noise. A correlation matrix has $\sigma^2 = 1$.
- $q = T/N$: Effective sample ratio, observations to assets.
- $n$: The number of grid points, which is the argument
n. - $T$: Number of observations.
- $N$: Number of assets.
The density is the unnumbered display of Example 1 of Section 1 of [27], with their concentration $c = q$ and their scale $\tau = \sigma^2 / q$. Equation 3 of [29] states it in the form above. That work treats $\sigma^2$ as a parameter to fit, and it names no criterion for the fit. Snippet 2 of Section 4.2 of [30] first states the fit by the sum of squared errors, and Section 2.4 of [26] restates it. That fit reads a Gaussian kernel density estimate of the eigenvalues on the grid of the theoretical density, searches $\sigma^2 \in [10^{-5}, 1 - 10^{-5}]$, and takes $\sigma^2 = 1$ when the search fails. This function reads an average shifted histogram instead of a kernel density estimate, and it searches $[0, 1]$.
Algorithm
- Compute the two edge factors of a unit variance,
op_sqrt_iq_sqfor $\lambda_+$ andom_sqrt_iq_sqfor $\lambda_-$. - Estimate the density of
valswithAverageShiftedHistograms.ash, underkernelandm, over the range ofvalsitself. When that range is a single point, span[v, v + 1]instead, because the estimator needs a range it can bin. The function builds the estimate once, so its support is fixed and does not depend on the trial variance. A support that moves with the trial variance makes the estimate renormalise over a smaller window, and the objective then gets a false local minimum far below the true variance. - Define the objective on a trial variance
x. Steps 4 to 7 are its body. - Scale the two edge factors by
xto gete_minande_max. Putnequally spaced points over[e_min, e_max]intorg. - Evaluate the theoretical density on
rginto column 1 ofpdf. The function clamps the product under the square root at zero, so a round-off outside the support gives zero and not a domain error. - Read the estimate of step 2 on
rg, the same abscissa as column 1, into column 2 ofpdf. Replace a non-finite entry by zero. - Return the sum of the squared differences of the two columns.
- Minimise the objective over
xin[0, 1]withOptim.optimize, underargsandkwargs. - Take
xas the minimiser when the search converged. When it did not, warn and substitutex = 1, the variance of a correlation matrix, so the returned edge is the unit-variance edge exactly. - Return
x * op_sqrt_iq_sq, the fitted upper edge.
Arguments
vals: The observed eigenvalues.denoise!passes them in ascending order.q: The effective sample ratioobservations / assets.kernel: The kernel function of AverageShiftedHistograms.ash.m: The number of adjacent histograms to smooth over.n: The number of points in the range of eigenvalues for the density estimate.args: The positional arguments of Optim.optimize.kwargs: The keyword arguments of Optim.optimize.
Returns
e_max::Number: The estimated upper edge of the spectrum of the noise eigenvalues.
Related
References
- [27] V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967). Section 1, Example 1.
- [29] L. Laloux, P. Cizeau, J.-P. Bouchaud and M. Potters. Noise dressing of financial correlation matrices. Phys. Rev. Lett. 83, 1467–1470 (1999). Equation 3.
- [30] M. López de Prado. A robust estimator of the efficient frontier. SSRN Electronic Journal (2019). Section 4.2, Snippet 2.
- [26] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Section 2.4, Snippet 2.4.
PortfolioOptimisers._denoise! — Function
_denoise!(
alg::AbstractDenoiseAlgorithm,
X::MatNum,
vals::VecNum,
vecs::MatNum,
num_factors::Integer
) -> MatNumDenoises a correlation matrix in place under one denoising algorithm.
denoise! and denoise call these methods when they run a Denoise estimator. A user does not call them directly.
Algorithm
The method that Julia selects is the algorithm. vals is sorted ascending, so the first num_factors entries are the noise eigenvalues and the rest are the signal eigenvalues.
algis aSpectralDenoise: zero the noise eigenvalues, rebuild from the signal components alone, and rescale to unit diagonal withStatsBase.cov2cor. When every eigenvalue is noise, the reconstruction is the zero matrix and the rescaling is undefined, so this branch writes the identity instead.algis aFixedDenoise: replace the noise eigenvalues by their own mean, rebuild from the flattened spectrum, and rescale to unit diagonal withStatsBase.cov2cor.algis aShrunkDenoise: rebuild the two blocks separately, combine them underalg.alpha, and pin the diagonal to one. This branch does not route throughStatsBase.cov2cor, so it is the only branch that pins its own diagonal.
Every branch writes into X and returns it.
Arguments
alg: The denoising algorithm.X: Covariance-like or correlation-like matrixassets × assets.vals: The eigenvalues ofX, in ascending order.vecs: The eigenvectors ofX, in the order ofvals.num_factors: The number of eigenvalues that are noise.
Returns
X::MatNum: The input matrixX, modified in place.
Related
denoise!DenoiseSpectralDenoise: the closed form of the branch of step 1.FixedDenoise: the closed form of the branch of step 2.ShrunkDenoise: the closed form of the branch of step 3.MatNumVecNum
References
- [28] V. Plerou, P. Gopikrishnan, B. Rosenow, L. A. Amaral, T. Guhr and H. E. Stanley. Random matrix approach to cross correlations in financial data. Phys. Rev. E 65, 066126 (2002). Section VIII.
- [29] L. Laloux, P. Cizeau, J.-P. Bouchaud and M. Potters. Noise dressing of financial correlation matrices. Phys. Rev. Lett. 83, 1467–1470 (1999). Page 1469.
- [30] M. López de Prado. A robust estimator of the efficient frontier. SSRN Electronic Journal (2019). Section 4.2, Snippet 3.
- [26] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Sections 2.5.1 and 2.5.2.
- [12] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 3.5.2.2, Equation 3.55. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
References
- [12]
- D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025).
- [26]
- M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020).
- [27]
- V. A. Marčenko and L. A. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik 1, 457 (1967).
- [28]
- V. Plerou, P. Gopikrishnan, B. Rosenow, L. A. Amaral, T. Guhr and H. E. Stanley. Random matrix approach to cross correlations in financial data. Phys. Rev. E 65, 066126 (2002).
- [29]
- L. Laloux, P. Cizeau, J.-P. Bouchaud and M. Potters. Noise dressing of financial correlation matrices. Phys. Rev. Lett. 83, 1467–1470 (1999).
- [30]
- M. López de Prado. A robust estimator of the efficient frontier. SSRN Electronic Journal (2019).