Normal Uncertainty Sets: private API
PortfolioOptimisers.commutation_matrix — Function
commutation_matrix(X::MatNum)Builds the commutation matrix for the size of X.
The commutation matrix is the permutation that maps the column-major vectorisation of a matrix to the vectorisation of its transpose. For a square matrix, it is its own inverse. It holds one non-zero entry in each row, so the function returns a sparse matrix. The function reads the size of X, and it does not read the entries.
Mathematical definition
\[\mathbf{K}_{m,n} \operatorname{vec}(\mathbf{A}) = \operatorname{vec}\!\left(\mathbf{A}^{\intercal}\right)\,, \qquad \mathbf{A} \in \mathbb{R}^{m \times n}\,.\]
Where:
- $\mathbf{K}_{m,n}$: Commutation matrix of an $m \times n$ matrix, of size $mn \times mn$.
- $\mathbf{A}$: Any $m \times n$ matrix.
- $\operatorname{vec}$: Column-major vectorisation.
For a symmetric $\mathbf{A}$, $\mathbf{K}_{n,n} \operatorname{vec}(\mathbf{A}) = \operatorname{vec}(\mathbf{A})$. So a symmetric matrix cannot show the difference between $\mathbf{K}$ and the identity.
Algorithm
- Read
m, n = size(X), and setmn = m * n, the side of the result. - Number the entries of an
mbynmatrix in column-major order. This givesrow = 1:mn. - Find the position of each entry in the vectorised transpose with
col = vec(transpose(reshape(row, m, n))). - Write a one at each
(row, col)pair. The result is the sparsemnbymnmatrixcX.
Arguments
X: Matrix whose size sets the size of the commutation matrix.
Returns
cX::SparseArrays.SparseMatrixCSC{<:Number, <:Integer}: Sparse commutation matrix of size(m*n, m*n), wherem, n = size(X).
Examples
julia> PortfolioOptimisers.commutation_matrix(rand(2, 2))4×4 SparseArrays.SparseMatrixCSC{Float64, Int64} with 4 stored entries: 1.0 ⋅ ⋅ ⋅ ⋅ ⋅ 1.0 ⋅ ⋅ 1.0 ⋅ ⋅ ⋅ ⋅ ⋅ 1.0julia> PortfolioOptimisers.commutation_matrix(rand(2, 3))6×6 SparseArrays.SparseMatrixCSC{Float64, Int64} with 6 stored entries: 1.0 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 1.0 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 1.0 ⋅ ⋅ 1.0 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 1.0 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 1.0julia> PortfolioOptimisers.commutation_matrix(rand(3, 2))6×6 SparseArrays.SparseMatrixCSC{Float64, Int64} with 6 stored entries: 1.0 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 1.0 ⋅ ⋅ ⋅ 1.0 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 1.0 ⋅ ⋅ ⋅ 1.0 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ 1.0Related
References
- [169] J. R. Magnus and H. Neudecker. The Commutation Matrix: Some Properties and Applications. The Annals of Statistics 7, 381–394 (1979). Section 3, Definition 3.1 and Theorem 3.1 (vii). The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
- [305] D. S. Tracy and P. S. Dwyer. Multivariate Maxima and Minima with Matrix Derivatives. Journal of the American Statistical Association 64, 1576–1594 (1969).
PortfolioOptimisers.choose_scaling_parameter — Method
choose_scaling_parameter(ue, pr, k::Integer)Returns $T$, the effective number of observations of moment k of the prior. The covariance divided by $T$ is the asymptotic covariance of the mean.
The index k follows the field eoc of the prior: 1 is the mean and 2 is the covariance. A set of the mean reads k = 1, and a set of the covariance reads k = 2. A set of both reads each count for its own half, see normal_sigma_scaling.
The function reads two sources in order. The first is the ens of the estimator, a count that the caller states for every moment. The second is the count of the moment that the prior states, from moment_observation_count. A weighted prior has fewer effective observations than rows, and a prior under a Scenario Cap has fewer rows than the observations of its moments. The prior states both counts in eoc, and nothing there means the number of rows. The ens of the prior is a diagnostic of its weights, and the function does not read it.
Each set of this estimator scales with $T$. The width of a set is proportional to $T^{-1/2}$, so a prior with fewer effective observations than rows gives a wider set. A division of $T$ by four doubles the width of the mean box.
Algorithm
- If
ue.ensis notnothing, return it. It is the effective count that the estimator states. - Otherwise, return
moment_observation_countofprandk.
Arguments
ue:NormalUncertaintySetestimator.pr: Prior Result.k: The index of the moment,1for the mean and2for the covariance.
Returns
T::Number: Effective number of observations.
Related
PortfolioOptimisers.normal_sigma_scaling — Function
normal_sigma_scaling(ue::NormalUncertaintySet, pr::AbstractPriorResult, sigma::MatNum,
T_mu::Number, sigma_mu::MatNum) -> TupleReturns the scaling parameter of the covariance half of a joint set, and the mean asymptotic covariance at that parameter.
A joint method of ucs builds the mean half at the count of the mean, T_mu. The covariance half reads the count of the covariance from choose_scaling_parameter, and its shape reads the mean asymptotic covariance at that count. The two counts differ when the mean estimator and the covariance estimator of the prior carry different weights. When they are equal, the function returns sigma_mu itself, so the usual case computes nothing again.
Arguments
ue: Normal uncertainty set estimator.pr: Prior Result that the set fits on.sigma: The covariance ofpron the investable assets.T_mu: The scaling parameter of the mean half.sigma_mu: The mean asymptotic covariance atT_mu.
Returns
(T, sigma_mu): The scaling parameter of the covariance half, and the mean asymptotic covariance at it.
Related
PortfolioOptimisers.normal_mu_error_sample — Function
normal_mu_error_sample(ue::NormalUncertaintySet, rng::Random.AbstractRNG, mu::VecNum,
sigma_mu::MatNum)Draws ue.n_sim estimation errors of the mean from the sampling law of normal returns.
Each row of the sample is an estimation error, a draw minus mu. The draws come from $\mathcal{N}(\hat{\boldsymbol{\mu}}, \hat{\mathbf{\Sigma}}/T)$, so the sample and its shape matrix have the same scale. A draw from $\mathcal{N}(\hat{\boldsymbol{\mu}}, \hat{\mathbf{\Sigma}})$ multiplies every deviation by $\sqrt{T}$, so it also multiplies the radius by $\sqrt{T}$. The caller resolves the generator and gives it to the function. So a caller that needs both axes draws them from one stream.
Algorithm
- Draw
ue.n_simvectors fromDistributions.MvNormal(mu, sigma_mu). The result has one column for each draw. - Subtract
mufrom each column. Transpose the result, so that each row holds one estimation error.
Arguments
ue: Normal uncertainty set estimator.rng: Random number generator, which the caller resolves withresolve_rng.mu: Point estimate of the expected returns vector, the centre of the draws.sigma_mu: Mean asymptotic covariance frommu_asymptotic_cov.
Returns
X_mu::MatNum: Sampled estimation errors,ue.n_simbyN.
Related
PortfolioOptimisers.normal_sigma_error_sample — Function
normal_sigma_error_sample(ue::NormalUncertaintySet, rng::Random.AbstractRNG,
sigma::MatNum, sigma_mu::MatNum, T::Number, N::Integer)Draws ue.n_sim estimation errors of the vectorised covariance from the Wishart law of normal returns.
Each row of the sample is an estimation error, a draw minus sigma. The variance of an entry of a $\mathrm{Wishart}(T, \hat{\mathbf{\Sigma}}/T)$ draw is the matching diagonal entry of $\mathbf{\Sigma}_{\mathbf{\Sigma}}$, the asymptotic covariance of the covariance estimator. So the sample and its shape matrix have the same scale. The caller gives N, and the function does not read it from sigma. So every caller reads the asset count from one source, size(pr.X, 2).
Algorithm
- Draw
ue.n_simmatrices fromDistributions.Wishart(T, sigma_mu), one at a time into one buffer. - Subtract
sigmafrom each draw into anNbyNbyue.n_simarray. The function holds only this array and the buffer, and not all the draws. - Reshape the array to
N^2rows, with one column for each draw. Transpose it, so that each row holds one vectorised estimation error.
Arguments
ue: Normal uncertainty set estimator.rng: Random number generator, which the caller resolves withresolve_rng.sigma: Point estimate of the covariance matrix, the centre of the draws.sigma_mu: Mean asymptotic covariance frommu_asymptotic_cov, the scale of the Wishart law.T: Scaling parameter fromchoose_scaling_parameter, the degrees of freedom of the Wishart law.N: Asset count. The caller reads it fromsize(pr.X, 2).
Returns
X_sigma::MatNum: Sampled estimation errors,ue.n_simbyN^2.
Related
PortfolioOptimisers.mu_asymptotic_cov — Function
mu_asymptotic_cov(pdm, sigma, T)Returns the asymptotic covariance of the mean estimator, projected to the nearest positive definite matrix with pdm.
Every route of NormalUncertaintySet calls this function, so all routes use the same mean shape. A division by a positive scalar keeps the definiteness of sigma. So posdef! changes nothing when sigma is positive definite.
Mathematical definition
\[\mathbf{\Sigma}_{\boldsymbol{\mu}} = \dfrac{\hat{\mathbf{\Sigma}}}{T}\,.\]
Where:
- $\mathbf{\Sigma}_{\boldsymbol{\mu}}$: Asymptotic covariance of the mean estimator.
- $\hat{\mathbf{\Sigma}}$: Estimated covariance matrix.
- $T$: Number of observations.
Algorithm
- Divide
sigmabyTto getsigma_mu. - Repair
sigma_muin place withposdef!underpdm. - Return
sigma_mu.
Arguments
pdm: Positive definite matrix estimator.nothingskips the repair.sigma: Point estimate of the covariance matrix.T: Scaling parameter fromchoose_scaling_parameter.
Returns
sigma_mu::MatNum: Asymptotic covariance of the mean estimator.
Related
References
- [12] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Equations 11.16 and 11.24. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
PortfolioOptimisers.sigma_asymptotic_cov — Function
sigma_asymptotic_cov(pdm, sigma_mu, sigma, T)Returns the asymptotic covariance of the vectorised covariance estimator, projected to the nearest positive definite matrix with pdm.
sigma_mu is the full mean asymptotic covariance from mu_asymptotic_cov, before any diagonal step. $\mathbf{K}$ is the commutation_matrix of sigma. The two matrix arguments depend on each other. The form below is the sampling law of a normal sample only if sigma_mu comes from the same sigma and the same T.
Mathematical definition
\[\mathbf{\Sigma}_{\mathbf{\Sigma}} = T \left(\mathbf{I} + \mathbf{K}\right) \left(\mathbf{\Sigma}_{\boldsymbol{\mu}} \otimes \mathbf{\Sigma}_{\boldsymbol{\mu}}\right) = \dfrac{\left(\mathbf{I} + \mathbf{K}\right) \left(\hat{\mathbf{\Sigma}} \otimes \hat{\mathbf{\Sigma}}\right)}{T}\,.\]
Where:
- $\mathbf{\Sigma}_{\mathbf{\Sigma}}$: Asymptotic covariance of the vectorised covariance estimator.
- $\mathbf{\Sigma}_{\boldsymbol{\mu}}$: Asymptotic covariance of the mean estimator.
- $\mathbf{I}$: Identity matrix.
- $\mathbf{K}$: Commutation matrix.
- $\otimes$: Kronecker product.
- $\hat{\mathbf{\Sigma}}$: Estimated covariance matrix.
- $T$: Number of observations.
The left-hand form is Corollary 4.2 of [169], $\operatorname{Var}(\operatorname{vec} \mathbf{S}) = k (\mathbf{I} + \mathbf{K})(\mathbf{V} \otimes \mathbf{V})$ for $\mathbf{S} \sim \mathrm{Wishart}(k, \mathbf{V})$, at $k = T$ and $\mathbf{V} = \mathbf{\Sigma}_{\boldsymbol{\mu}}$. The right-hand form is the asymptotic covariance of $\operatorname{vec}(\hat{\mathbf{\Sigma}})$ for a normal sample. Its diagonal entry at $(i,j)$ is $\left(\hat{\Sigma}_{ii} \hat{\Sigma}_{jj} + \hat{\Sigma}_{ij}^{2}\right) / T$, the variance of the $(i,j)$ entry of a $\mathrm{Wishart}(T, \hat{\mathbf{\Sigma}}/T)$ draw. So the sample that ucs draws and the shape matrix that measures the sample have the same scale.
A set with diagonal = true keeps this diagonal under DiagonalOfShape, the default. ShapeOfDiagonal builds the diagonal from the variances alone. diagonal_sigma_shape states both rules.
Algorithm
- Build
K = commutation_matrix(sigma).sigmais square, soKis $N^{2} \times N^{2}$. - Form
sigma_sigma = T * (LinearAlgebra.I + K) * kron(sigma_mu, sigma_mu). - Repair
sigma_sigmain place withposdef!underpdm. - Return
sigma_sigma.
Arguments
pdm: Positive definite matrix estimator.nothingskips the repair.sigma_mu: Mean asymptotic covariance frommu_asymptotic_cov.sigma: Point estimate of the covariance matrix. The function reads only its size, throughcommutation_matrix.T: Scaling parameter fromchoose_scaling_parameter.
Returns
sigma_sigma::MatNum: Asymptotic covariance of the vectorised covariance estimator, $N^{2} \times N^{2}$.
Related
References
- [169] J. R. Magnus and H. Neudecker. The Commutation Matrix: Some Properties and Applications. The Annals of Statistics 7, 381–394 (1979). Section 4, Corollary 4.2.
PortfolioOptimisers.normal_sigma_shape — Function
normal_sigma_shape(ue::NormalUncertaintySet, sigma_mu::MatNum, sigma::MatNum, T::Number)Returns the covariance shape of an ellipsoidal or norm-ball NormalUncertaintySet.
Each ellipsoidal and norm-ball method of ucs and sigma_ucs builds its shape with this function. So the rule of ue.dc applies on every route that reads a diagonal shape. The box methods read no shape.
Algorithm
- If
ue.alg.diagonalistrue, returndiagonal_sigma_shapeunderue.dc. - Otherwise, return the full shape of
sigma_asymptotic_cov. The function does not readue.dcin this case.
Arguments
ue: Normal uncertainty set estimator, whosealgis anEllipsoidalUncertaintySetAlgorithmor aNormBallUncertaintySetAlgorithm.sigma_mu: Mean asymptotic covariance frommu_asymptotic_cov.sigma: Point estimate of the covariance matrix.T: Scaling parameter fromchoose_scaling_parameter.
Returns
sigma_sigma::MatNum: Covariance shape, $N^{2} \times N^{2}$.
Related
PortfolioOptimisers.mu_normal_box_set — Function
mu_normal_box_set(mu::VecNum, sigma_mu::MatNum, q::Number)Builds the box of the mean axis under normal returns, with a lower bound of zero and the full width in the upper bound.
q is the significance level after the halving. set_ucs_return_constraints! reads only the half-width $(\boldsymbol{\mu}_{u} - \boldsymbol{\mu}_{l})/2$ of the pair, and it centres that width on val. So the half-width is $z_{q} \sqrt{\operatorname{diag}(\mathbf{\Sigma}_{\boldsymbol{\mu}})}$. Neither bound is a bound on the mean. The box methods of ucs and mu_ucs for NormalUncertaintySet call this function.
Mathematical definition
\[\begin{align} \boldsymbol{\mu}_{l} &= \boldsymbol{0}\,, \\ \mu_{u,i} &= 2 z_{q} \sqrt{\left(\mathbf{\Sigma}_{\boldsymbol{\mu}}\right)_{ii}}\,, \qquad z_{q} = \Phi^{-1}(1 - q)\,. \end{align}\]
Where:
- $\boldsymbol{\mu}_{l}$, $\mu_{u,i}$: Lower bound, and $i$-th entry of the upper bound.
- $\mathbf{\Sigma}_{\boldsymbol{\mu}}$: Asymptotic covariance of the mean estimator.
- $z_{q}$: Normal complementary quantile at the halved significance level.
- $q$: Half significance level.
Each interval covers $1 - 2q$ of the sampling law of its entry, because each of the two tails holds $q$. Each interval applies to one entry, so the box is not a joint region for the full vector.
The half-width is the $\delta_{\boldsymbol{\mu},i}$ of Equation 11.17 of [12]. The text of that equation calls $\sigma_i$ the standard deviation of the asset. But its Listing 11.7 and this method use the standard error of the mean, $\sqrt{(\hat{\mathbf{\Sigma}} / T)_{ii}}$.
Algorithm
- Get the $1 - q$ standard normal quantile with
Distributions.cquantile. This is the multiplier $z_{q}$. - Multiply it by the square roots of
LinearAlgebra.diag(sigma_mu)and by two. The resultmu_uis the width of the box on each asset. - Build
mu_l, a range of zeros with the same length and element type. - Return a
BoxUncertaintySetthat holdsmu_l,mu_uandmu.muis the centre of the width.
Arguments
mu: Point estimate of the expected returns vector. The set holds it as its centre.sigma_mu: Mean asymptotic covariance frommu_asymptotic_cov.q: Half significance level, whichnormal_box_preamblehalved.
Returns
mu_ucs::BoxUncertaintySet: Expected returns uncertainty set.
Related
NormalUncertaintySetmu_asymptotic_covnormal_box_preambleBoxUncertaintySetset_ucs_return_constraints!
References
- [12] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Equation 11.17 and Listing 11.7. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
PortfolioOptimisers.sigma_normal_box_set — Function
sigma_normal_box_set(ue::NormalUncertaintySet, pr::LowOrderPrior, T::Number,
sigma_mu::MatNum, q::Number)Builds the box of the covariance axis of a NormalUncertaintySet.
The function draws ue.n_sim Wishart matrices with T degrees of freedom and scale sigma_mu. The element-wise q and 1 - q quantiles of the draws are the bounds. The function then projects each bound to the nearest positive definite matrix. The box methods of ucs and sigma_ucs both call it, so the two give the same box under one seed.
The mean of a $\mathrm{Wishart}(T, \mathbf{\Sigma}_{\boldsymbol{\mu}})$ draw is $T \mathbf{\Sigma}_{\boldsymbol{\mu}}$, which is $\hat{\mathbf{\Sigma}}$. So the draws are centred on the point estimate, and the two bounds are on the two sides of it.
Algorithm
- Read the point estimate
sigma = pr.sigma, the centre of the bounds. - Resolve the random number generator from
ue.rngandue.seedwithresolve_rng. - Draw
ue.n_simmatricessigmasfromDistributions.Wishart(T, sigma_mu). - Take the element-wise
qand1 - qquantiles ofsigmaswithbox_quantile_boundsto getsigma_landsigma_u. - Repair
sigma_landsigma_uin place withposdef!. The quantiles are entry by entry, so neither bound is a Wishart draw, and neither bound is always positive definite.posdef!does not change a bound that is already positive definite. - Return a
BoxUncertaintySetthat holdssigma_l,sigma_uandsigma.
Arguments
ue: Normal uncertainty set estimator.pr: Prior Result. The function reads the point estimatepr.sigmaand the asset countsize(pr.X, 2).T: Scaling parameter fromchoose_scaling_parameter, the degrees of freedom of the Wishart law.sigma_mu: Mean asymptotic covariance frommu_asymptotic_cov, the scale of the Wishart law.q: Half significance level.
Returns
sigma_ucs::BoxUncertaintySet: Covariance uncertainty set.
Related
PortfolioOptimisers.normal_box_preamble — Function
normal_box_preamble(ue::NormalUncertaintySet, pr::AbstractPriorResult, k::Integer)Does the first steps that the three box methods of a NormalUncertaintySet have in common.
The function finds the scaling parameter of moment k from the Prior Result, and then it makes the two values that the box of that moment uses. The box methods of ucs, mu_ucs and sigma_ucs all call it first. The mean box reads k = 1 and the covariance box reads k = 2, so ucs calls it once for each box.
The returned q is half of ue.q, and the ellipsoidal route does not halve the level. A box bounds each entry on two sides, so each tail gets half of the significance level. An ellipsoid cuts only the upper tail of a non-negative distance. So one cut at the $1 - q$ quantile covers $1 - q$.
Algorithm
- Resolve the scaling parameter of moment
kwithchoose_scaling_parameterto getT. - Build the mean asymptotic covariance with
mu_asymptotic_covto getsigma_mu. - Halve
ue.qto getq, the significance level of one tail. - Return
T,sigma_muandqas a tuple.
Arguments
ue: Normal uncertainty set estimator.pr: Prior Result that the set fits on.k: The index of the moment of the box,1for the mean and2for the covariance.
Returns
(T, sigma_mu, q): Scaling parameter, mean asymptotic covariance, and half significance level.
Related
References
- [12]
- D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025).
- [169]
- J. R. Magnus and H. Neudecker. The Commutation Matrix: Some Properties and Applications. The Annals of Statistics 7, 381–394 (1979).
- [305]
- D. S. Tracy and P. S. Dwyer. Multivariate Maxima and Minima with Matrix Derivatives. Journal of the American Statistical Association 64, 1576–1594 (1969).