Normal Uncertainty Sets: private API

PortfolioOptimisers.commutation_matrix — Function
commutation_matrix(X::MatNum)

Builds the commutation matrix for the size of X.

The commutation matrix is the permutation that maps the column-major vectorisation of a matrix to the vectorisation of its transpose. For a square matrix, it is its own inverse. It holds one non-zero entry in each row, so the function returns a sparse matrix. The function reads the size of X, and it does not read the entries.

Mathematical definition

\[\mathbf{K}_{m,n} \operatorname{vec}(\mathbf{A}) = \operatorname{vec}\!\left(\mathbf{A}^{\intercal}\right)\,, \qquad \mathbf{A} \in \mathbb{R}^{m \times n}\,.\]

Where:

  • $\mathbf{K}_{m,n}$: Commutation matrix of an $m \times n$ matrix, of size $mn \times mn$.
  • $\mathbf{A}$: Any $m \times n$ matrix.
  • $\operatorname{vec}$: Column-major vectorisation.

For a symmetric $\mathbf{A}$, $\mathbf{K}_{n,n} \operatorname{vec}(\mathbf{A}) = \operatorname{vec}(\mathbf{A})$. So a symmetric matrix cannot show the difference between $\mathbf{K}$ and the identity.

Algorithm

  1. Read m, n = size(X), and set mn = m * n, the side of the result.
  2. Number the entries of an m by n matrix in column-major order. This gives row = 1:mn.
  3. Find the position of each entry in the vectorised transpose with col = vec(transpose(reshape(row, m, n))).
  4. Write a one at each (row, col) pair. The result is the sparse mn by mn matrix cX.

Arguments

  • X: Matrix whose size sets the size of the commutation matrix.

Returns

  • cX::SparseArrays.SparseMatrixCSC{<:Number, <:Integer}: Sparse commutation matrix of size (m*n, m*n), where m, n = size(X).

Examples

julia> PortfolioOptimisers.commutation_matrix(rand(2, 2))4×4 SparseArrays.SparseMatrixCSC{Float64, Int64} with 4 stored entries: 1.0   ⋅    ⋅    ⋅  ⋅    ⋅   1.0   ⋅  ⋅   1.0   ⋅    ⋅  ⋅    ⋅    ⋅   1.0julia> PortfolioOptimisers.commutation_matrix(rand(2, 3))6×6 SparseArrays.SparseMatrixCSC{Float64, Int64} with 6 stored entries: 1.0   ⋅    ⋅    ⋅    ⋅    ⋅  ⋅    ⋅   1.0   ⋅    ⋅    ⋅  ⋅    ⋅    ⋅    ⋅   1.0   ⋅  ⋅   1.0   ⋅    ⋅    ⋅    ⋅  ⋅    ⋅    ⋅   1.0   ⋅    ⋅  ⋅    ⋅    ⋅    ⋅    ⋅   1.0julia> PortfolioOptimisers.commutation_matrix(rand(3, 2))6×6 SparseArrays.SparseMatrixCSC{Float64, Int64} with 6 stored entries: 1.0   ⋅    ⋅    ⋅    ⋅    ⋅  ⋅    ⋅    ⋅   1.0   ⋅    ⋅  ⋅   1.0   ⋅    ⋅    ⋅    ⋅  ⋅    ⋅    ⋅    ⋅   1.0   ⋅  ⋅    ⋅   1.0   ⋅    ⋅    ⋅  ⋅    ⋅    ⋅    ⋅    ⋅   1.0

Related

References

  • [169] J. R. Magnus and H. Neudecker. The Commutation Matrix: Some Properties and Applications. The Annals of Statistics 7, 381–394 (1979). Section 3, Definition 3.1 and Theorem 3.1 (vii). The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
  • [305] D. S. Tracy and P. S. Dwyer. Multivariate Maxima and Minima with Matrix Derivatives. Journal of the American Statistical Association 64, 1576–1594 (1969).
source
PortfolioOptimisers.choose_scaling_parameter — Method
choose_scaling_parameter(ue, pr, k::Integer)

Returns $T$, the effective number of observations of moment k of the prior. The covariance divided by $T$ is the asymptotic covariance of the mean.

The index k follows the field eoc of the prior: 1 is the mean and 2 is the covariance. A set of the mean reads k = 1, and a set of the covariance reads k = 2. A set of both reads each count for its own half, see normal_sigma_scaling.

The function reads two sources in order. The first is the ens of the estimator, a count that the caller states for every moment. The second is the count of the moment that the prior states, from moment_observation_count. A weighted prior has fewer effective observations than rows, and a prior under a Scenario Cap has fewer rows than the observations of its moments. The prior states both counts in eoc, and nothing there means the number of rows. The ens of the prior is a diagnostic of its weights, and the function does not read it.

Each set of this estimator scales with $T$. The width of a set is proportional to $T^{-1/2}$, so a prior with fewer effective observations than rows gives a wider set. A division of $T$ by four doubles the width of the mean box.

Algorithm

  1. If ue.ens is not nothing, return it. It is the effective count that the estimator states.
  2. Otherwise, return moment_observation_count of pr and k.

Arguments

  • ue: NormalUncertaintySet estimator.
  • pr: Prior Result.
  • k: The index of the moment, 1 for the mean and 2 for the covariance.

Returns

  • T::Number: Effective number of observations.

Related

source
PortfolioOptimisers.normal_sigma_scaling — Function
normal_sigma_scaling(ue::NormalUncertaintySet, pr::AbstractPriorResult, sigma::MatNum,
                     T_mu::Number, sigma_mu::MatNum) -> Tuple

Returns the scaling parameter of the covariance half of a joint set, and the mean asymptotic covariance at that parameter.

A joint method of ucs builds the mean half at the count of the mean, T_mu. The covariance half reads the count of the covariance from choose_scaling_parameter, and its shape reads the mean asymptotic covariance at that count. The two counts differ when the mean estimator and the covariance estimator of the prior carry different weights. When they are equal, the function returns sigma_mu itself, so the usual case computes nothing again.

Arguments

  • ue: Normal uncertainty set estimator.
  • pr: Prior Result that the set fits on.
  • sigma: The covariance of pr on the investable assets.
  • T_mu: The scaling parameter of the mean half.
  • sigma_mu: The mean asymptotic covariance at T_mu.

Returns

  • (T, sigma_mu): The scaling parameter of the covariance half, and the mean asymptotic covariance at it.

Related

source
PortfolioOptimisers.normal_mu_error_sample — Function
normal_mu_error_sample(ue::NormalUncertaintySet, rng::Random.AbstractRNG, mu::VecNum,
                       sigma_mu::MatNum)

Draws ue.n_sim estimation errors of the mean from the sampling law of normal returns.

Each row of the sample is an estimation error, a draw minus mu. The draws come from $\mathcal{N}(\hat{\boldsymbol{\mu}}, \hat{\mathbf{\Sigma}}/T)$, so the sample and its shape matrix have the same scale. A draw from $\mathcal{N}(\hat{\boldsymbol{\mu}}, \hat{\mathbf{\Sigma}})$ multiplies every deviation by $\sqrt{T}$, so it also multiplies the radius by $\sqrt{T}$. The caller resolves the generator and gives it to the function. So a caller that needs both axes draws them from one stream.

Algorithm

  1. Draw ue.n_sim vectors from Distributions.MvNormal(mu, sigma_mu). The result has one column for each draw.
  2. Subtract mu from each column. Transpose the result, so that each row holds one estimation error.

Arguments

  • ue: Normal uncertainty set estimator.
  • rng: Random number generator, which the caller resolves with resolve_rng.
  • mu: Point estimate of the expected returns vector, the centre of the draws.
  • sigma_mu: Mean asymptotic covariance from mu_asymptotic_cov.

Returns

  • X_mu::MatNum: Sampled estimation errors, ue.n_sim by N.

Related

source
PortfolioOptimisers.normal_sigma_error_sample — Function
normal_sigma_error_sample(ue::NormalUncertaintySet, rng::Random.AbstractRNG,
                          sigma::MatNum, sigma_mu::MatNum, T::Number, N::Integer)

Draws ue.n_sim estimation errors of the vectorised covariance from the Wishart law of normal returns.

Each row of the sample is an estimation error, a draw minus sigma. The variance of an entry of a $\mathrm{Wishart}(T, \hat{\mathbf{\Sigma}}/T)$ draw is the matching diagonal entry of $\mathbf{\Sigma}_{\mathbf{\Sigma}}$, the asymptotic covariance of the covariance estimator. So the sample and its shape matrix have the same scale. The caller gives N, and the function does not read it from sigma. So every caller reads the asset count from one source, size(pr.X, 2).

Algorithm

  1. Draw ue.n_sim matrices from Distributions.Wishart(T, sigma_mu), one at a time into one buffer.
  2. Subtract sigma from each draw into an N by N by ue.n_sim array. The function holds only this array and the buffer, and not all the draws.
  3. Reshape the array to N^2 rows, with one column for each draw. Transpose it, so that each row holds one vectorised estimation error.

Arguments

  • ue: Normal uncertainty set estimator.
  • rng: Random number generator, which the caller resolves with resolve_rng.
  • sigma: Point estimate of the covariance matrix, the centre of the draws.
  • sigma_mu: Mean asymptotic covariance from mu_asymptotic_cov, the scale of the Wishart law.
  • T: Scaling parameter from choose_scaling_parameter, the degrees of freedom of the Wishart law.
  • N: Asset count. The caller reads it from size(pr.X, 2).

Returns

  • X_sigma::MatNum: Sampled estimation errors, ue.n_sim by N^2.

Related

source
PortfolioOptimisers.mu_asymptotic_cov — Function
mu_asymptotic_cov(pdm, sigma, T)

Returns the asymptotic covariance of the mean estimator, projected to the nearest positive definite matrix with pdm.

Every route of NormalUncertaintySet calls this function, so all routes use the same mean shape. A division by a positive scalar keeps the definiteness of sigma. So posdef! changes nothing when sigma is positive definite.

Mathematical definition

\[\mathbf{\Sigma}_{\boldsymbol{\mu}} = \dfrac{\hat{\mathbf{\Sigma}}}{T}\,.\]

Where:

  • $\mathbf{\Sigma}_{\boldsymbol{\mu}}$: Asymptotic covariance of the mean estimator.
  • $\hat{\mathbf{\Sigma}}$: Estimated covariance matrix.
  • $T$: Number of observations.

Algorithm

  1. Divide sigma by T to get sigma_mu.
  2. Repair sigma_mu in place with posdef! under pdm.
  3. Return sigma_mu.

Arguments

  • pdm: Positive definite matrix estimator. nothing skips the repair.
  • sigma: Point estimate of the covariance matrix.
  • T: Scaling parameter from choose_scaling_parameter.

Returns

  • sigma_mu::MatNum: Asymptotic covariance of the mean estimator.

Related

References

  • [12] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Equations 11.16 and 11.24. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
source
PortfolioOptimisers.sigma_asymptotic_cov — Function
sigma_asymptotic_cov(pdm, sigma_mu, sigma, T)

Returns the asymptotic covariance of the vectorised covariance estimator, projected to the nearest positive definite matrix with pdm.

sigma_mu is the full mean asymptotic covariance from mu_asymptotic_cov, before any diagonal step. $\mathbf{K}$ is the commutation_matrix of sigma. The two matrix arguments depend on each other. The form below is the sampling law of a normal sample only if sigma_mu comes from the same sigma and the same T.

Mathematical definition

\[\mathbf{\Sigma}_{\mathbf{\Sigma}} = T \left(\mathbf{I} + \mathbf{K}\right) \left(\mathbf{\Sigma}_{\boldsymbol{\mu}} \otimes \mathbf{\Sigma}_{\boldsymbol{\mu}}\right) = \dfrac{\left(\mathbf{I} + \mathbf{K}\right) \left(\hat{\mathbf{\Sigma}} \otimes \hat{\mathbf{\Sigma}}\right)}{T}\,.\]

Where:

  • $\mathbf{\Sigma}_{\mathbf{\Sigma}}$: Asymptotic covariance of the vectorised covariance estimator.
  • $\mathbf{\Sigma}_{\boldsymbol{\mu}}$: Asymptotic covariance of the mean estimator.
  • $\mathbf{I}$: Identity matrix.
  • $\mathbf{K}$: Commutation matrix.
  • $\otimes$: Kronecker product.
  • $\hat{\mathbf{\Sigma}}$: Estimated covariance matrix.
  • $T$: Number of observations.

The left-hand form is Corollary 4.2 of [169], $\operatorname{Var}(\operatorname{vec} \mathbf{S}) = k (\mathbf{I} + \mathbf{K})(\mathbf{V} \otimes \mathbf{V})$ for $\mathbf{S} \sim \mathrm{Wishart}(k, \mathbf{V})$, at $k = T$ and $\mathbf{V} = \mathbf{\Sigma}_{\boldsymbol{\mu}}$. The right-hand form is the asymptotic covariance of $\operatorname{vec}(\hat{\mathbf{\Sigma}})$ for a normal sample. Its diagonal entry at $(i,j)$ is $\left(\hat{\Sigma}_{ii} \hat{\Sigma}_{jj} + \hat{\Sigma}_{ij}^{2}\right) / T$, the variance of the $(i,j)$ entry of a $\mathrm{Wishart}(T, \hat{\mathbf{\Sigma}}/T)$ draw. So the sample that ucs draws and the shape matrix that measures the sample have the same scale.

A set with diagonal = true keeps this diagonal under DiagonalOfShape, the default. ShapeOfDiagonal builds the diagonal from the variances alone. diagonal_sigma_shape states both rules.

Algorithm

  1. Build K = commutation_matrix(sigma). sigma is square, so K is $N^{2} \times N^{2}$.
  2. Form sigma_sigma = T * (LinearAlgebra.I + K) * kron(sigma_mu, sigma_mu).
  3. Repair sigma_sigma in place with posdef! under pdm.
  4. Return sigma_sigma.

Arguments

Returns

  • sigma_sigma::MatNum: Asymptotic covariance of the vectorised covariance estimator, $N^{2} \times N^{2}$.

Related

References

  • [169] J. R. Magnus and H. Neudecker. The Commutation Matrix: Some Properties and Applications. The Annals of Statistics 7, 381–394 (1979). Section 4, Corollary 4.2.
source
PortfolioOptimisers.normal_sigma_shape — Function
normal_sigma_shape(ue::NormalUncertaintySet, sigma_mu::MatNum, sigma::MatNum, T::Number)

Returns the covariance shape of an ellipsoidal or norm-ball NormalUncertaintySet.

Each ellipsoidal and norm-ball method of ucs and sigma_ucs builds its shape with this function. So the rule of ue.dc applies on every route that reads a diagonal shape. The box methods read no shape.

Algorithm

  1. If ue.alg.diagonal is true, return diagonal_sigma_shape under ue.dc.
  2. Otherwise, return the full shape of sigma_asymptotic_cov. The function does not read ue.dc in this case.

Arguments

Returns

  • sigma_sigma::MatNum: Covariance shape, $N^{2} \times N^{2}$.

Related

source
PortfolioOptimisers.mu_normal_box_set — Function
mu_normal_box_set(mu::VecNum, sigma_mu::MatNum, q::Number)

Builds the box of the mean axis under normal returns, with a lower bound of zero and the full width in the upper bound.

q is the significance level after the halving. set_ucs_return_constraints! reads only the half-width $(\boldsymbol{\mu}_{u} - \boldsymbol{\mu}_{l})/2$ of the pair, and it centres that width on val. So the half-width is $z_{q} \sqrt{\operatorname{diag}(\mathbf{\Sigma}_{\boldsymbol{\mu}})}$. Neither bound is a bound on the mean. The box methods of ucs and mu_ucs for NormalUncertaintySet call this function.

Mathematical definition

\[\begin{align} \boldsymbol{\mu}_{l} &= \boldsymbol{0}\,, \\ \mu_{u,i} &= 2 z_{q} \sqrt{\left(\mathbf{\Sigma}_{\boldsymbol{\mu}}\right)_{ii}}\,, \qquad z_{q} = \Phi^{-1}(1 - q)\,. \end{align}\]

Where:

  • $\boldsymbol{\mu}_{l}$, $\mu_{u,i}$: Lower bound, and $i$-th entry of the upper bound.
  • $\mathbf{\Sigma}_{\boldsymbol{\mu}}$: Asymptotic covariance of the mean estimator.
  • $z_{q}$: Normal complementary quantile at the halved significance level.
  • $q$: Half significance level.

Each interval covers $1 - 2q$ of the sampling law of its entry, because each of the two tails holds $q$. Each interval applies to one entry, so the box is not a joint region for the full vector.

The half-width is the $\delta_{\boldsymbol{\mu},i}$ of Equation 11.17 of [12]. The text of that equation calls $\sigma_i$ the standard deviation of the asset. But its Listing 11.7 and this method use the standard error of the mean, $\sqrt{(\hat{\mathbf{\Sigma}} / T)_{ii}}$.

Algorithm

  1. Get the $1 - q$ standard normal quantile with Distributions.cquantile. This is the multiplier $z_{q}$.
  2. Multiply it by the square roots of LinearAlgebra.diag(sigma_mu) and by two. The result mu_u is the width of the box on each asset.
  3. Build mu_l, a range of zeros with the same length and element type.
  4. Return a BoxUncertaintySet that holds mu_l, mu_u and mu. mu is the centre of the width.

Arguments

  • mu: Point estimate of the expected returns vector. The set holds it as its centre.
  • sigma_mu: Mean asymptotic covariance from mu_asymptotic_cov.
  • q: Half significance level, which normal_box_preamble halved.

Returns

  • mu_ucs::BoxUncertaintySet: Expected returns uncertainty set.

Related

References

  • [12] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Equation 11.17 and Listing 11.7. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
source
PortfolioOptimisers.sigma_normal_box_set — Function
sigma_normal_box_set(ue::NormalUncertaintySet, pr::LowOrderPrior, T::Number,
                     sigma_mu::MatNum, q::Number)

Builds the box of the covariance axis of a NormalUncertaintySet.

The function draws ue.n_sim Wishart matrices with T degrees of freedom and scale sigma_mu. The element-wise q and 1 - q quantiles of the draws are the bounds. The function then projects each bound to the nearest positive definite matrix. The box methods of ucs and sigma_ucs both call it, so the two give the same box under one seed.

The mean of a $\mathrm{Wishart}(T, \mathbf{\Sigma}_{\boldsymbol{\mu}})$ draw is $T \mathbf{\Sigma}_{\boldsymbol{\mu}}$, which is $\hat{\mathbf{\Sigma}}$. So the draws are centred on the point estimate, and the two bounds are on the two sides of it.

Algorithm

  1. Read the point estimate sigma = pr.sigma, the centre of the bounds.
  2. Resolve the random number generator from ue.rng and ue.seed with resolve_rng.
  3. Draw ue.n_sim matrices sigmas from Distributions.Wishart(T, sigma_mu).
  4. Take the element-wise q and 1 - q quantiles of sigmas with box_quantile_bounds to get sigma_l and sigma_u.
  5. Repair sigma_l and sigma_u in place with posdef!. The quantiles are entry by entry, so neither bound is a Wishart draw, and neither bound is always positive definite. posdef! does not change a bound that is already positive definite.
  6. Return a BoxUncertaintySet that holds sigma_l, sigma_u and sigma.

Arguments

  • ue: Normal uncertainty set estimator.
  • pr: Prior Result. The function reads the point estimate pr.sigma and the asset count size(pr.X, 2).
  • T: Scaling parameter from choose_scaling_parameter, the degrees of freedom of the Wishart law.
  • sigma_mu: Mean asymptotic covariance from mu_asymptotic_cov, the scale of the Wishart law.
  • q: Half significance level.

Returns

  • sigma_ucs::BoxUncertaintySet: Covariance uncertainty set.

Related

source
PortfolioOptimisers.normal_box_preamble — Function
normal_box_preamble(ue::NormalUncertaintySet, pr::AbstractPriorResult, k::Integer)

Does the first steps that the three box methods of a NormalUncertaintySet have in common.

The function finds the scaling parameter of moment k from the Prior Result, and then it makes the two values that the box of that moment uses. The box methods of ucs, mu_ucs and sigma_ucs all call it first. The mean box reads k = 1 and the covariance box reads k = 2, so ucs calls it once for each box.

The returned q is half of ue.q, and the ellipsoidal route does not halve the level. A box bounds each entry on two sides, so each tail gets half of the significance level. An ellipsoid cuts only the upper tail of a non-negative distance. So one cut at the $1 - q$ quantile covers $1 - q$.

Algorithm

  1. Resolve the scaling parameter of moment k with choose_scaling_parameter to get T.
  2. Build the mean asymptotic covariance with mu_asymptotic_cov to get sigma_mu.
  3. Halve ue.q to get q, the significance level of one tail.
  4. Return T, sigma_mu and q as a tuple.

Arguments

  • ue: Normal uncertainty set estimator.
  • pr: Prior Result that the set fits on.
  • k: The index of the moment of the box, 1 for the mean and 2 for the covariance.

Returns

  • (T, sigma_mu, q): Scaling parameter, mean asymptotic covariance, and half significance level.

Related

source

References

[12]
D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025).
[169]
J. R. Magnus and H. Neudecker. The Commutation Matrix: Some Properties and Applications. The Annals of Statistics 7, 381–394 (1979).
[305]
D. S. Tracy and P. S. Dwyer. Multivariate Maxima and Minima with Matrix Derivatives. Journal of the American Statistical Association 64, 1576–1594 (1969).