Factor Prior: private API
Types
PortfolioOptimisers.LiftedFactorMoments — Type
struct LiftedFactorMoments{__T_mu, __T_sigma, __T_chol, __T_esigma, __T_edof, __T_ediv}The asset moments that a factor lift projects from the factor moments, with the residual variances that it measured.
This type is an implementation detail and is not intended for direct use. factor_lift makes it. FactorPrior and FactorBlackLittermanPrior read it.
Fields
mu:mu: The asset expected returns.
sigma:sigma: The asset covariance matrix.
chol:chol: The square-root factor, whose trailing block is the residual standard deviations when the lift adds a residual block.
esigma:esigma: The residual variances, ornothingwhen the lift adds no residual block.
edof:edof: The degrees of freedom of the residual variances, ornothingwhen none is known.
ediv:ediv: The divisor of the residual variances, ornothingwhen the variance estimator states none.
Related
PortfolioOptimisers.ResidualVarianceCounts — Type
struct ResidualVarianceCounts{__T_edof, __T_ediv}The degrees of freedom and the divisor of each residual variance that a lift measures.
This type is an implementation detail and is not intended for direct use. residual_variance_counts and cross_sectional_variance_counts make it. factor_lift and cross_sectional_assemble read it.
Fields
edof:edof: The degrees of freedom of each variance, ornothingwhen they are not known.
ediv:ediv: The divisor of each variance, ornothingwhen the variance estimator states none.
Related
PortfolioOptimisers.ResidualBlockConfig — Type
struct ResidualBlockConfig{__T_ve, __T_pdm, __T_rsd}How a prior estimator adds a residual block to the covariance it lifts.
This type is an implementation detail and is not intended for direct use. factor_residual_config makes it. assert_factor_residual_config checks it, and HighOrderFactorPriorEstimator reads it.
Fields
ve:ve: The variance estimator that sizes the residual block.
pdm:pdm: The positive definite matrix estimator that re-conditions the covariance after a consumer removes the block, ornothingwhen the estimator adds no block.
rsd:rsd: Whether the estimator adds the residual block.
Related
Functions
PortfolioOptimisers.factor_reconstruction — Function
factor_reconstruction(re::AbstractTimeSeriesRegressionEstimator, X::MatNum,
F::MatNum) -> Tuple{AbstractLoadingsRegressionResult, MatNum}Fit the loadings and rebuild the asset returns from the factor returns.
This is the first half of the factor lift, and the only half that every factor-axis estimator uses. It fits re on (X, F), and returns the regression result and the posterior returns matrix F * transpose(M) .+ transpose(b). FactorPrior, FactorBlackLittermanPrior and AugmentedBlackLittermanPrior each write this matrix into LowOrderPrior.X.
factor_lift is the second half, which projects the factor moments through the loadings. The two halves are separate functions because FactorBlackLittermanPrior needs the reconstruction before it has the moments to project. Its views change the factor distribution, so its factor moments exist only after the Black-Litterman update.
Algorithm
- Fit
reon(X, F)withregressionto getrr.rr.Mholds the $N \times K$ loadings, andrr.bholds the $N \times 1$ intercepts. - Rebuild the asset returns as
F * transpose(rr.M) .+ transpose(rr.b)to getposterior_X. It has the rows ofFand the columns ofX.
Arguments
re: Regression estimator.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.F: Data matrixobservations × factorsif thedimskeyword does not exist ordims = 1,factors × observationswhendims = 2.
Returns
rr::AbstractLoadingsRegressionResult: Regression result carrying the loadingsMand interceptsb.posterior_X::MatNum: Reconstructed asset returns,observations × assets.
Related
PortfolioOptimisers.factor_lift — Function
factor_lift(mp::AbstractMatrixProcessingEstimator, ve::AbstractVarianceEstimator,
rsd::Bool, rr::AbstractLoadingsRegressionResult, f_mu::VecNum, f_sigma::MatNum,
X::MatNum, posterior_X::MatNum, T::Real = size(posterior_X, 1);
mtx_sqrt::Option{<:AbstractMatrixSquareRootAlgorithm} = EigenFallbackSquareRoot(),
kwargs...) -> LiftedFactorMomentsProject factor moments onto the asset axis through the regression loadings.
This is the second half of the factor lift, and FactorPrior and FactorBlackLittermanPrior both call it. It maps f_mu and f_sigma through the loadings and processes the covariance. When rsd is true, it adds the diagonal residual block to the covariance and to the square-root factor.
The two callers differ only in the factor moments that they pass. FactorPrior passes the moments of the wrapped prior, and FactorBlackLittermanPrior passes the Black-Litterman posterior moments.
Mathematical definition
\[\begin{align} \hat{\boldsymbol{\mu}} &= \mathbf{B} \hat{\boldsymbol{f}} + \boldsymbol{\alpha}\,, \\ \hat{\mathbf{\Sigma}} &= \mathbf{B} \mathbf{\Sigma}_f \mathbf{B}^\intercal + \mathbf{\Sigma}_\varepsilon\,. \end{align}\]
Where:
- $\hat{\boldsymbol{\mu}}$: $N \times 1$ asset expected returns vector.
- $\hat{\mathbf{\Sigma}}$: $N \times N$ asset covariance matrix.
- $\mathbf{B}$: $N \times K$ factor loadings matrix,
rr.M. - $\boldsymbol{\alpha}$: $N \times 1$ vector of regression intercepts,
rr.b. - $\hat{\boldsymbol{f}}$: $K \times 1$ factor expected returns vector,
f_mu. - $\mathbf{\Sigma}_f$: $K \times K$ factor covariance matrix,
f_sigma. - $\mathbf{\Sigma}_\varepsilon$: $N \times N$ diagonal matrix of residual variances, present only when
rsdistrue.
The returned chol is the transpose of $[\mathbf{B} \mathbf{L}_f \quad \mathbf{\Sigma}_\varepsilon^{1/2}]$, where $\mathbf{L}_f$ is the square root of $\mathbf{\Sigma}_f$ that matrix_square_root takes under mtx_sqrt. By default it is the lower Cholesky factor of a positive definite matrix, and the eigen square root of a singular one. chol is $(K + N) \times N$ when rsd is true, and $K \times N$ when rsd is false. With rsd = false, neither chol nor $\hat{\mathbf{\Sigma}}$ holds the residual block.
The identity $\mathtt{chol}^\intercal \mathtt{chol} = \hat{\mathbf{\Sigma}}$ holds before matrix processing only. The lift builds chol from the f_sigma that the caller passed, so step 4 of the algorithm changes sigma and does not change chol. The default MatrixProcessing leaves the projected covariance as it is, because its pdm changes nothing on a positive semi-definite matrix. Under that mp the identity holds on the returned pair. An mp that denoises or detones moves sigma and leaves chol as it was, so the identity then holds against the unprocessed covariance only. A consumer that needs a factor of the returned sigma must factorise it again.
Algorithm
- Read the loadings
Mand the interceptsboffrr. - Project the factor mean through the loadings to get
posterior_mu. - Project the factor covariance through the loadings to get
posterior_sigma, the systematic block. - Process
posterior_sigmain place withsample_matrix_processing!, undermp,posterior_XandT. The matrix is assembled from the factor moments, so the denoising step reads the count of the factor covariance, and not the number of rows ofposterior_X. - Take the square root of
f_sigmaundermtx_sqrtwithmatrix_square_root, and multiply it by the loadings to getposterior_csigma. This reads thef_sigmathat the caller passed, which step 4 does not change. - When
rsdistrue, take the reconstruction errorerr = X - posterior_X. Measureesigma, the column variances oferr, withve. Read the counts of those variances withvariance_count, and combine them with theedofofrrthroughresidual_variance_countsto getedofandediv. Leterr_sigmabe the diagonal matrix of the variances. Whenrsdisfalse,esigmaandedivarenothing, andedofis theedofofrr. - Under
rsd, adderr_sigmatoposterior_sigma, and re-condition the sum withposdef!under the estimator thatmatrix_processing_posdefgives formp. This is the only explicitposdef!call of the function. For aMatrixProcessingthat estimator ismp.pdm, and it also acts onposterior_sigmainside step 4 when:pdmis a member ofmp.order. - Under
rsd, append the columns ofsqrt.(err_sigma)toposterior_csigma, so the factor also holds the block that step 7 added to the covariance. - Reshape
posterior_csigmatolength(posterior_mu)columns and transpose it intochol. Returnmu,sigma,chol,esigma,edofandediv.
Arguments
mp: Matrix processing estimator.ve: Variance estimator.rsd: Whether to include residual variance in the posterior covariance.rr: Regression result carrying the loadingsMand interceptsb.f_mu: Factor expected returns,factors × 1.f_sigma: Factor covariance matrix,factors × factors.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.posterior_X: Reconstructed asset returns fromfactor_reconstruction.T: The effective number of observations off_sigma, which the denoising step ofmpreads. A caller passes the count that the factor prior states, frommoment_observation_count. It issize(posterior_X, 1)by default.mtx_sqrt: Square-root algorithm of the factor covariance, and of the idiosyncratic covariance when that block is a full matrix, whose square roots make upchol. The defaultEigenFallbackSquareRoottakes the Cholesky factor where it exists, and the eigen square root of a singular positive semidefinite matrix otherwise.RidgeCholeskySquareRoottakes the Cholesky factor of the matrix with a small ridge added, andnothingtakes the plain Cholesky factor, which raises aLinearAlgebra.PosDefExceptionon a matrix that is not positive definite.matrix_square_rootstates each algorithm.kwargs...: Additional keyword arguments passed to matrix processing.
Returns
lift::LiftedFactorMoments: Asset expected returns, asset covariance, the square-root factor whose trailing block is the residual standard deviations whenrsdistrue, the residual variances, and their degrees of freedom and divisors.esigmaandedivarenothingwhenrsdisfalse, because the lift added no residual block. A caller writes the last three onto the loadings result that it returns. A consumer that needs the idiosyncratic variances, or their sampling law, then reads them off the block, and does not compute them again from the reconstruction error.
Related
PortfolioOptimisers.residual_variance_counts — Function
residual_variance_counts(cnt::Nothing, edof, T::Integer) -> ResidualVarianceCounts
residual_variance_counts(cnt::VarianceCount, edof::Nothing, T::Integer) -> ResidualVarianceCounts
residual_variance_counts(cnt::VarianceCount, edof::VecNum, T::Integer) -> ResidualVarianceCountsDegrees of freedom and divisor of each residual variance that a lift measures.
The sampling law of a residual variance needs one fact from the regression and one from the variance estimator. The regression records in edof the count of equally weighted observations that remain after the parameters it spent on each asset. variance_count states in cnt the effective count of the observations that the variance estimator reads, and its divisor. This function combines the two. Each case is a method, so no caller writes an isnothing test.
Mathematical definition
\[\begin{align} \nu_{i} &= n_{i} - \left(T - \nu_{i}^{\mathrm{fit}}\right)\,. \end{align}\]
Where:
- $\nu_{i}$: Degrees of freedom of the residual variance of asset $i$.
- $n_{i}$: Effective count of the observations that the variance estimator reads,
cnt.n. - $T$: Number of observations.
- $\nu_{i}^{\mathrm{fit}}$: Degrees of freedom that the regression left,
edof. $T - \nu_{i}^{\mathrm{fit}}$ is the number of parameters it spent on the asset.
With equal weights $n_{i} = T$, and the result is the edof of the regression. Under unequal weights the function subtracts the parameters that the regression spent from the effective count. This approximates the law of a weighted sum of squared residuals.
Arguments
cnt: The count and divisor of the variance estimator, ornothingwhen it states none.edof: Theedofof the loadings block, ornothingwhen the regression recorded none.T: Number of observations of the residuals.
Returns
counts::ResidualVarianceCounts: The degrees of freedom and the divisor of each variance. Both arenothingwhen the variance estimator states no count, because neither fact is then known.edofalone isnothingwhen the regression recorded no count.
Related
PortfolioOptimisers.factor_residual_config — Function
factor_residual_config(pe::AbstractPriorEstimator) -> Option{<:ResidualBlockConfig}Declare how a prior estimator adds a residual block to the covariance it lifts.
HighOrderFactorPriorEstimator subtracts the residual block to recover the systematic covariance, because its residual cokurtosis correction is defined on that covariance. It cannot read ve and the repair of mp off the fields of the estimator that it wraps. Its pe field has the bound AbstractLowOrderPriorEstimator_F_AF, and in that bound only FactorPrior and FactorBlackLittermanPrior carry both fields. The other members wrap a prior, pool priors, or build the asset moments with no residual block, as AugmentedBlackLittermanPrior does. A field access on one of them raises a FieldError.
The declaration states whether the estimator added a block, and how to re-condition the covariance after a consumer removes the block. The block itself is on the loadings result, at rr.esigma, and the fit measured it under its own weights. A prior that embeds this estimator can give the fit weights other than those of ve, for example the w of a caller on an entropy pooling prior, or the pooled weights of an opinion pool. So a consumer reads the block off rr, and does not measure it again under ve.
Each estimator declares its answer in a method beside its own definition:
- The two estimators that add a residual block report it.
- A wrapper forwards the answer of the estimator that it wraps. A pooling estimator is a wrapper here, and it forwards the one estimator that its moments come from.
- An estimator that adds no residual block returns
nothingfrom an explicit method. CrossSectionalFactorPriorthrows anArgumentError. Its block is the idiosyncratic covariance that it measured, which a variance undervedoes not rebuild.
An answer of nothing and an answer whose rsd is false both mean that the estimator added no residual block, so the consumer leaves the covariance as it is.
There is no default. A fallback that returns nothing cannot tell an estimator that adds no residual block from an estimator whose author forgot the method. In the second case the consumer does not subtract a residual block that the covariance holds. So the method for an undeclared type throws an ArgumentError that names the type. range_tails uses the same rule for a per-type declaration whose absence is a defect.
Arguments
pe: Prior estimator.
Validation
- Throws an
ArgumentErrorwhen the type ofpedeclares no method.
Returns
nothing: The estimator adds no residual block.cfg::ResidualBlockConfig: The variance estimator that sizes the residual block, the positive definite matrix estimator that re-conditions the covariance after a consumer removes the block, and whether the estimator adds the block. A factor prior givespdmfrommatrix_processing_posdefwhenrsdistrue, andnothingwhenrsdisfalse, because no consumer repairs a covariance that has no residual block.
Related
PortfolioOptimisers.assert_factor_residual_config — Function
assert_factor_residual_config(pe::AbstractPriorEstimator, cfg) -> NothingCheck the shape of a factor_residual_config answer before a consumer reads it.
A consumer reads ve, pdm and rsd off the answer by property access, which checks no shape. A declaration that returns a wrong value would raise a FieldError deep inside the correction, far from the declaration that caused it. This function checks the two shapes that the contract permits, and its error names the estimator that answered.
Arguments
pe: Prior estimator that answered.cfg: The answer offactor_residual_config(pe).
Validation
cfgisnothing, or aResidualBlockConfig.
Related
PortfolioOptimisers.coverage_regression — Function
coverage_regression(re::AbstractTimeSeriesRegressionEstimator, cmsk::Nothing)
coverage_regression(re::AbstractTimeSeriesRegressionEstimator, cmsk::BitVector)Takes the view of a regression estimator at the assets that a factor prior fits.
FactorPrior, FactorBlackLittermanPrior and AugmentedBlackLittermanPrior fit the regression on the columns of the returns that a mask keeps. A regression can hold one value per asset, such as the factor set of each asset in included of a StepwiseRegression. That value must describe the same columns as the returns that the fit reads, so the prior takes the view of the regression with the same mask.
Algorithm
The method that Julia selects is the algorithm.
- If
cmskisnothing, every asset is in the fit, so returnre. - If
cmskis a mask, returnport_opt_viewofreatfindall(cmsk).
Arguments
re: Regression estimator.cmsk: The mask of the assets in the fit, ornothing.
Returns
re::AbstractTimeSeriesRegressionEstimator: The regression on the assets in the fit.
Related
PortfolioOptimisers.show_fields — Method
show_fields(_::FactorPrior) -> NTuple{7, Symbol}
Renders every field of a FactorPrior except cache.
The state in cache belongs to an incremental fit and is not part of the configuration, and it would print under the estimator at every site that renders one. Set set_show_nothing_fields!(:FactorPrior, true) to render it.
Arguments
::FactorPrior: Prior estimator, read for its type alone.
Returns
fields::Tuple: The field names to render, which is(:pe, :mp, :re, :ve, :rsd, :mtx_sqrt, :x_src).
Related