High order factor Prior estimator
PortfolioOptimisers.HighOrderFactorPriorEstimator — Type
struct HighOrderFactorPriorEstimator{__T_pe, __T_kte, __T_ske, __T_ex, __T_rsd, __T_cache, __T_x_src} <: AbstractHighOrderPriorEstimator_FProjects factor coskewness and cokurtosis onto the asset axis through the regression loadings.
HighOrderFactorPriorEstimator holds a low order factor prior estimator, a cokurtosis estimator and a coskewness estimator. Its prior fits the co-moments of the factor returns, and lifts them onto the asset axis with the loadings of the regression. When rsd is true, it adds the residual co-moments of the regression.
Fields
pe: Prior estimator.
kte: Cokurtosis estimator.
ske: Coskewness estimator.
ex: Parallel execution strategy.
rsd: Whether to include residual variance in the posterior covariance.
cache: Optional partial-fit state. It isnothinguntilpartial_fit!writes one. When the caller gives no data matrix, the batch verb of the estimator reads it. Each propagation channel does one thing with it.factorykeeps it unchanged, because a factory call resolves the configuration, not the sample.port_opt_viewcopies the entries of the selected assets into a new state, so the viewed estimator holds the state of those assets alone.obs_weights_viewremoves it, because a state has no slice on the observation axis. Only a type with an observation-weights field has that channel. A type with no such field falls through to the identity method, so it keeps its state. A family whose state has no exact asset slice removes it on both axes and names the reason.
x_src: The source of the returns that the prior fits on,nothingor anAbstractReturnsSource.nothingandDataSourceread the asset returnsrd.X.ExcessSourcereads the returns in excess of the benchmark,rd.X - rd.B, the Excess Frame, and refuses returns data with no benchmark.nothingmeans that the caller stated no source, so an optimiser that holds a reader of the Excess Frame warns about a prior atnothing, and not about a prior atDataSource(). A prior reads no prior, so it refusesPriorSource. A prior inside another estimator, such as thepeof aFactorPrior, keepsnothing, and the outer estimator states the source.
Constructors
HighOrderFactorPriorEstimator(; pe::AbstractLowOrderPriorEstimator_F_AF = FactorPrior(), kte::Option{<:CokurtosisEstimator} = Cokurtosis(; alg = FullMoment()), ske::Option{<:CoskewnessEstimator} = Coskewness(; alg = FullMoment()), ex::FLoops.Transducers.Executor = FLoops.ThreadedEx(), rsd::Bool = true, cache::Option{<:AbstractPartialFitState} = nothing, x_src::Option{<:AbstractReturnsSource} = nothing) -> HighOrderFactorPriorEstimatorKeywords correspond to the struct's fields.
Validation
x_srcisnothing,DataSource()orExcessSource(), seeassert_returns_source.- The
x_srcofpeisnothing, because the prior picks the returns that it reads, seeassert_inner_reader. ktestates its processor, seeassert_cokurtosis_processor.- When
rsdistrue,kteandskestate what the residual corrections read, seeassert_factor_comoment_residuals.
Propagated parameters
factory passes its arguments on to these @fprop fields:
pe: Recursively updated throughfactory.kte: Recursively updated throughfactory.ske: Recursively updated throughfactory.cache: Carried unchanged throughfactory.
View parameters
port_opt_view takes the view of these @vprop fields at the selected indices:
pe: Recursively viewed throughport_opt_view.cache: Sliced to the selected assets throughport_opt_view.
Examples
julia> HighOrderFactorPriorEstimator()HighOrderFactorPriorEstimator pe ┼ FactorPrior │ pe ┼ EmpiricalPrior │ │ ce ┼ PortfolioOptimisersCovariance │ │ │ ce ┼ Covariance │ │ │ │ me ┼ SimpleExpectedReturns │ │ │ │ │ w ┴ nothing │ │ │ │ ce ┼ GeneralCovariance │ │ │ │ │ ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true) │ │ │ │ │ w ┴ nothing │ │ │ │ alg ┼ FullMoment() │ │ │ │ w ┴ nothing │ │ │ mp ┼ MatrixProcessing │ │ │ │ pdm ┼ Posdef │ │ │ │ │ alg ┼ UnionAll: NearestCorrelationMatrix.Newton │ │ │ │ │ kwargs ┴ @NamedTuple{}: NamedTuple() │ │ │ │ dn ┼ nothing │ │ │ │ dt ┼ nothing │ │ │ │ alg ┼ nothing │ │ │ │ order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg) │ │ me ┼ SimpleExpectedReturns │ │ │ w ┴ nothing │ │ horizon ┼ nothing │ │ fill_limit ┼ nothing │ │ x_src ┴ nothing │ mp ┼ MatrixProcessing │ │ pdm ┼ Posdef │ │ │ alg ┼ UnionAll: NearestCorrelationMatrix.Newton │ │ │ kwargs ┴ @NamedTuple{}: NamedTuple() │ │ dn ┼ nothing │ │ dt ┼ nothing │ │ alg ┼ nothing │ │ order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg) │ re ┼ StepwiseRegression │ │ crit ┼ PValue │ │ │ t ┴ Float64: 0.05 │ │ alg ┼ ForwardSelection() │ │ tgt ┼ LinearModel │ │ │ kwargs ┴ @NamedTuple{}: NamedTuple() │ │ choice ┼ BatchChoice() │ │ included ┼ nothing │ │ x_src ┴ nothing │ ve ┼ SimpleVariance │ │ me ┼ SimpleExpectedReturns │ │ │ w ┴ nothing │ │ w ┼ nothing │ │ corrected ┴ Bool: true │ rsd ┼ Bool: true │ mtx_sqrt ┼ EigenFallbackSquareRoot() │ x_src ┴ nothing kte ┼ Cokurtosis │ me ┼ SimpleExpectedReturns │ │ w ┴ nothing │ mp ┼ MatrixProcessing │ │ pdm ┼ Posdef │ │ │ alg ┼ UnionAll: NearestCorrelationMatrix.Newton │ │ │ kwargs ┴ @NamedTuple{}: NamedTuple() │ │ dn ┼ nothing │ │ dt ┼ nothing │ │ alg ┼ nothing │ │ order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg) │ alg ┼ FullMoment() │ w ┼ nothing │ cache ┴ nothing ske ┼ Coskewness │ me ┼ SimpleExpectedReturns │ │ w ┴ nothing │ mp ┼ MatrixProcessing │ │ pdm ┼ Posdef │ │ │ alg ┼ UnionAll: NearestCorrelationMatrix.Newton │ │ │ kwargs ┴ @NamedTuple{}: NamedTuple() │ │ dn ┼ nothing │ │ dt ┼ nothing │ │ alg ┼ nothing │ │ order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg) │ alg ┼ FullMoment() │ w ┼ nothing │ cache ┴ nothing ex ┼ Transducers.ThreadedEx{@NamedTuple{}}: Transducers.ThreadedEx() rsd ┼ Bool: true x_src ┴ nothingThe incremental fit
This prior has no exact incremental recursion, and it takes the online step by a refit from a sample buffer. Online seeds cache, and partial_fit! adds each observation to it with no change. The prior method with one argument then runs the batch verb of this estimator over the rows that the buffer kept. The result is the batch fit over those rows, and a max_history on the wrapper sets the window of the whole fit.
cache goes through the three propagation channels as every partial-fit state does. factory carries it with no change, and port_opt_view cuts it to the selected assets. obs_weights_view drops it, because a state has no cut on the observation axis. The show method does not print cache, because a buffer that changes at each step is not part of the configuration.
Related
AbstractHighOrderPriorEstimator_FFactorPriorCokurtosisEstimatorCoskewnessEstimatorHighOrderPriorfactoryport_opt_view
References
- [164] K. Boudt, W. Lu and B. Peeters. Higher order comoments of multifactor models and asset allocation. Finance Research Letters 13, 225–233 (2015). Pages 225–233. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
- [165] L. Martellini and V. Ziemann. Improved estimates of higher-order comoments and implications for portfolio selection. The Review of Financial Studies 23, 1467–1502 (2010).
PortfolioOptimisers.prior — Function
prior(pe::HighOrderFactorPriorEstimator, X::MatNum, F::MatNum,
pnl::Option{<:AssetPanel} = nothing; dims::Int = 1,
kwargs...)Compute the high order prior of the asset returns through a factor model.
prior fits the low order moments with the embedded factor prior pe.pe. It fits the co-moments of the factor returns, and maps them to the asset axis with the Kronecker product of the loadings. When pe.rsd is true, it adds the residual co-moments.
A Black-Litterman prior can go under this estimator. Each wrapper estimator passes rr and the factor block on, so HighOrderFactorPriorEstimator(; pe = BlackLittermanPrior(; pe = FactorPrior(…))) gets to a regression, and it does not throw an IsNothingError.
The co-moments go through rr.M, and mu and sigma hold the views. Black-Litterman gives no third or fourth moments, so the projection of the factor co-moments is the only estimate of them.
The method computes the co-moments from F as the caller gives it. So they describe the factor distribution before the views, for each Black-Litterman prior under this estimator. FactorBlackLittermanPrior and BayesianBlackLittermanPrior give a posterior factor block. Under them, the orders of the nested fpr do not agree. fpr.mu and fpr.sigma hold the views, and fpr.kt, fpr.sk and fpr.V do not. The invariant fpr.pr === pr.fpr still holds, because both routes get the same posterior low order block. The views stop at the second order.
The cause is that Black-Litterman has no update for the third and fourth moments. The method discards no value. BlackLittermanPrior behaves in the same way. Its low order factor block is also before the views, so all the orders of its prior result agree.
An entropy pooling prior and an opinion pooling prior are different. They state their views and opinions as the probabilities w of the scenarios. The factor co-moments read w, so the views get to every order.
Mathematical definition
The method maps the factor co-moments to the asset axis through the loadings matrix $\mathbf{B}$:
\[\begin{align} \hat{\mathbf{\Sigma}}_4 &= (\mathbf{B} \otimes \mathbf{B}) \hat{\mathbf{\Sigma}}_{4,f} (\mathbf{B} \otimes \mathbf{B})^\intercal + \hat{\mathbf{\Sigma}}_{4,\varepsilon}\,, \\ \hat{\mathbf{M}}_3 &= \mathbf{B} \hat{\mathbf{M}}_{3,f} (\mathbf{B} \otimes \mathbf{B})^\intercal + \hat{\mathbf{M}}_{3,\varepsilon}\,. \end{align}\]
Where:
- $\hat{\mathbf{\Sigma}}_4$: $N^2 \times N^2$ asset square cokurtosis matrix,
kt. - $\hat{\mathbf{M}}_3$: $N \times N^2$ asset coskewness matrix,
sk. - $\mathbf{B}$: $N \times K$ factor loadings matrix,
pr.rr.M. - $\hat{\mathbf{\Sigma}}_{4,f}$: $K^2 \times K^2$ factor square cokurtosis matrix,
fpr.kt. - $\hat{\mathbf{M}}_{3,f}$: $K \times K^2$ factor coskewness matrix,
fpr.sk. - $\hat{\mathbf{\Sigma}}_{4,\varepsilon}$: Residual cokurtosis from
cokurtosis_residuals. It is present only whenrsdistrue. - $\hat{\mathbf{M}}_{3,\varepsilon}$: Residual coskewness from
coskewness_residuals. It is present only whenrsdistrue. - $\otimes$: Kronecker product.
pe.kte and pe.ske fit the factor co-moments on F. When the low order block gives the probabilities w of its scenarios, w replaces the observation weights of both estimators. The fit then reads the last length(w) rows of F, which w weighs. The residual corrections do not read w. The residuals belong to the model, and the views change the probabilities of the scenarios, not the model. A different alg on either estimator gives a different formula in place of its display above. When an estimator is nothing, the asset result and the nested factor block both hold nothing for its moment.
Algorithm
- Orient
XandFtoobservations × variableswithdims_oriented. - Compute the low order block
prwithpe.pe, and check that it holds a regression result. Derive the Investable Mask from it withinvestable_mask. Cut the whole prior result to the investable universe withport_opt_view, and call the resultrpr.Xis an argument and not a block of the prior result, so cut it in the same way, and call itXr. Take the reconstructed returnsposterior_X = rpr.Xand the loadingsM = rpr.rr.M. When every asset is investable, the mask isnothing, the method cuts nothing, and the steps below run onpr. The cut comes before the lift, because the lift spreads a non-investable asset much further. OneNaNrow of loadings makes a fullNaNband ofkron(M, M), and the projection spreads it over most of the cokurtosis.posdef!then refuses the matrix with a LAPACK error that does not name the asset. - Read the probabilities
w = pr.wof the scenarios. CutFto the rowsFwthatwweighs, withscenario_rows. Read the countsT_skandT_ktoffactory(pe.ske, w)andfactory(pe.kte, w)overFwwithhigh_order_observation_count. Compute the factor square cokurtosisf_ktwithfactory(pe.kte, w)onFw. Whenf_ktis notnothing, buildkM = kron(M, M), and projectposterior_kt = kM * f_kt * transpose(kM). Processposterior_ktwith the estimator thatcokurtosis_processor(pe.kte)gives, throughsample_matrix_processing!at the countT_kt. - Compute the factor coskewness
f_skand its negative spectral formf_Vwithfactory(pe.ske, w)onFw. Whenf_skis notnothing, buildkMif step 3 did not, and projectposterior_sk = M * f_sk * transpose(kM). - Build the structure matrices with
dup_elim_sum_matricestwo times. BuildD2,L2andS2at the full asset count, andf_D2,f_L2andf_S2at the factor count. The asset count is the full count, because step 12 expands the co-moments, and the constructor checks the matrices againstlength(pr.mu). At both dimensions, the method selects the matrices by the rule ofprior(::HighOrderPriorEstimator, …). - When
pe.rsdistrue, take the reconstruction errorerr = Xr - posterior_X. Under a Scenario Cap,posterior_Xholds only the last rows. Then the method usesF * transpose(M) .+ transpose(b)over every row in place ofposterior_X, as the residual block ofsigmadoes. - Under
pe.rsd, when step 4 gave aposterior_sk, addcoskewness_residuals(err, me)to it.mecomes fromcoskewness_expected_returns(pe.ske). - Under
pe.rsd, when step 3 gave aposterior_kt, read the declaration of the residual block of the wrapped estimator withfactor_residual_config. Check its shape withassert_factor_residual_config. - Get the systematic covariance
sigmafromrpr.sigma. Anothingdeclaration, or a declaration whosersdisfalse, means that the wrapped estimator added no residual block. Thensigmaisrpr.sigmawith no change. Otherwise, read the block withidiosyncratic_variancesofrpr.rr, and call iterr_sigma. These are the residual variances that the wrapped estimator added, under the weights of its own fit. A variance oferrunderrsd_cfg.vedoes not read these weights. Subtract the diagonal matrix oferr_sigmafromrpr.sigma. When an entry oferr_sigmais larger than the diagonal entry ofrpr.sigmathat matches it, the subtraction gives a negative variance. Then the step warns, and keepsrpr.sigmawhole. This happens when the covariance of the wrapped estimator does not hold the block, such as a posterior that shrinks the sum of the lift. In both cases, repair the result withposdef!underrsd_cfg.pdm. - Under
pe.rsd, addcokurtosis_residuals(sigma, err, me, pe.ex)toposterior_kt.mecomes fromcokurtosis_expected_returns(pe.kte). Repair the sum withposdef!, under the estimator thatmatrix_processing_posdefgives for the processor of step 3. - When step 4 gave a
posterior_sk, computeposterior_Vfrom it withnegative_spectral_coskewness, at the countT_sk.Vthen describes the corrected coskewness, and not the projected coskewness. - Expand
posterior_kt,posterior_skandposterior_Vto the full asset universe withexpand_moment. The entries outside the investable block areNaN. The method expandsVand does not compute it again, becauseVis a spectral value of the cut coskewness, and the full frame holds no cut returns. Anothingmask needs no branch of its own. For anothingmask,expand_momentreturns the block with no change, for each count of arguments. - Build the nested factor prior result
fproverpr.fpr, from the factor moments of steps 3 to 5. Whenf_ktandf_skare bothnothing,fprisnothing. Steps 2 and 12 do not change the factor block, because the mask is on the asset axis and these co-moments are on the factor axis. - Build the asset
HighOrderPriorwith its keyword constructor, over the fullprand not the cutrpr, because every prior result covers the full asset universe. Both it andfprstate the counts(T_sk, T_kt)of step 3 ineoc. - Warn with
assert_matched_coveragewhen the co-moments hold fewer assets than the low order block.
The order of steps 9 and 10 is necessary. cokurtosis_residuals reads the systematic covariance. So step 9 must remove the residual block that the lift of the wrapped estimator added, before step 10 adds the residual cokurtosis.
Arguments
pe: High order factor prior estimator.X: Asset returns matrix (observations × assets).F: Factor returns matrix (observations × factors).pnl: OptionalAssetPanel, the panel that the returns data held. A wrapping prior forwards it unchanged, so that it can compose an estimator that is fitted on a panel. An estimator that reads no panel ignores it.dims: Dimension along which the computation runs.kwargs...: Additional keyword arguments, passed to the estimators thatpeholds.
Validation
dims in (1, 2).- The prior result of
pe.pemust hold a regression result, seeassert_prior_regression. - The wrapped estimator must declare its residual block with a method of
factor_residual_config. The verb has no default. For an estimator with no method, it throws anArgumentError, and it does not read as no residual block.assert_factor_residual_configchecks the shape of the declaration. It throws anArgumentErrorwhen the declaration is notnothingand not aNamedTuplethat holdsve,pdmandrsd. These two errors occur only whenpe.rsdistrueand a cokurtosis is present.
Returns
pr::HighOrderPrior: Result that holds the asset returns, the mean, the covariance, the coskewness, the square cokurtosis and the factor moments, on the full asset universe. An asset that the wrapped prior did not estimate holdsNaNinmu, on the diagonal ofsigma, and at each index ofkt,skandVthat names it.HighOrderPriorEstimatormakes the same prior result on the same panel with gaps, andport_opt_viewcuts theNaNentries out of it. Itsfpris a nestedHighOrderPriorover the factor block of the wrapped prior, sofpr.pr === pr.fpr. Both routes give one distribution for the factor co-moments and the low order factor moments.
Related
HighOrderFactorPriorEstimatorassert_prior_regressionHighOrderPriorHighOrderPriorEstimatorFactorPriorpriorinvestable_mask: the mask of step 2. Itsnothingvalue, for a universe in which every asset is investable, makes the method cut nothing.port_opt_view: the cut of step 2, which cuts every block that the lift reads.expand_moment: the expansion of step 12.factor_residual_config: the declaration of the residual block that step 9 removes. The bound ofpe.peisAbstractLowOrderPriorEstimator_F_AF. OnlyFactorPriorandFactorBlackLittermanPriorhold the fields that size the block. So a wrapper over either passes the declaration on, and each other estimator declaresnothingin a method of its own.assert_factor_residual_config: the check of the shape of that declaration.coskewness_residualscokurtosis_residualsscenario_rows: the cut of step 3, to the rows ofFthat the probabilities of the scenarios weigh.
PortfolioOptimisers.cokurtosis_processor — Function
cokurtosis_processor(kte::CokurtosisEstimator) -> Option{<:AbstractMatrixProcessingEstimator}
cokurtosis_processor(kte::Cokurtosis) -> AbstractMatrixProcessingEstimator
cokurtosis_processor(kte::WindowedCokurtosis) -> Option{<:AbstractMatrixProcessingEstimator}Give the matrix processing estimator that HighOrderFactorPriorEstimator applies to the cokurtosis that it lifts onto the asset axis.
The prior processes the lifted cokurtosis with this estimator. Under rsd = true, it adds the residual cokurtosis. Then it repairs the sum with the estimator that matrix_processing_posdef gives for this one.
Interfaces
A subtype of CokurtosisEstimator that goes into a HighOrderFactorPriorEstimator adds a method of this verb. The method returns nothing when the subtype needs no processing. The prior then does not process the lifted cokurtosis, and does not repair the sum. The constructor of the prior calls assert_cokurtosis_processor on kte.
Algorithm
The method that Julia selects is the algorithm.
- A
Cokurtosisgiveskte.mp. - A
WindowedCokurtosisgives the processor of the estimator that it wraps. - The fallback refuses every other subtype. A default of
nothingskips the repair of the sum underrsd = truewith no message, and the cokurtosis can then be indefinite.
Arguments
kte: Cokurtosis estimator.
Validation
- The type of
ktehas a method of its own. AnArgumentErrorthat names the type is thrown otherwise.
Returns
mp::Option{<:AbstractMatrixProcessingEstimator}: The matrix processing estimator,nothingwhen the method ofktenames none.
Related
PortfolioOptimisers.cokurtosis_expected_returns — Function
cokurtosis_expected_returns(kte::CokurtosisEstimator) -> AbstractExpectedReturnsEstimator
cokurtosis_expected_returns(kte::Cokurtosis) -> AbstractExpectedReturnsEstimator
cokurtosis_expected_returns(kte::WindowedCokurtosis) -> AbstractExpectedReturnsEstimatorGive the expected returns estimator with which HighOrderFactorPriorEstimator averages the residual moments of its correction of the cokurtosis.
cokurtosis_residuals averages the squared and the fourth-power residuals with this estimator. The prior reads it only under rsd = true.
Interfaces
A subtype of CokurtosisEstimator that goes into a HighOrderFactorPriorEstimator with rsd = true adds a method of this verb.
Algorithm
The method that Julia selects is the algorithm.
- A
Cokurtosisgiveskte.me. - A
WindowedCokurtosisgives the estimator of the estimator that it wraps. - The fallback refuses every other subtype. An average has no
nothingform. A default rule that the library selects for the caller hides a missing method.
Arguments
kte: Cokurtosis estimator.
Validation
- The type of
ktehas a method of its own. AnArgumentErrorthat names the type is thrown otherwise.
Returns
me::AbstractExpectedReturnsEstimator: The expected returns estimator of the residual moments.
Related
PortfolioOptimisers.coskewness_expected_returns — Function
coskewness_expected_returns(ske::CoskewnessEstimator) -> AbstractExpectedReturnsEstimator
coskewness_expected_returns(ske::Coskewness) -> AbstractExpectedReturnsEstimator
coskewness_expected_returns(ske::WindowedCoskewness) -> AbstractExpectedReturnsEstimatorGive the expected returns estimator with which HighOrderFactorPriorEstimator averages the residual moments of its correction of the coskewness.
coskewness_residuals averages the cubed residuals with this estimator. The prior reads it only under rsd = true.
Interfaces
A subtype of CoskewnessEstimator that goes into a HighOrderFactorPriorEstimator with rsd = true adds a method of this verb.
Algorithm
The method that Julia selects is the algorithm.
- A
Coskewnessgivesske.me. - A
WindowedCoskewnessgives the estimator of the estimator that it wraps. - The fallback refuses every other subtype, for the reason that
cokurtosis_expected_returnsstates.
Arguments
ske: Coskewness estimator.
Validation
- The type of
skehas a method of its own. AnArgumentErrorthat names the type is thrown otherwise.
Returns
me::AbstractExpectedReturnsEstimator: The expected returns estimator of the residual moments.
Related
References
- [164]
- K. Boudt, W. Lu and B. Peeters. Higher order comoments of multifactor models and asset allocation. Finance Research Letters 13, 225–233 (2015).
- [165]
- L. Martellini and V. Ziemann. Improved estimates of higher-order comoments and implications for portfolio selection. The Review of Financial Studies 23, 1467–1502 (2010).