High order factor Prior estimator

PortfolioOptimisers.HighOrderFactorPriorEstimator — Type
struct HighOrderFactorPriorEstimator{__T_pe, __T_kte, __T_ske, __T_ex, __T_rsd, __T_cache, __T_x_src} <: AbstractHighOrderPriorEstimator_F

Projects factor coskewness and cokurtosis onto the asset axis through the regression loadings.

HighOrderFactorPriorEstimator holds a low order factor prior estimator, a cokurtosis estimator and a coskewness estimator. Its prior fits the co-moments of the factor returns, and lifts them onto the asset axis with the loadings of the regression. When rsd is true, it adds the residual co-moments of the regression.

Fields

  • pe: Prior estimator.
  • kte: Cokurtosis estimator.
  • ske: Coskewness estimator.
  • ex: Parallel execution strategy.
  • rsd: Whether to include residual variance in the posterior covariance.
  • cache: Optional partial-fit state. It is nothing until partial_fit! writes one. When the caller gives no data matrix, the batch verb of the estimator reads it. Each propagation channel does one thing with it. factory keeps it unchanged, because a factory call resolves the configuration, not the sample. port_opt_view copies the entries of the selected assets into a new state, so the viewed estimator holds the state of those assets alone. obs_weights_view removes it, because a state has no slice on the observation axis. Only a type with an observation-weights field has that channel. A type with no such field falls through to the identity method, so it keeps its state. A family whose state has no exact asset slice removes it on both axes and names the reason.
  • x_src: The source of the returns that the prior fits on, nothing or an AbstractReturnsSource. nothing and DataSource read the asset returns rd.X. ExcessSource reads the returns in excess of the benchmark, rd.X - rd.B, the Excess Frame, and refuses returns data with no benchmark. nothing means that the caller stated no source, so an optimiser that holds a reader of the Excess Frame warns about a prior at nothing, and not about a prior at DataSource(). A prior reads no prior, so it refuses PriorSource. A prior inside another estimator, such as the pe of a FactorPrior, keeps nothing, and the outer estimator states the source.

Constructors

HighOrderFactorPriorEstimator(;    pe::AbstractLowOrderPriorEstimator_F_AF = FactorPrior(),    kte::Option{<:CokurtosisEstimator} = Cokurtosis(; alg = FullMoment()),    ske::Option{<:CoskewnessEstimator} = Coskewness(; alg = FullMoment()),    ex::FLoops.Transducers.Executor = FLoops.ThreadedEx(),    rsd::Bool = true,    cache::Option{<:AbstractPartialFitState} = nothing,    x_src::Option{<:AbstractReturnsSource} = nothing) -> HighOrderFactorPriorEstimator

Keywords correspond to the struct's fields.

Validation

Propagated parameters

factory passes its arguments on to these @fprop fields:

  • pe: Recursively updated through factory.
  • kte: Recursively updated through factory.
  • ske: Recursively updated through factory.
  • cache: Carried unchanged through factory.

View parameters

port_opt_view takes the view of these @vprop fields at the selected indices:

Examples

julia> HighOrderFactorPriorEstimator()HighOrderFactorPriorEstimator     pe ┼ FactorPrior        │         pe ┼ EmpiricalPrior        │            │           ce ┼ PortfolioOptimisersCovariance        │            │              │   ce ┼ Covariance        │            │              │      │    me ┼ SimpleExpectedReturns        │            │              │      │       │   w ┴ nothing        │            │              │      │    ce ┼ GeneralCovariance        │            │              │      │       │   ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true)        │            │              │      │       │    w ┴ nothing        │            │              │      │   alg ┼ FullMoment()        │            │              │      │     w ┴ nothing        │            │              │   mp ┼ MatrixProcessing        │            │              │      │     pdm ┼ Posdef        │            │              │      │         │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton        │            │              │      │         │   kwargs ┴ @NamedTuple{}: NamedTuple()        │            │              │      │      dn ┼ nothing        │            │              │      │      dt ┼ nothing        │            │              │      │     alg ┼ nothing        │            │              │      │   order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg)        │            │           me ┼ SimpleExpectedReturns        │            │              │   w ┴ nothing        │            │      horizon ┼ nothing        │            │   fill_limit ┼ nothing        │            │        x_src ┴ nothing        │         mp ┼ MatrixProcessing        │            │     pdm ┼ Posdef        │            │         │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton        │            │         │   kwargs ┴ @NamedTuple{}: NamedTuple()        │            │      dn ┼ nothing        │            │      dt ┼ nothing        │            │     alg ┼ nothing        │            │   order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg)        │         re ┼ StepwiseRegression        │            │       crit ┼ PValue        │            │            │   t ┴ Float64: 0.05        │            │        alg ┼ ForwardSelection()        │            │        tgt ┼ LinearModel        │            │            │   kwargs ┴ @NamedTuple{}: NamedTuple()        │            │     choice ┼ BatchChoice()        │            │   included ┼ nothing        │            │      x_src ┴ nothing        │         ve ┼ SimpleVariance        │            │          me ┼ SimpleExpectedReturns        │            │             │   w ┴ nothing        │            │           w ┼ nothing        │            │   corrected ┴ Bool: true        │        rsd ┼ Bool: true        │   mtx_sqrt ┼ EigenFallbackSquareRoot()        │      x_src ┴ nothing    kte ┼ Cokurtosis        │      me ┼ SimpleExpectedReturns        │         │   w ┴ nothing        │      mp ┼ MatrixProcessing        │         │     pdm ┼ Posdef        │         │         │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton        │         │         │   kwargs ┴ @NamedTuple{}: NamedTuple()        │         │      dn ┼ nothing        │         │      dt ┼ nothing        │         │     alg ┼ nothing        │         │   order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg)        │     alg ┼ FullMoment()        │       w ┼ nothing        │   cache ┴ nothing    ske ┼ Coskewness        │      me ┼ SimpleExpectedReturns        │         │   w ┴ nothing        │      mp ┼ MatrixProcessing        │         │     pdm ┼ Posdef        │         │         │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton        │         │         │   kwargs ┴ @NamedTuple{}: NamedTuple()        │         │      dn ┼ nothing        │         │      dt ┼ nothing        │         │     alg ┼ nothing        │         │   order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg)        │     alg ┼ FullMoment()        │       w ┼ nothing        │   cache ┴ nothing     ex ┼ Transducers.ThreadedEx{@NamedTuple{}}: Transducers.ThreadedEx()    rsd ┼ Bool: true  x_src ┴ nothing

The incremental fit

This prior has no exact incremental recursion, and it takes the online step by a refit from a sample buffer. Online seeds cache, and partial_fit! adds each observation to it with no change. The prior method with one argument then runs the batch verb of this estimator over the rows that the buffer kept. The result is the batch fit over those rows, and a max_history on the wrapper sets the window of the whole fit.

cache goes through the three propagation channels as every partial-fit state does. factory carries it with no change, and port_opt_view cuts it to the selected assets. obs_weights_view drops it, because a state has no cut on the observation axis. The show method does not print cache, because a buffer that changes at each step is not part of the configuration.

Related

References

  • [164] K. Boudt, W. Lu and B. Peeters. Higher order comoments of multifactor models and asset allocation. Finance Research Letters 13, 225–233 (2015). Pages 225–233. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
  • [165] L. Martellini and V. Ziemann. Improved estimates of higher-order comoments and implications for portfolio selection. The Review of Financial Studies 23, 1467–1502 (2010).
source
PortfolioOptimisers.prior — Function
prior(pe::HighOrderFactorPriorEstimator, X::MatNum, F::MatNum,
      pnl::Option{<:AssetPanel} = nothing; dims::Int = 1,
      kwargs...)

Compute the high order prior of the asset returns through a factor model.

prior fits the low order moments with the embedded factor prior pe.pe. It fits the co-moments of the factor returns, and maps them to the asset axis with the Kronecker product of the loadings. When pe.rsd is true, it adds the residual co-moments.

Note

A Black-Litterman prior can go under this estimator. Each wrapper estimator passes rr and the factor block on, so HighOrderFactorPriorEstimator(; pe = BlackLittermanPrior(; pe = FactorPrior(…))) gets to a regression, and it does not throw an IsNothingError.

The co-moments go through rr.M, and mu and sigma hold the views. Black-Litterman gives no third or fourth moments, so the projection of the factor co-moments is the only estimate of them.

Warning

The method computes the co-moments from F as the caller gives it. So they describe the factor distribution before the views, for each Black-Litterman prior under this estimator. FactorBlackLittermanPrior and BayesianBlackLittermanPrior give a posterior factor block. Under them, the orders of the nested fpr do not agree. fpr.mu and fpr.sigma hold the views, and fpr.kt, fpr.sk and fpr.V do not. The invariant fpr.pr === pr.fpr still holds, because both routes get the same posterior low order block. The views stop at the second order.

The cause is that Black-Litterman has no update for the third and fourth moments. The method discards no value. BlackLittermanPrior behaves in the same way. Its low order factor block is also before the views, so all the orders of its prior result agree.

An entropy pooling prior and an opinion pooling prior are different. They state their views and opinions as the probabilities w of the scenarios. The factor co-moments read w, so the views get to every order.

Mathematical definition

The method maps the factor co-moments to the asset axis through the loadings matrix $\mathbf{B}$:

\[\begin{align} \hat{\mathbf{\Sigma}}_4 &= (\mathbf{B} \otimes \mathbf{B}) \hat{\mathbf{\Sigma}}_{4,f} (\mathbf{B} \otimes \mathbf{B})^\intercal + \hat{\mathbf{\Sigma}}_{4,\varepsilon}\,, \\ \hat{\mathbf{M}}_3 &= \mathbf{B} \hat{\mathbf{M}}_{3,f} (\mathbf{B} \otimes \mathbf{B})^\intercal + \hat{\mathbf{M}}_{3,\varepsilon}\,. \end{align}\]

Where:

  • $\hat{\mathbf{\Sigma}}_4$: $N^2 \times N^2$ asset square cokurtosis matrix, kt.
  • $\hat{\mathbf{M}}_3$: $N \times N^2$ asset coskewness matrix, sk.
  • $\mathbf{B}$: $N \times K$ factor loadings matrix, pr.rr.M.
  • $\hat{\mathbf{\Sigma}}_{4,f}$: $K^2 \times K^2$ factor square cokurtosis matrix, fpr.kt.
  • $\hat{\mathbf{M}}_{3,f}$: $K \times K^2$ factor coskewness matrix, fpr.sk.
  • $\hat{\mathbf{\Sigma}}_{4,\varepsilon}$: Residual cokurtosis from cokurtosis_residuals. It is present only when rsd is true.
  • $\hat{\mathbf{M}}_{3,\varepsilon}$: Residual coskewness from coskewness_residuals. It is present only when rsd is true.
  • $\otimes$: Kronecker product.

pe.kte and pe.ske fit the factor co-moments on F. When the low order block gives the probabilities w of its scenarios, w replaces the observation weights of both estimators. The fit then reads the last length(w) rows of F, which w weighs. The residual corrections do not read w. The residuals belong to the model, and the views change the probabilities of the scenarios, not the model. A different alg on either estimator gives a different formula in place of its display above. When an estimator is nothing, the asset result and the nested factor block both hold nothing for its moment.

Algorithm

  1. Orient X and F to observations × variables with dims_oriented.
  2. Compute the low order block pr with pe.pe, and check that it holds a regression result. Derive the Investable Mask from it with investable_mask. Cut the whole prior result to the investable universe with port_opt_view, and call the result rpr. X is an argument and not a block of the prior result, so cut it in the same way, and call it Xr. Take the reconstructed returns posterior_X = rpr.X and the loadings M = rpr.rr.M. When every asset is investable, the mask is nothing, the method cuts nothing, and the steps below run on pr. The cut comes before the lift, because the lift spreads a non-investable asset much further. One NaN row of loadings makes a full NaN band of kron(M, M), and the projection spreads it over most of the cokurtosis. posdef! then refuses the matrix with a LAPACK error that does not name the asset.
  3. Read the probabilities w = pr.w of the scenarios. Cut F to the rows Fw that w weighs, with scenario_rows. Read the counts T_sk and T_kt of factory(pe.ske, w) and factory(pe.kte, w) over Fw with high_order_observation_count. Compute the factor square cokurtosis f_kt with factory(pe.kte, w) on Fw. When f_kt is not nothing, build kM = kron(M, M), and project posterior_kt = kM * f_kt * transpose(kM). Process posterior_kt with the estimator that cokurtosis_processor(pe.kte) gives, through sample_matrix_processing! at the count T_kt.
  4. Compute the factor coskewness f_sk and its negative spectral form f_V with factory(pe.ske, w) on Fw. When f_sk is not nothing, build kM if step 3 did not, and project posterior_sk = M * f_sk * transpose(kM).
  5. Build the structure matrices with dup_elim_sum_matrices two times. Build D2, L2 and S2 at the full asset count, and f_D2, f_L2 and f_S2 at the factor count. The asset count is the full count, because step 12 expands the co-moments, and the constructor checks the matrices against length(pr.mu). At both dimensions, the method selects the matrices by the rule of prior(::HighOrderPriorEstimator, …).
  6. When pe.rsd is true, take the reconstruction error err = Xr - posterior_X. Under a Scenario Cap, posterior_X holds only the last rows. Then the method uses F * transpose(M) .+ transpose(b) over every row in place of posterior_X, as the residual block of sigma does.
  7. Under pe.rsd, when step 4 gave a posterior_sk, add coskewness_residuals(err, me) to it. me comes from coskewness_expected_returns(pe.ske).
  8. Under pe.rsd, when step 3 gave a posterior_kt, read the declaration of the residual block of the wrapped estimator with factor_residual_config. Check its shape with assert_factor_residual_config.
  9. Get the systematic covariance sigma from rpr.sigma. A nothing declaration, or a declaration whose rsd is false, means that the wrapped estimator added no residual block. Then sigma is rpr.sigma with no change. Otherwise, read the block with idiosyncratic_variances of rpr.rr, and call it err_sigma. These are the residual variances that the wrapped estimator added, under the weights of its own fit. A variance of err under rsd_cfg.ve does not read these weights. Subtract the diagonal matrix of err_sigma from rpr.sigma. When an entry of err_sigma is larger than the diagonal entry of rpr.sigma that matches it, the subtraction gives a negative variance. Then the step warns, and keeps rpr.sigma whole. This happens when the covariance of the wrapped estimator does not hold the block, such as a posterior that shrinks the sum of the lift. In both cases, repair the result with posdef! under rsd_cfg.pdm.
  10. Under pe.rsd, add cokurtosis_residuals(sigma, err, me, pe.ex) to posterior_kt. me comes from cokurtosis_expected_returns(pe.kte). Repair the sum with posdef!, under the estimator that matrix_processing_posdef gives for the processor of step 3.
  11. When step 4 gave a posterior_sk, compute posterior_V from it with negative_spectral_coskewness, at the count T_sk. V then describes the corrected coskewness, and not the projected coskewness.
  12. Expand posterior_kt, posterior_sk and posterior_V to the full asset universe with expand_moment. The entries outside the investable block are NaN. The method expands V and does not compute it again, because V is a spectral value of the cut coskewness, and the full frame holds no cut returns. A nothing mask needs no branch of its own. For a nothing mask, expand_moment returns the block with no change, for each count of arguments.
  13. Build the nested factor prior result fpr over pr.fpr, from the factor moments of steps 3 to 5. When f_kt and f_sk are both nothing, fpr is nothing. Steps 2 and 12 do not change the factor block, because the mask is on the asset axis and these co-moments are on the factor axis.
  14. Build the asset HighOrderPrior with its keyword constructor, over the full pr and not the cut rpr, because every prior result covers the full asset universe. Both it and fpr state the counts (T_sk, T_kt) of step 3 in eoc.
  15. Warn with assert_matched_coverage when the co-moments hold fewer assets than the low order block.

The order of steps 9 and 10 is necessary. cokurtosis_residuals reads the systematic covariance. So step 9 must remove the residual block that the lift of the wrapped estimator added, before step 10 adds the residual cokurtosis.

Arguments

  • pe: High order factor prior estimator.
  • X: Asset returns matrix (observations × assets).
  • F: Factor returns matrix (observations × factors).
  • pnl: Optional AssetPanel, the panel that the returns data held. A wrapping prior forwards it unchanged, so that it can compose an estimator that is fitted on a panel. An estimator that reads no panel ignores it.
  • dims: Dimension along which the computation runs.
  • kwargs...: Additional keyword arguments, passed to the estimators that pe holds.

Validation

  • dims in (1, 2).
  • The prior result of pe.pe must hold a regression result, see assert_prior_regression.
  • The wrapped estimator must declare its residual block with a method of factor_residual_config. The verb has no default. For an estimator with no method, it throws an ArgumentError, and it does not read as no residual block. assert_factor_residual_config checks the shape of the declaration. It throws an ArgumentError when the declaration is not nothing and not a NamedTuple that holds ve, pdm and rsd. These two errors occur only when pe.rsd is true and a cokurtosis is present.

Returns

  • pr::HighOrderPrior: Result that holds the asset returns, the mean, the covariance, the coskewness, the square cokurtosis and the factor moments, on the full asset universe. An asset that the wrapped prior did not estimate holds NaN in mu, on the diagonal of sigma, and at each index of kt, sk and V that names it. HighOrderPriorEstimator makes the same prior result on the same panel with gaps, and port_opt_view cuts the NaN entries out of it. Its fpr is a nested HighOrderPrior over the factor block of the wrapped prior, so fpr.pr === pr.fpr. Both routes give one distribution for the factor co-moments and the low order factor moments.

Related

source
PortfolioOptimisers.cokurtosis_processor — Function
cokurtosis_processor(kte::CokurtosisEstimator) -> Option{<:AbstractMatrixProcessingEstimator}
cokurtosis_processor(kte::Cokurtosis) -> AbstractMatrixProcessingEstimator
cokurtosis_processor(kte::WindowedCokurtosis) -> Option{<:AbstractMatrixProcessingEstimator}

Give the matrix processing estimator that HighOrderFactorPriorEstimator applies to the cokurtosis that it lifts onto the asset axis.

The prior processes the lifted cokurtosis with this estimator. Under rsd = true, it adds the residual cokurtosis. Then it repairs the sum with the estimator that matrix_processing_posdef gives for this one.

Interfaces

A subtype of CokurtosisEstimator that goes into a HighOrderFactorPriorEstimator adds a method of this verb. The method returns nothing when the subtype needs no processing. The prior then does not process the lifted cokurtosis, and does not repair the sum. The constructor of the prior calls assert_cokurtosis_processor on kte.

Algorithm

The method that Julia selects is the algorithm.

  1. A Cokurtosis gives kte.mp.
  2. A WindowedCokurtosis gives the processor of the estimator that it wraps.
  3. The fallback refuses every other subtype. A default of nothing skips the repair of the sum under rsd = true with no message, and the cokurtosis can then be indefinite.

Arguments

  • kte: Cokurtosis estimator.

Validation

  • The type of kte has a method of its own. An ArgumentError that names the type is thrown otherwise.

Returns

  • mp::Option{<:AbstractMatrixProcessingEstimator}: The matrix processing estimator, nothing when the method of kte names none.

Related

source
PortfolioOptimisers.cokurtosis_expected_returns — Function
cokurtosis_expected_returns(kte::CokurtosisEstimator) -> AbstractExpectedReturnsEstimator
cokurtosis_expected_returns(kte::Cokurtosis) -> AbstractExpectedReturnsEstimator
cokurtosis_expected_returns(kte::WindowedCokurtosis) -> AbstractExpectedReturnsEstimator

Give the expected returns estimator with which HighOrderFactorPriorEstimator averages the residual moments of its correction of the cokurtosis.

cokurtosis_residuals averages the squared and the fourth-power residuals with this estimator. The prior reads it only under rsd = true.

Interfaces

A subtype of CokurtosisEstimator that goes into a HighOrderFactorPriorEstimator with rsd = true adds a method of this verb.

Algorithm

The method that Julia selects is the algorithm.

  1. A Cokurtosis gives kte.me.
  2. A WindowedCokurtosis gives the estimator of the estimator that it wraps.
  3. The fallback refuses every other subtype. An average has no nothing form. A default rule that the library selects for the caller hides a missing method.

Arguments

  • kte: Cokurtosis estimator.

Validation

  • The type of kte has a method of its own. An ArgumentError that names the type is thrown otherwise.

Returns

  • me::AbstractExpectedReturnsEstimator: The expected returns estimator of the residual moments.

Related

source
PortfolioOptimisers.coskewness_expected_returns — Function
coskewness_expected_returns(ske::CoskewnessEstimator) -> AbstractExpectedReturnsEstimator
coskewness_expected_returns(ske::Coskewness) -> AbstractExpectedReturnsEstimator
coskewness_expected_returns(ske::WindowedCoskewness) -> AbstractExpectedReturnsEstimator

Give the expected returns estimator with which HighOrderFactorPriorEstimator averages the residual moments of its correction of the coskewness.

coskewness_residuals averages the cubed residuals with this estimator. The prior reads it only under rsd = true.

Interfaces

A subtype of CoskewnessEstimator that goes into a HighOrderFactorPriorEstimator with rsd = true adds a method of this verb.

Algorithm

The method that Julia selects is the algorithm.

  1. A Coskewness gives ske.me.
  2. A WindowedCoskewness gives the estimator of the estimator that it wraps.
  3. The fallback refuses every other subtype, for the reason that cokurtosis_expected_returns states.

Arguments

  • ske: Coskewness estimator.

Validation

  • The type of ske has a method of its own. An ArgumentError that names the type is thrown otherwise.

Returns

  • me::AbstractExpectedReturnsEstimator: The expected returns estimator of the residual moments.

Related

source

References

[164]
K. Boudt, W. Lu and B. Peeters. Higher order comoments of multifactor models and asset allocation. Finance Research Letters 13, 225–233 (2015).
[165]
L. Martellini and V. Ziemann. Improved estimates of higher-order comoments and implications for portfolio selection. The Review of Financial Studies 23, 1467–1502 (2010).