Factor Prior
PortfolioOptimisers.FactorPrior — Type
struct FactorPrior{__T_pe, __T_mp, __T_re, __T_ve, __T_rsd, __T_mtx_sqrt, __T_cache, __T_x_src} <: AbstractLowOrderPriorEstimator_FFactor-based prior estimator for asset returns.
FactorPrior is a low order prior estimator that computes the mean and covariance of asset returns from a factor model. pe fits the factor moments on the factor returns, and re fits the loadings of each asset on the factors. The estimator projects the factor moments through the loadings, and processes the covariance with mp. When rsd is true, it adds to the covariance the residual variances that ve measures.
The covariance is the plug-in estimator of the factor model that Fan et al. [151] state. It is the loadings times the factor covariance times the transposed loadings, plus the diagonal matrix of the residual variances. The library differs from the source in three ways:
- The mean adds the regression intercepts. The model of the source has no intercepts.
- The default
reis stepwise, so an asset can drop a factor. The source fits every factor by least squares. vesizes the residual variances. The defaultSimpleVariancedivides byT - 1, and the source divides byT.
Fields
pe: Prior estimator.
mp: Matrix processing estimator.
re: Regression estimator.
ve: Variance estimator.
rsd: Whether to include residual variance in the posterior covariance.
mtx_sqrt: Square-root algorithm of the factor covariance, and of the idiosyncratic covariance when that block is a full matrix, whose square roots make upchol. The defaultEigenFallbackSquareRoottakes the Cholesky factor where it exists, and the eigen square root of a singular positive semidefinite matrix otherwise.RidgeCholeskySquareRoottakes the Cholesky factor of the matrix with a small ridge added, andnothingtakes the plain Cholesky factor, which raises aLinearAlgebra.PosDefExceptionon a matrix that is not positive definite.matrix_square_rootstates each algorithm.
cache: Optional partial-fit state. It isnothinguntilpartial_fit!writes one. When the caller gives no data matrix, the batch verb of the estimator reads it. Each propagation channel does one thing with it.factorykeeps it unchanged, because a factory call resolves the configuration, not the sample.port_opt_viewcopies the entries of the selected assets into a new state, so the viewed estimator holds the state of those assets alone.obs_weights_viewremoves it, because a state has no slice on the observation axis. Only a type with an observation-weights field has that channel. A type with no such field falls through to the identity method, so it keeps its state. A family whose state has no exact asset slice removes it on both axes and names the reason.
x_src: The source of the returns that the prior fits on,nothingor anAbstractReturnsSource.nothingandDataSourceread the asset returnsrd.X.ExcessSourcereads the returns in excess of the benchmark,rd.X - rd.B, the Excess Frame, and refuses returns data with no benchmark.nothingmeans that the caller stated no source, so an optimiser that holds a reader of the Excess Frame warns about a prior atnothing, and not about a prior atDataSource(). A prior reads no prior, so it refusesPriorSource. A prior inside another estimator, such as thepeof aFactorPrior, keepsnothing, and the outer estimator states the source.
Constructors
FactorPrior(; pe::AbstractLowOrderPriorEstimator_A_AF = EmpiricalPrior(), mp::AbstractMatrixProcessingEstimator = MatrixProcessing(), re::AbstractTimeSeriesRegressionEstimator = StepwiseRegression(), ve::AbstractVarianceEstimator = SimpleVariance(), rsd::Bool = true, mtx_sqrt::Option{<:AbstractMatrixSquareRootAlgorithm} = EigenFallbackSquareRoot(), cache::Option{<:AbstractPartialFitState} = nothing, x_src::Option{<:AbstractReturnsSource} = nothing) -> FactorPriorKeywords correspond to the struct's fields.
Validation
x_srcisnothing,DataSource()orExcessSource(), seeassert_returns_source.- The
x_srcofpe,reisnothing, because the prior picks the returns that it reads, seeassert_inner_reader. - When
rsdistrue,mpstates the estimator that repairs the covariance, seeassert_matrix_processing_posdef.
Propagated parameters
factory passes its arguments on to these @fprop fields:
pe: Recursively updated throughfactory.re: Recursively updated throughfactory.ve: Recursively updated throughfactory.cache: Carried unchanged throughfactory.
View parameters
port_opt_view takes the view of these @vprop fields at the selected indices:
re: Recursively viewed throughport_opt_view.ve: Recursively viewed throughport_opt_view.cache: Sliced to the selected assets throughport_opt_view.
The incremental fit
This prior has no exact incremental recursion, so the online step refits from a sample buffer. Online seeds cache, and partial_fit! appends each observation to it. The one-argument prior runs the batch verb of this estimator over the rows in the buffer. The answer equals a batch fit over those rows, and a max_history on the wrapper cuts the whole fit.
obs_weights_view drops cache, because a partial-fit state has no slice on the observation axis. show_fields does not render it, because a running buffer is not part of the configuration.
Composition: what this estimator forwards
This estimator lifts a factor-axis prior onto the asset axis. It rebuilds X as F * transpose(M) .+ transpose(b), so it builds its prior result itself and does not forward one along its own axis. The forwarding rule still governs each field. This estimator does not change the factor distribution. FactorBlackLittermanPrior is this estimator with views on the factor block before the lift.
- The factor block
fpris the result of the wrapped factor prior, forwarded whole. It needs no reconstruction, because the asset moments are its projection and not an update of it. muandsigmaare that block projected through the loadings, somu == rr.M * fpr.mu + rr.bholds for every investable asset. Whenrsdistrue,sigmaalso holds the residual variances.cholis not forwarded. The lift rebuilds it on the asset axis fromM * matrix_square_root(mtx_sqrt, fpr.sigma), and adds the residual block whenrsdistrue. It factorisessigmabefore matrix processing, seefactor_lift. Anmpthat denoises or detones movessigmaand leavescholas it was.wis the observation weights of the factor prior. This estimator wraps only a factor prior, andposterior_Xhas the rows ofF, so these are the only weights, and they are on the correct axis. Theens,kldandowof the factor prior come with them.
Examples
julia> FactorPrior()FactorPrior pe ┼ EmpiricalPrior │ ce ┼ PortfolioOptimisersCovariance │ │ ce ┼ Covariance │ │ │ me ┼ SimpleExpectedReturns │ │ │ │ w ┴ nothing │ │ │ ce ┼ GeneralCovariance │ │ │ │ ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true) │ │ │ │ w ┴ nothing │ │ │ alg ┼ FullMoment() │ │ │ w ┴ nothing │ │ mp ┼ MatrixProcessing │ │ │ pdm ┼ Posdef │ │ │ │ alg ┼ UnionAll: NearestCorrelationMatrix.Newton │ │ │ │ kwargs ┴ @NamedTuple{}: NamedTuple() │ │ │ dn ┼ nothing │ │ │ dt ┼ nothing │ │ │ alg ┼ nothing │ │ │ order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg) │ me ┼ SimpleExpectedReturns │ │ w ┴ nothing │ horizon ┼ nothing │ fill_limit ┼ nothing │ x_src ┴ nothing mp ┼ MatrixProcessing │ pdm ┼ Posdef │ │ alg ┼ UnionAll: NearestCorrelationMatrix.Newton │ │ kwargs ┴ @NamedTuple{}: NamedTuple() │ dn ┼ nothing │ dt ┼ nothing │ alg ┼ nothing │ order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg) re ┼ StepwiseRegression │ crit ┼ PValue │ │ t ┴ Float64: 0.05 │ alg ┼ ForwardSelection() │ tgt ┼ LinearModel │ │ kwargs ┴ @NamedTuple{}: NamedTuple() │ choice ┼ BatchChoice() │ included ┼ nothing │ x_src ┴ nothing ve ┼ SimpleVariance │ me ┼ SimpleExpectedReturns │ │ w ┴ nothing │ w ┼ nothing │ corrected ┴ Bool: true rsd ┼ Bool: true mtx_sqrt ┼ EigenFallbackSquareRoot() x_src ┴ nothingRelated
AbstractLowOrderPriorEstimator_FAbstractLowOrderPriorEstimator_A_AFAbstractMatrixProcessingEstimatorAbstractTimeSeriesRegressionEstimatorAbstractVarianceEstimatorEmpiricalPriorStepwiseRegressionSimpleVariancepriorfactoryport_opt_view
References
- [151] J. Fan, Y. Fan and J. Lv. High dimensional covariance matrix estimation using a factor model. Journal of Econometrics 147, 186–197 (2008). Section 1.2, Equations 1.3 and 1.4. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
PortfolioOptimisers.prior — Function
prior(pe::FactorPrior, X::MatNum, F::MatNum, pnl::Option{<:AssetPanel} = nothing;
dims::Int = 1, strict::Bool = false, ts::Option{<:VecDate} = nothing,
active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing, cache_reg::Bool = false,
kwargs...)Compute the prior moments of asset returns from a factor model.
pe.pe fits the factor moments on F, and pe.re fits the loadings of the assets on the factors. factor_lift projects the factor moments onto the assets through the loadings. When pe.rsd is true, the lift adds the residual variances to the covariance. The method returns a LowOrderPrior.
Mathematical definition
The factor model maps the factor moments to the assets through the loadings matrix $\mathbf{B}$ and the intercepts $\boldsymbol{\alpha}$:
\[\begin{align} \hat{\boldsymbol{\mu}} &= \mathbf{B} \hat{\boldsymbol{f}} + \boldsymbol{\alpha}\,, \\ \hat{\mathbf{\Sigma}} &= \mathbf{B} \mathbf{\Sigma}_f \mathbf{B}^\intercal + \mathbf{\Sigma}_\varepsilon\,. \end{align}\]
Where:
- $\mathbf{B}$: $N \times K$ factor loadings matrix,
rr.M. - $\hat{\boldsymbol{f}}$: $K \times 1$ vector of factor expected returns.
- $\boldsymbol{\alpha}$: $N \times 1$ vector of regression intercepts,
rr.b. - $\mathbf{\Sigma}_f$: $K \times K$ factor covariance matrix.
- $\mathbf{\Sigma}_\varepsilon$: $N \times N$ diagonal matrix of residual variances, present only when
rsdistrue.
pe.pe fits $\hat{\boldsymbol{f}}$ and $\mathbf{\Sigma}_f$ on F, and pe.re fits the loadings on (X, F). factor_lift computes the two equations.
Algorithm
- Orient
XandFwithdims_oriented, toobservations × assetsandobservations × factors. - Fit the wrapped prior
pe.peonFto getf_prior, the factor-axis prior result. The method passesstrictto it, becausepe.pecan be aBlackLittermanPrioror anEntropyPoolingPrior, which resolve view names against a universe. The method also passescache_reg, so that such a prior stores the cache that a refit under new scenario weights reads fromfpr. - Reduce
Xto its Coverage Universe withcoverage_reduction. When the caller givesactive_mask, the reduction reads it in place of the active mask ofpnl. Take the view ofpe.reat the Coverage Universe withcoverage_regression. Fit the loadings and rebuild the asset returns withfactor_reconstructionto getrrandposterior_X. - Project
f_prior.muandf_prior.sigmathroughrrwithfactor_liftto getmu,sigma,chol,esigma,edofandediv. The lift processes the covariance at the count off_prior.sigma, frommoment_observation_count. - Write
esigma,edofandedivontorrwithset_idiosyncratic_covariance. Underpe.rsd = truethe fields hold the residual variances that the lift measured and their counts. Underpe.rsd = false,esigmaandedivholdnothing, because the lift added no residual block. - Let
rbe the rows thatf_prior.Xcarries, which are the lastsize(f_prior.X, 1)observations. A Scenario Cap ofpe.pekeeps only the last rows ofF. The asset side keeps the same rows, so the two blocks describe the same observations. With no cap,ris every observation. Steps 2 to 4 fit the moments over every observation, so the cut changes no moment. - Write
randts[r]ontorras its row key withset_row_key, ornothingfortswhentsisnothing. - Assemble a
LowOrderPriorover the rowsrofposterior_X, with the rowsrof the orientedXundero_X, both cut byscenario_window. It holds the three lifted moments, thew,ens,kldandowof the factor prior, the regression result underrr, andf_prioritself underfpr. It holds no Feature MatrixZ, because a Feature Matrix comes from the Asset Panel on the returns data, or from a producer on the distance that reads the loadings off this result.
Arguments
pe: Factor prior estimator.X: Asset returns matrix (observations × assets).F: Factor returns matrix (observations × factors).pnl: OptionalAssetPanel, the panel that the returns data held. A wrapping prior forwards it unchanged, so that it can compose an estimator that is fitted on a panel. An estimator that reads no panel ignores it. The nested prior fits the factors, and no panel describes the factor axis, so the method does not pass the panel on.dims: Dimension along which the computation runs.strict: Whether to throw an error ifsetsdoes not contain the desired value insets.dict[key].ts: Timestamp vector. It has one entry for each observation ofX, and the loadings result records the entries of the rows the prior result carries, ornothing. The method that takes aReturnsResultpassesrd.ts.active_mask: The active mask ofX, of the shape ofX, ornothing. When it is notnothing, it takes the place of the active mask ofpnl.cache_reg: Iftrue, a prior stores on its result the cache that its refit under new scenario weights reads: aCrossSectionalFactorPriorthe active mask and the Return Forecast history on its block, and anOpinionPoolingPriorthe rows ofpe1, its pooled weights and the cache ofpe2inpc. The refit of an entropy pooling prior around it then runs no regression and no opinion. Every other prior forwards it to the prior that it wraps. Default isfalse.kwargs...: Additional keyword arguments passed to matrix processing and estimators.
Validation
dims in (1, 2).- If
tsis notnothing,length(ts)equals the number of observations ofX. Otherwise the method raises aDimensionMismatch.
Returns
pr::LowOrderPrior: The prior result. It holds the posterior asset returns, the mean vector, the covariance matrix, the square-root factorchol, the regression result and the factor prior result.
Related
PortfolioOptimisers.prior — Method
prior(pe::FactorPrior, rd::ReturnsResult; strict::Bool = false, dims::Int = 1,
cache_reg::Bool = false) -> LowOrderPriorFit a Factor Prior from a ReturnsResult.
This method passes the fields of rd to the returns-matrix method, which runs the fit. It replaces prior(pe::AbstractPriorEstimator, rd::ReturnsResult) for this estimator, so that the timestamps rd.ts reach the fit. The loadings result then records the timestamp of each observation that the prior result carries.
Algorithm
- Check that
rdcarries asset returns and factor returns. - Check that every
UniverseSetsof the tree ofpenames the columns that it reads, withassert_universe_axis_order. A universe in the factor prior names the factors ofrd, because that prior reads the factor returns. - Drop the observations of a Shared Series gap with
shared_series_view: a gap of the factor returns, and a gap of a vector benchmark underpe.x_src = ExcessSource(). The factor prior, the regression and the reconstruction then read the same observations. - Read the asset returns with
benchmark_relative_returnsunderpe.x_src, givingX. - Call the returns-matrix method with
X,rd.Fandrd.pnl, forwardingrd.iv,rd.ivpa,rd.ne,rd.E,rd.ts,strictandcache_regas keyword arguments, anddims = 1, because aReturnsResultholds its observations along the rows. - Record the observations that step 3 dropped with
shared_series_record, andpe.x_srcwithstamp_returns_source.
Arguments
pe: Factor prior estimator.rd: The returns result to use. It must carry asset returns inrd.Xand factor returns inrd.F.strict: Passed to the returns-matrix method of the prior, where it decides if a scenario fill or a coverage peel raises in place of a warning.dims: Accepted and ignored. AReturnsResultholds its observations along the rows, so the call passesdims = 1. The method takes no other keyword that it does not list: an unknown keyword raises theMethodErrorof the language, and no estimator swallows it.cache_reg: Iftrue, a prior stores on its result the cache that its refit under new scenario weights reads: aCrossSectionalFactorPriorthe active mask and the Return Forecast history on its block, and anOpinionPoolingPriorthe rows ofpe1, its pooled weights and the cache ofpe2inpc. The refit of an entropy pooling prior around it then runs no regression and no opinion. Every other prior forwards it to the prior that it wraps. Default isfalse.
Validation
rd.Xis notnothing. Otherwise the method raises anIsNothingError.- The rules of
benchmark_relative_returns. rd.Fis notnothing, throughassert_factor_returns.- The rules of
assert_universe_axis_order. - The rules of the returns-matrix method.
Returns
pr::LowOrderPrior: The prior the returns-matrix method fitted.
Related
References
- [151]
- J. Fan, Y. Fan and J. Lv. High dimensional covariance matrix estimation using a factor model. Journal of Econometrics 147, 186–197 (2008).