Cross-Sectional Factor Prior (a)
A cross-sectional factor prior fits a factor model at each observation from the exposures in an AssetPanel, and turns the model into moments of the assets. It returns a LowOrderPrior whose rr field holds the fitted CrossSectionalFactorModel.
PortfolioOptimisers.CrossSectionalFactorPrior — Type
struct CrossSectionalFactorPrior{__T_factors, __T_neutralise, __T_families, __T_unseen, __T_cre, __T_wa, __T_pe, __T_ve, __T_ce, __T_pdm, __T_f_mp, __T_mp, __T_srep, __T_th, __T_bp, __T_mcap, __T_bw, __T_lag, __T_minra, __T_rfe, __T_lambda, __T_c, __T_ofit, __T_l1mask, __T_lx, __T_mtx_sqrt, __T_ex, __T_choice, __T_carry, __T_cache, __T_x_src} <: AbstractLowOrderPriorEstimator_AEstimates a point-in-time cross-sectional factor model from an Asset Panel, and lifts it onto the assets.
The estimator reads per-asset Panel Fields, builds a Factor Exposure from each one, regresses the returns of every observation on the lagged exposures across the assets, and returns the asset moments beside a CrossSectionalFactorModel block. It is the cross-sectional counterpart of FactorPrior, which regresses the returns of each asset on a factor-return series over time.
The factor list can also hold observed factors, whose returns the caller observes and the fit does not estimate. A member of AbstractObservedExposureEstimator declares them: CurrencyExposure gives one Currency Factor per currency, and ObservedExposure gives one factor, such as a macro series or an observed market return. Each reads its return from the Exogenous Series rd.E by name. A member that gives no factor on the data, such as a CurrencyExposure on a universe of its base currency alone, reads no series, and the fit is the fit without it. The regression reads the returns net of the observed factors: by default the fit derives them as X - Z_obs * r_obs, and lx can name a Panel Field of net returns the caller measured. cross_sectional_local_returns states both paths. Under Currency Factors the net returns are the local returns, and rd.X stays in the base currency. The observed factors then join the factor model after the estimated factors, whatever their place in the list, so the factor covariance, mu, the scenarios, the factor attribution and the exposure constraints see them, and the Neutralisation and the Factor Family Basis pass them through. An observed series can start after the first fitted observation. Its NaN rows drop the assets that hold the factor from the regression at those observations, and the factor prior pe states the moments of the factor by its own rule: the default, mask-aware, estimates them from the observations the series has, and EmpiricalPrior() leaves them NaN, so the assets that hold the factor leave the Investable Mask.
The warm-ups of a fit add up, so a caller who sizes a window must sum three of them rather than take the longest. The longest warm-up of the Descriptors sets the first observation of the factor-return history, and lag adds lag observations to it. pe then warms up over that history, and ve over the idiosyncratic returns beside it. The default pe and the default ve each need 40 observations, their min_obs, so a default fit needs 40 observations after the Descriptor warm-up and the lag. A window that covers the Descriptors alone can leave pe too few observations to state a factor covariance, and the fit then refuses with a message that names the cause. Cross-validation meets this most often. A fold gives the estimator its own rows alone, so the Descriptors warm up again in every fold, and a rolling train window never grows past the warm-up. Size the train window against the sum of the three warm-ups.
On the online step the prior takes one of two routes. Unwrapped, it folds as a carry: its first step of partial_fit! seeds a CrossSectionalCarryState, and each step computes the exposures, the regression and the idiosyncratic variance of the new observations alone. Wrapped in Online, it refits: each step records the returns, both masks and the Panel Fields in a sample buffer, the call with no data prior(pe) is the batch fit over the rows of the buffer, and a max_history on the wrapper bounds them. When a member reads the Exogenous Series, as reads_exogenous_series answers, the buffer records every column of the series too, so an observed factor takes the refit. The carry fold records the series of the rows it carries, so an observed factor takes the carry too. The automatic choice of a dropped member reads every row, so choice states whether each fit chooses again or the first fit pins it.
The Carry Rule carry states what the carry fold does with a part that has no fold of its own. Under FoldOrRefit, the default, the carry fits such a part again over every row that it reads. The call with no data stays equal to the batch fit, but the cost of the refit grows with the stream:
- A factor or a Return Forecast with no finite
lookbackmakes the carry keep every panel row, and each step computes the Factor Exposures over all of them. - A Return Forecast with no fold of its rows fits again over every row at each call with no data.
- A
vethat does not fold, assupports_partial_fitanswers, fits every idiosyncratic return again at each step. - A factor prior
pethat does not fold, ascarry_foldsanswers, fits every factor return again at each call with no data. - The idiosyncratic correlation under
th > 0with acethat does not fold is estimated again over every standardised idiosyncratic return at each call with no data. Acethat folds, as the defaultExpWeightedCovariancedoes, folds the rows of each step.
FoldOnly makes the constructor refuse each of these parts, and names it. A batch fit and the refit under Online ignore the rule.
A Scenario Cap is a route through pe. The prior builds its asset scenarios from the last rows of the factor scenarios and of the standardised idiosyncratic returns, as cross_sectional_scenarios states. So pe = EmpiricalPrior(; …, max_scenarios = w) gives the last w asset scenarios, and mu and sigma stay fitted over every observation. The max_history of Online is a different knob: it is a window on the whole fit, so the moments move with it.
ofit is the Orthogonal Forecast Fit. A fitted Return Forecast regresses the forward idiosyncratic return, which the regression makes orthogonal to the Factor Exposures, so the part of its forecast that the exposures do not span is under-scaled. The default ScoreNeutralisation neutralises the scores of such a member against every estimated factor of the block before the fit, OrthogonalPartCalibration scales the orthogonal part by a slope fitted on it, and UnadjustedForecast reads the member as it stands.
l1mask is the Leverage-One Mask. The residual of a Leverage-One Pair is zero by construction, and the default MaskedLeverageOne reads it as a gap in the standardised idiosyncratic returns, as the forecast fit, the idiosyncratic diagnostics and the attribution do. UnmaskedLeverageOne standardises it as it stands.
lambda and c are Calibration Slots. Each holds a number, or a rule that computes the number from the fit: lambda takes a rule of AbstractSpannedShrinkageCalibrationAlgorithm, and c takes a rule of AbstractOrthogonalForecastScaleCalibrationAlgorithm. Each fit, each refit and each online step resolves the rules with cross_sectional_calibration, after the split of the Return Forecast and before the mean. The block records the resolved numbers in its fields lambda and c.
Fields
factors: Pairs offactor name => Exposure Estimator. The estimated factors take the factor axis in the order they are written, and the observed factors follow them in the order they are written. A one-hot member contributes one factor per level of its categorical Panel Field, and so does aCurrencyExposure, so a Pair is not always one factor. The list holds at least one estimated member, because the cross-sectional regression fits the estimated factors, and the constructor refuses a list of observed members alone.
neutralise: Neutralisation, as Pairs ofkey => targetsrun in order, ornothing. A key names a factor or a Factor Family, and so does each target.
families: Constrained Factor Families, as Pairs offamily label => dropped member, ornothing. When the dropped member isnothing,factor_family_basischooses it.unseenstates the return of a member that no asset of an observation loads on.
unseen: Unseen Member rule of the constrained Factor Families, a member ofAbstractUnseenMemberRule. An Unseen Member is a member that no asset of positive regression weight loads on at one observation. UnderZeroUnseenMember, the default, it has a return of zero there, and the zero-sum condition of the observation holds over the other members. UnderSolvedUnseenMemberit stays in the condition, and the solve algorithm ofcregives its return, which then depends on the dropped member.
cre: Cross-Sectional Regression Estimator of the fit, and of the Neutralisation. It must fit no intercept, because the prior states the moments through the factor returns alone, and the fit refuses an intercept. A"market" => ConstantExposure()factor states the common return instead.
wa: Weight policy of the cross-sectional fit. Itspis the power the regression weights raise the market capitalisation to.
pe: Prior estimator. The fit gives it the reduced factor-return series, so a constrained Factor Family gives it a full-rank covariance. The default is anEmpiricalPriorof anExpWeightedExpectedReturnsand aRegimeAdjustedExpWeightedCovariancewithcentring = PreCentred(). Its half-life is the 40 observations of the defaultve, so the systematic part and the specific part ofsigmaanswer on one horizon, and after a change of regime both of them move. The factor covariance is centred, which is the convention ofceand ofve.
ve: Variance estimator.variance_serieson it gives the idiosyncratic variance history, whose last row is the idiosyncratic risk of the latest observation. The default is aRegimeAdjustedExpWeightedVariancewithcentring = PreCentred(), which measures the second moment of each idiosyncratic series about zero. The prior states the mean of an asset through the factors alone, so the model sets the mean of the idiosyncratic return to zero, and the specific risk is its second moment. A variance about a running mean would understate it, and would also move the regime multiplier. APortfolioOptimisersCovariancewith a step in itsmp, as its defaultmpholds, raises on this history, as its docstring states.
ce: Covariance estimator. It estimates the covariance of the standardised idiosyncratic returns, and the fit reads it only whenthis positive.gap_fill_valueon it gives the value of the cell of an inactive asset. The default givesNaN, so the fit passes the gap and the active mask of the panel to the estimator. A plain moment estimator gives zero, and the fit writes zero into the cell.
pdm: Positive definite matrix estimator. It repairs the idiosyncratic covariance alone, after the threshold, over the assets with a finite variance, and the fit reads it only whenthis positive.mp.pdmrepairs a different matrix: the asset covariance, which is the sum of the systematic block and this one. The default clips the correlation eigenvalues withClippedNearestCorrelation, because the carry fold runs this repair at each call with no data, and the clip costs less than the nearest correlation matrix.pdm = Posdef()gives the nearest correlation matrix byNearestCorrelationMatrix.Newton.
f_mp: Factor matrix processing estimator. It processes the factor covariance thatpestates, which is a different matrix from the asset covariance thatmpprocesses. The factor covariance is on the factor axis,peestimates it from the factor-return series, and a Factor Family that drops a member can leave it singular.cross_sectional_lifttakes its square root undermtx_sqrtfor the low-rank square root, so under the defaultmtx_sqrta factor covariance that is not positive definite fails there rather than in the asset block.
mp: Matrix processing estimator.
srep: Systematic Repair rule of the lift, a member ofAbstractSystematicRepair, ornothing.nothingruns every step ofmpon the systematic block except the:pdmstep, andSystematicRepairruns every step. Under each rule the lift then adds the idiosyncratic block and repairs the sum under the estimator thatmatrix_processing_posdefgives formp. The default isnothing. The batch fit and the carry fold run the same lift when they read the prior, so the carry fold equals the batch fit under each rule.
th: Idiosyncratic correlation threshold. A value of zero leaves the idiosyncratic covariance diagonal, and a positive value keeps every correlation above it and zeroes the rest, so the block becomes a matrix.
bp: Power the benchmark weights raise the market capitalisation to. A value of zero gives every asset of the estimation universe the same benchmark weight, and reads no market capitalisation.
mcap: Name of the numeric Panel Field holding the market capitalisation.
bw: Name of the numeric Panel Field the prior writes its benchmark weights onto, and the one every Exposure Estimator reads them from. The Asset Panel must not already hold a field of this name.
lag: Number of observations by which the exposures lag the returns.
minra: Smallest eligible asset count an observation may carry, ornothingformax(2K, 30)over the reduced factor countK.
rfe: Return Forecast Estimator whose forecast entersmu, ornothing. It is fitted on the coverage universe, after the factor model, and its forecast is split against the latest Factor Exposures into the part they span and the part they do not.
lambda: Spanned Shrinkage of the expected factor returns towards the spanned part of the Return Forecast, in[0, 1], or a rule that computes it. A value of one keeps the fitted factor mean, and a value of zero takes the spanned forecast alone. With no Return Forecast Estimator the spanned part is zero, so the value shrinks the factor mean towards zero. The bound isNum_SpanShrinkCal. The default isPrecisionBlend, which weighs the factor mean and the spanned part by the error of each.
c: Orthogonal Forecast Scale of the Return Forecast, in[0, ∞), or a rule that computes it. It scales the part of the forecast the factors do not span, which the block carries inb. A value of zero discards it where it is finite, and a value above one scales it up. The bound isNum_OrthFcScaleCal. An asset whose forecast is not finite carriesNaNinbwhatevercis, because0 * NaNisNaN, so the prior states no expected return for it and it leaves the Investable Mask. A forecast that is not finite at every asset splits into two zero parts, and changes no moment.
ofit: Orthogonal Forecast Fit, the rule that makes the orthogonal part of a fitted Return Forecast carry the scale of its fit.ScoreNeutralisation, the default, neutralises the scores of a member that answerstruetofits_idiosyncratic_targetagainst every estimated factor of the block, under the regression weights of the block, before the fit.OrthogonalPartCalibrationfits the calibration slope ofTargetReturnForecaston the orthogonal part of its prediction too, and scales the orthogonal part by the ratio of the two slopes.UnadjustedForecastreads the member as it stands. The rule acts beforec, which multiplies the part that it gives.
l1mask: Leverage-One Mask, the rule by which the prior reads the residual of a Leverage-One Pair when it standardises the idiosyncratic returns.MaskedLeverageOne, the default, reads it as a gap, so the scenarios give the pair the average standardised return of its observation and the idiosyncratic correlation leaves it out.UnmaskedLeverageOnereads the zero residual as it stands: zero over a positive variance, and the average over a zero variance.
lx: Name of a numeric Panel Field of the returns net of the observed factors, ornothingto derive them asX - Z_obs * r_obs. The regression reads the named field unchanged. Under Currency Factors the net returns are the local returns. A caller names a field when some asset has no currency label, which the derived path gives aNaNnet return, or when their own local returns are notX - Z_obs * r_obs, which includes every panel of simple returns because of the cross term, seecurrency_excess_index.
mtx_sqrt: Square-root algorithm of the factor covariance, and of the idiosyncratic covariance when that block is a full matrix, whose square roots make upchol. The defaultEigenFallbackSquareRoottakes the Cholesky factor where it exists, and the eigen square root of a singular positive semidefinite matrix otherwise.RidgeCholeskySquareRoottakes the Cholesky factor of the matrix with a small ridge added, andnothingtakes the plain Cholesky factor, which raises aLinearAlgebra.PosDefExceptionon a matrix that is not positive definite.matrix_square_rootstates each algorithm.
ex: Parallel execution strategy. It computes the Factor Exposures of one dependency layer, whose members read no exposure of each other. Each member writes its own columns of the exposure history, so every executor gives the same prior. The regression runs under the executor ofcre, and a Return Forecast under the executor of its own Descriptor Scores.
choice: Choice Rule of the dropped member of each Factor Family whose member isnothinginfamilies. UnderBatchChoice, the default, each fit chooses again over every observation. UnderPinnedChoice, the online step writes the choice of the first fit intofamiliesof the estimator that it returns. The two agree in a batch fit. On the carry fold, a step that moves the member underBatchChoicesolves no observation of full rank again: it selects the columns of the raw factor returns that the new member keeps, and folds the factor prior again over every carried factor return, ascross_sectional_fold_movestates. SoFoldOnlyrefusesBatchChoicewith an automatic member.
carry: Carry Rule of the carry fold, a member ofAbstractCarryRule.FoldOrRefit, the default, folds each part that has a fold and fits every other part again at each step, so the carry fold equals the batch fit at a cost that can grow with the stream.FoldOnlymakes the constructor refuse a part whose step cost grows, ascarry_growing_partslists it. A batch fit and the refit underOnlineignore the rule.
cache: Optional partial-fit state. It isnothinguntilpartial_fit!writes one. When the caller gives no data matrix, the batch verb of the estimator reads it. Each propagation channel does one thing with it.factorykeeps it unchanged, because a factory call resolves the configuration, not the sample.port_opt_viewcopies the entries of the selected assets into a new state, so the viewed estimator holds the state of those assets alone.obs_weights_viewremoves it, because a state has no slice on the observation axis. Only a type with an observation-weights field has that channel. A type with no such field falls through to the identity method, so it keeps its state. A family whose state has no exact asset slice removes it on both axes and names the reason.
x_src: The source of the returns that the prior fits on,nothingor anAbstractReturnsSource.nothingandDataSourceread the asset returnsrd.X.ExcessSourcereads the returns in excess of the benchmark,rd.X - rd.B, the Excess Frame, and refuses returns data with no benchmark.nothingmeans that the caller stated no source, so an optimiser that holds a reader of the Excess Frame warns about a prior atnothing, and not about a prior atDataSource(). A prior reads no prior, so it refusesPriorSource. A prior inside another estimator, such as thepeof aFactorPrior, keepsnothing, and the outer estimator states the source.
Constructors
CrossSectionalFactorPrior(; factors::Dict_VecPair, neutralise::Option{<:Dict_VecPair} = nothing, families::Option{<:Dict_VecPair} = nothing, unseen::AbstractUnseenMemberRule = ZeroUnseenMember(), cre::AbstractCrossSectionalRegressionEstimator = CrossSectionalLinearRegression(), wa::AbstractCrossSectionalWeightsAlgorithm = MarketCapWeights(), pe::AbstractLowOrderPriorEstimator_A_AF = EmpiricalPrior(; me = ExpWeightedExpectedReturns(), ce = RegimeAdjustedExpWeightedCovariance(; centring = PreCentred())), ve::AbstractCovarianceEstimator = RegimeAdjustedExpWeightedVariance(; centring = PreCentred()), ce::StatsBase.CovarianceEstimator = ExpWeightedCovariance(; centring = PreCentred()), pdm::AbstractPosdefEstimator = Posdef(; alg = ClippedNearestCorrelation()), f_mp::AbstractMatrixProcessingEstimator = MatrixProcessing(), mp::AbstractMatrixProcessingEstimator = MatrixProcessing(), srep::Option{<:AbstractSystematicRepair} = nothing, th::Real = 0.0, bp::Real = 1.0, mcap::AbstractString = "market_cap", bw::AbstractString = "benchmark_weights", lag::Integer = 1, minra::Option{<:Integer} = nothing, rfe::Option{<:AbstractReturnForecastEstimator} = nothing, lambda::Num_SpanShrinkCal = PrecisionBlend(), c::Num_OrthFcScaleCal = 1.0, ofit::AbstractOrthogonalForecastFit = ScoreNeutralisation(), l1mask::AbstractLeverageOneMask = MaskedLeverageOne(), lx::Option{<:AbstractString} = nothing, mtx_sqrt::Option{<:AbstractMatrixSquareRootAlgorithm} = EigenFallbackSquareRoot(), ex::FLoops.Transducers.Executor = ThreadedEx(), choice::AbstractChoiceRule = BatchChoice(), carry::AbstractCarryRule = FoldOrRefit(), cache::Option{<:AbstractPartialFitState} = nothing, x_src::Option{<:AbstractReturnsSource} = nothing) -> CrossSectionalFactorPriorKeywords correspond to the struct's fields. factors, neutralise and families also take a dictionary, and the constructor collects each one into a vector of Pairs.
Validation
ceis not anAbstractVarianceEstimator. A variance estimator has nocovorcor, soassert_covariance_estimatorraises anArgumentErrorthat namesce.vereads onlyvar, so it accepts a variance estimator.factorsis not empty and repeats no factor name.- No Factor Family holds both estimated and observed factors, and
familiesconstrains no observed family, asassert_cross_sectional_observed_familiesstates. lx, when it is stated, is not empty, andfactorsholds an observed factor, because the prior reads net returns only under observed factors.neutraliseandfamiliesrepeat no key.thlies in[0, 1].bpis finite and>= 0.mcapandbware not empty.- Every factor whose Exposure Estimator reads benchmark weights names
bwin its ownbw, asassert_cross_sectional_benchmark_fieldstates. lagis> 0.minra, when it is stated, is> 0.- A stated
lambdalies in[0, 1], byassert_spanned_shrinkage. - A stated
cis finite and>= 0, byassert_orthogonal_forecast_scale. - A rule in
lambdaor incthat reads the Return Forecast history needs a fittedrfe, asassert_forecast_history_rulestates. A rule of the wrong family is refused by the bound of the slot. The number a rule returns is checked when the prior resolves it. OrthogonalPartCalibrationinofitneeds a member that can fit the slope of the orthogonal part, asassert_orthogonal_forecast_fitstates.- Under
FoldOnlyincarry, no part makes the cost of a step of the carry fold grow with the stream, asassert_carry_rulestates. - Under
srep = nothing,mpis aMatrixProcessing, asassert_systematic_repairstates. mpstates the estimator with which the lift repairs the asset covariance, seeassert_matrix_processing_posdef.x_srcisnothing,DataSource()orExcessSource(), seeassert_returns_source.- The
x_srcofpeisnothing, because the prior picks the returns that it reads, seeassert_inner_reader.
Propagated parameters
factory passes its arguments on to these @fprop fields:
cre: Recursively updated throughfactory.wa: Recursively updated throughfactory.pe: Recursively updated throughfactory.ve: Recursively updated throughfactory.ce: Recursively updated throughfactory.f_mp: Recursively updated throughfactory.mp: Recursively updated throughfactory.rfe: Recursively updated throughfactory.cache: Carried unchanged throughfactory.
View parameters
port_opt_view takes the view of these @vprop fields at the selected indices:
ve: Recursively viewed throughport_opt_view.ce: Recursively viewed throughport_opt_view.cache: Sliced to the selected assets throughport_opt_view.
Examples
julia> CrossSectionalFactorPrior(; factors = ["mkt" => ConstantExposure()], lag = 2).lag2Related
AbstractLowOrderPriorEstimator_ApriorCrossSectionalFactorModelFactorPriorAssetPanelfactor_exposurecross_sectional_regressionfactor_family_basisneutralise_exposures!AbstractReturnForecastEstimatorCurrencyExposureAbstractObservedExposureEstimatorObservedExposurecross_sectional_local_returnsfactoryport_opt_viewOnlineCrossSectionalCarryStateAbstractChoiceRuleAbstractSystematicRepairAbstractCarryRule
PortfolioOptimisers.prior — Function
prior(pe::CrossSectionalFactorPrior, X::MatNum, F::Option{<:MatNum} = nothing,
pnl::Option{<:AssetPanel} = nothing; dims::Int = 1, ne::Option{<:VecStr} = nothing,
E::Option{<:MatNum} = nothing, iv::Option{<:MatNum} = nothing,
ivpa::Option{<:Num_VecNum} = nothing, ts::Option{<:VecDate} = nothing,
strict::Bool = false, cache_reg::Bool = false, kwargs...) -> LowOrderPriorFit a cross-sectional factor model on an Asset Panel, and return the asset prior it lifts.
This is the returns-matrix method that every prior estimator implements, and the fit runs in it. The panel is the third positional argument, so a wrapping prior composes this estimator when it forwards the panel it received. The method below that takes a ReturnsResult passes its fields to it.
Mathematical definition
\[\begin{align} \boldsymbol{f}_{t} &= \underset{\boldsymbol{f}}{\arg\min} \left(\boldsymbol{x}^{\mathrm{loc}}_{t} - \mathbf{Z}_{t - \ell} \boldsymbol{f}\right)^\intercal \mathbf{Q}_{t} \left(\boldsymbol{x}^{\mathrm{loc}}_{t} - \mathbf{Z}_{t - \ell} \boldsymbol{f}\right)\,, \\ \boldsymbol{\varepsilon}_{t} &= \boldsymbol{x}^{\mathrm{loc}}_{t} - \mathbf{Z}_{t - \ell} \boldsymbol{f}_{t}\,, \\ \boldsymbol{g} &= \underset{\boldsymbol{g}}{\arg\min} \left(\boldsymbol{\alpha}_{T} - \mathbf{Z}_{T} \boldsymbol{g}\right)^\intercal \mathbf{Q}_{T} \left(\boldsymbol{\alpha}_{T} - \mathbf{Z}_{T} \boldsymbol{g}\right)\,, \\ \tilde{\boldsymbol{\mu}}_{f} &= \lambda \hat{\boldsymbol{\mu}}_{f} + (1 - \lambda) \boldsymbol{g}\,, \\ \boldsymbol{\mu}_{\mathcal{S}} &= \mathbf{Z}_{T, \mathcal{S}} \tilde{\boldsymbol{\mu}}_{f} + c \left(\boldsymbol{\alpha}_{T} - \mathbf{Z}_{T} \boldsymbol{g}\right)_{\mathcal{S}}\,, \\ \mathbf{\Sigma}_{\mathcal{S} \mathcal{S}} &= \mathbf{Z}_{T, \mathcal{S}} \hat{\mathbf{\Sigma}}_{f} \mathbf{Z}_{T, \mathcal{S}}^\intercal + \mathbf{D}_{\mathcal{S} \mathcal{S}}\,. \end{align}\]
Where:
- $\boldsymbol{f}_{t}$: Factor returns of observation $t$, $K \times 1$, with entry $f_{tk}$ for factor $k$.
- $\boldsymbol{x}_t$: Asset returns for observation $t$, the $t$-th row of the returns matrix.
- $\boldsymbol{x}^{\mathrm{loc}}_{t}$: Returns the regression reads at observation $t$, $N \times 1$. They are $\boldsymbol{x}_{t}$ without Currency Factors, and the local returns of
cross_sectional_local_returnswith them. - $\mathbf{Z}_{t}$: Factor Exposures of observation $t$, $N \times K$, after the Neutralisation and on the reduced factor axis. $\mathbf{Z}_{T, \mathcal{S}}$ holds the rows of the assets of $\mathcal{S}$. Under Currency Factors the regression reads the estimated exposures alone, and every other line reads $\mathbf{Z}_{t}$ with the currency exposures appended as its last columns, $\boldsymbol{f}_{t}$ with the Currency Excess Returns appended, and $\boldsymbol{g}$ with zeros appended. The shrinkage $\lambda$ then reaches the estimated factors alone, and the expected return of a Currency Factor is the mean the nested factor prior states for it.
- $\ell$: Exposure lag, the count of observations by which the exposures lag the returns.
- $\mathbf{Q}_{t}$: Regression weight matrix of observation $t$, diagonal, holding the cross-sectional weight of each asset. An asset that is not eligible at observation $t$ takes a weight of zero.
- $\boldsymbol{\varepsilon}_{t}$: Idiosyncratic returns of observation $t$, $N \times 1$.
- $\boldsymbol{g}$: Coefficients of the part of the Return Forecast that the latest exposures span, $K \times 1$. It is zero when the prior states no Return Forecast Estimator.
- $\boldsymbol{\alpha}_{t}$: Return Forecast cross-section at observation $t$. The weight of an asset whose forecast or whose exposures are not finite is zero in the fit of $\boldsymbol{g}$.
- $\hat{\boldsymbol{\mu}}_{f}$, $\hat{\mathbf{\Sigma}}_{f}$: Expected factor returns and factor covariance that the nested factor prior states over the factor returns $\boldsymbol{f}_{t}$.
- $\tilde{\boldsymbol{\mu}}_{f}$: Blended expected factor returns, $K \times 1$.
- $\lambda \in [0, 1]$: Spanned Shrinkage of the expected factor returns towards $\boldsymbol{g}$. A rule in the slot gives the number it returns.
- $c \in [0, \infty)$: Orthogonal Forecast Scale, the scale of the part of the Return Forecast that the latest exposures do not span. A rule in the slot gives the number it returns.
- $\boldsymbol{\mu}$: Expected returns vector $N \times 1$.
- $\mathbf{\Sigma}$: Asset covariance matrix, $N \times N$.
- $\mathbf{D}$: Covariance of the idiosyncratic returns that the loadings block carries, $N \times N$. It is diagonal when the block stores a variance vector.
- $\mathcal{S}$: Assets whose latest exposures are finite, whose systematic part the model states. An inactive asset has no finite exposure, so every asset of $\mathcal{S}$ is active at the latest observation.
- $\mathcal{I}$: Investable assets, those that the Asset Panel activates at the latest observation and whose idiosyncratic variance and latest exposures are finite.
- $T$: Number of observations.
- $N$: Number of assets.
- $K$: Number of factors.
The prior states an entry exactly when the model determines it. An asset of $\mathcal{S}$ outside $\mathcal{I}$, for example one in the warm-up of its idiosyncratic variance, has a finite expected return, and under a diagonal $\mathbf{D}$ a finite covariance with each other asset of $\mathcal{S}$, because the model sets the idiosyncratic covariance of two assets to zero. Its variance is NaN, and under a positive th so is each of its covariances, which read its variance. Every entry of $\boldsymbol{\mu}$ and every row and column of $\mathbf{\Sigma}$ outside $\mathcal{S}$ is NaN. The Investable Mask needs a finite mean and a finite variance, so an optimiser still reads $\mathcal{I}$ alone. An Empty Factor, an estimated factor whose exposure is zero at every pair of positive weight, has $f_{tk} = 0$ at every observation, and $\hat{\mathbf{\Sigma}}_{f}$ holds a zero row and column for it. It keeps its place on every factor axis, so a sub-universe with no asset in one level of a one-hot factor still fits. Without a Return Forecast Estimator the forecast terms are zero, so $\boldsymbol{\mu}_{\mathcal{S}} = \lambda \mathbf{Z}_{T, \mathcal{S}} \hat{\boldsymbol{\mu}}_{f}$. At $\lambda = 0$ and $c = 1$ the two parts of the forecast add up to $\boldsymbol{\alpha}_{T}$, so the expected return of an asset of $\mathcal{S}$ with a finite forecast is that forecast.
Algorithm
- Orient
X,FandEbydims, rebuild the returns data that the Descriptors read, fromX,F,ne,E,pnl,ivandivpa, and take the two universe masks offpnlwithcross_sectional_panel_masks. Split the factor list into estimated and observed members withcross_sectional_factor_partition. - Build the benchmark weights
BWwithcross_sectional_cap_weights, over the assets of the estimation universe whose return and market capitalisation are finite, and write them onto a copy of the Asset Panel withcross_sectional_benchmark_returns. The universe and the warm-up readX, or the named net returns underlx, so a gap in an observed series does not move them. A benchmark power of zero reads no market capitalisation. - Read the observed factors from the copy with
cross_sectional_observed, so an observed member that wraps aCompositeExposurereads the benchmark weights. Take the returns the regression reads,Xl, withcross_sectional_local_returns. Build every estimated Factor Exposure withcross_sectional_exposure_history, one dependency layer after another and the members of a layer underex, givingMs,nfandfam. Under observed factors the estimated members readXlin place ofX, so a Descriptor of the returns measures the returns the regression explains. The firstpe.lagrows of the derivedXlhave no lagged exposure, so the derivation takes the exposure of the same observation there. An observed member cannot readXl, becauseXlis derived from its exposures. So the observed members that can read returns readXnet of the observed members that read none, ascross_sectional_observedstates. - Drop the leading observations the Descriptors warm up over, with
cross_sectional_warmupon the estimated and the observed exposures together, giving the rowsrw.cross_sectional_exposure_stageruns steps 1 to 4. - Neutralise the exposures with
neutralise_exposures!, under the benchmark weights and the prior's own regression estimator. - Build the Factor Family Basis
fbwithcross_sectional_family_basis, and reduce the exposures through it. - Lag the reduced exposures and the market capitalisation by
pe.lag, givingZlandmcl. Trim the observed factors to the fitted observations withcross_sectional_observed_block, which refuses an infinite observed return among them and keeps aNaNas a gap. Take the eligibility maskmskof the fit withcross_sectional_eligibleonXl, and drop from it every pair whose lagged market capitalisation is not finite. - Take the design of the regression under the Unseen Member rule
unseenwithunseen_member_design. Regress each observation'sXlon the design it gives withcross_sectional_live_regression, and map the coefficients back withunseen_member_returns, givingcsrand the masklvof the factors that are not empty, under the weightsWofcs_weights_initial. Refuse acsrthat carries an intercept withassert_cross_sectional_no_intercept. Whenneeds_second_passanswerstrue, refine the weights withcs_weights_refineand regress again. The refinement reads the idiosyncratic variance under the two universe masks, as step 9 does. - Take the idiosyncratic variance history
vswithvariance_series, under the active mask and the estimation mask of the fitted observations. An estimator that reads the masks resets an asset that the active mask turns off, and measures its regime over the estimation universe alone. An estimator that reads no mask ignores them. Standardise the idiosyncratic returns byvswithcross_sectional_standardised_residuals, givingS, and take the latest idiosyncratic covarianceesigmawithcross_sectional_idiosyncratic_covariance, from the same residuals with no fill, so a gap stays a gap for the correlation. Record the degrees of freedom and the divisor of each variance withvariance_countandcross_sectional_variance_counts. - Append the observed factors after the estimated ones with
cross_sectional_observed_append: the observed returns after the factor returns, the observed exposures after the loadings and the exposure history, the names and the family labels, and pass-through factors on the Factor Family Basis. Fitpe.peon the combined reduced factor returns of the factors that are not empty withcross_sectional_factor_moments, givingf_pr, which refuses a non-finite factor moment, divides the scenario weights by their sum, and processes the factor covariance underpe.f_mp, the matrix processing estimator of the factor axis and not the asset one. An Observed Factor is never empty. The method passesstricttope.pe, asFactorPriordoes, because the slot admitsBlackLittermanPriorandEntropyPoolingPrior, which resolve view names against a universe.cross_sectional_assembleruns the standardisation and the counts of step 9 and the steps 11 to 17, and the call with no data of the carry fold runs it too. - Build the
CrossSectionalFactorModelblockcsfm, with the raw exposures of the latest observation inM, the reduced onesLbeside them, a zerob, and the observed returns infx. Record the rows ofXthat the fit covers,rw[r], inidx, and their timestamps intswhentsis given. Under a family re-basis, expand the combined factor returns onto the raw axis withcross_sectional_expand, each row with the basis of its lagged exposures, and store them infr. - Fit the Return Forecast with
cross_sectional_return_forecast, on the full returns data that the estimated members read, so that a Descriptor of the forecast warms up over every observation the panel has, giving the spanned coefficientsgand the unscaled orthogonal partap. Under observed factors the forecast thus readsXl, and it forecasts the net return of each asset, which the split measures against loadings of the same returns. Resolvelambdaandcwithcross_sectional_calibration, and writeb = c ap, the Result and the two numbers onto the blockrrwithcross_sectional_forecast_block. Blend the spanned part into the mean of the estimated factors withcross_sectional_forecast_mu, and keep the mean of the observed factors, givingf_mu. - Expand the blended factor moments onto the raw factor axis with
cross_sectional_expand, sofprstates the distribution of the factors the caller named. Its factor returns arefr, or the combined factor returns when no family is constrained. - Take the investable assets
idxwithcross_sectional_investable. - Rebuild the asset return scenarios
Xswithcross_sectional_scenarios. - Lift the reduced factor distribution onto the assets with
cross_sectional_lift, and addbto the expected return it answers. The lift states every entry the model determines: the investable block, and the mean and the systematic covariances of an asset with finite exposures and no idiosyncratic variance. - Assemble a
LowOrderPrioroverXs, with the fitted base-currency returns undero_X, the three lifted moments, the factor prior'swdivided by its sum, itsens,kldandow, the block underrr, and the expanded factor prior underfpr. The factor returns offprand the returns ofo_Xkeep the last rows alone, as many asXsholds, so a factor prior with a Scenario Cap pairs each scenario with its own observation.
Arguments
pe: Cross-Sectional Factor Prior estimator.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.F: Data matrixobservations × factorsif thedimskeyword does not exist ordims = 1,factors × observationswhendims = 2. The fit does not read it, because this estimator builds its own factors from the panel. The method writes it onto the rebuilt returns data only so that the returns data states what the caller held.pnl: OptionalAssetPanel, the panel that the returns data held. A wrapping prior forwards it unchanged, so that it can compose an estimator that is fitted on a panel. An estimator that reads no panel ignores it. This estimator reads it, and refuses without it.dims: Dimension along which the observations lie.ne: Names of the Exogenous Series, written onto the rebuilt returns data.E: The Exogenous Series, oriented bydimsasXis. The observed factors read their returns from it.iv: Implied volatilities, written onto the rebuilt returns data.ivpa: Implied-volatility risk-premium adjustment, written onto the rebuilt returns data.ts: Timestamp vector. It has one entry for each observation ofX, and the block records the entries of the rows it fits, ornothing.strict: Whether to throw an error ifsetsdoes not contain the desired value insets.dict[key]. It is forwarded to the nested factor priorpe.pe.cache_reg: Iftrue, a prior stores on its result the cache that its refit under new scenario weights reads: aCrossSectionalFactorPriorthe active mask and the Return Forecast history on its block, and anOpinionPoolingPriorthe rows ofpe1, its pooled weights and the cache ofpe2inpc. The refit of an entropy pooling prior around it then runs no regression and no opinion. Every other prior forwards it to the prior that it wraps. Default isfalse.kwargs...: Additional keyword arguments passed to the verbs of the algorithm.
Validation
pnlis notnothing. Raises anIsNothingError.- If provided,
length(ts)equals the number of observations ofX. Raises aDimensionMismatch. - The Asset Panel is time-varying. Raises an
ArgumentError. - The Asset Panel holds no Panel Field named
pe.bw, because the fit writes its benchmark weights onto that name. Raises anArgumentError. - The history is longer than the exposure lag. Raises an
ArgumentError. - At least two observations are left after the Descriptor warm-up and the exposure lag, because a covariance of one observation is not a number. Raises an
ArgumentError. - Every fitted observation carries at least
minraeligible assets. Raises anArgumentError. - Under observed factors,
Ecarries a column for every observed series, and no observed return of the fitted observations is infinite. Raises anIsNothingError, anArgumentErroror anIsNonFiniteError, fromcross_sectional_observedandcross_sectional_observed_block. - At least one estimated factor is not empty. Raises an
ArgumentError. - The regression fits no intercept. The prior states the moments through the factor returns alone, so an intercept would leave its mean out of
muand its variance out ofsigma. Raises anArgumentError. - The factor prior determines the factor moments of at least one asset: the loadings of the asset read only factors with a finite mean and a finite variance. Raises an
IsNonFiniteError, fromassert_cross_sectional_factor_moments. - The scenario weights of the factor prior, when it carries any, sum to a positive number. Raises a
DomainError, fromcross_sectional_scenario_weights. - At least one asset is investable at the latest observation. Raises an
IsEmptyError. - The rules of every verb the algorithm names.
Returns
pr::LowOrderPrior: The prior on the full asset universe. An entry that the model does not determine isNaN: the whole entry ofmuand row and column ofsigmaof an asset with no finite latest exposure, each entry that reads a factor moment the factor prior does not state, and the variance of an asset with no idiosyncratic variance, with its covariances under a positiveth. An asset that loads on a factor whose mean or variance the factor prior does not state leaves the Investable Mask, and every other asset keeps its moments.rris aCrossSectionalFactorModel, andfpris the factor prior on the raw factor axis.wholds the scenario weights of the factor prior divided by their sum, so they sum to one, andfprholds the samew. It isnothingwhen the factor prior carries no weights.
Related
PortfolioOptimisers.prior — Method
prior(pe::CrossSectionalFactorPrior, rd::ReturnsResult; strict::Bool = false,
dims::Int = 1, cache_reg::Bool = false) -> LowOrderPriorFit a Cross-Sectional Factor Prior from a ReturnsResult.
The method passes the fields of rd to the returns-matrix method above, which runs the fit. The file defines this method instead of using prior(pe::AbstractPriorEstimator, rd::ReturnsResult), so that the refusal of an rd with no Asset Panel names rd.pnl, the field of rd that the caller built.
Algorithm
- Check that
rdcarries asset returns. - Drop the observations of a gap of a vector benchmark under
pe.x_src = ExcessSource()withshared_series_view. The estimator does not read the factor returns, so a gap in them drops nothing. - Read the asset returns with
benchmark_relative_returnsunderpe.x_src, givingX. - Call the returns-matrix method with
X,rd.Fandrd.pnl, forwardingrd.ne,rd.E,rd.iv,rd.ivpa,rd.ts,strictandcache_regas keyword arguments, anddims = 1, because aReturnsResultholds its observations along the rows. - Record the observations that step 2 dropped with
shared_series_record, andpe.x_srcwithstamp_returns_source.
Arguments
pe: Cross-Sectional Factor Prior estimator.rd: The returns result to use. It must carry asset returns inrd.X, and a time-varying Asset Panel inrd.pnlwhen an exposure reads a Panel Field. With no panel, the fit reads the mask-only panel ofreturns_with_universe.strict: Passed to the returns-matrix method of the prior, where it decides if a scenario fill or a coverage peel raises in place of a warning.dims: Accepted and ignored. AReturnsResultholds its observations along the rows, so the call passesdims = 1. The method takes no other keyword that it does not list: an unknown keyword raises theMethodErrorof the language, and no estimator swallows it.cache_reg: Iftrue, a prior stores on its result the cache that its refit under new scenario weights reads: aCrossSectionalFactorPriorthe active mask and the Return Forecast history on its block, and anOpinionPoolingPriorthe rows ofpe1, its pooled weights and the cache ofpe2inpc. The refit of an entropy pooling prior around it then runs no regression and no opinion. Every other prior forwards it to the prior that it wraps. Default isfalse.
Validation
rd.Xis notnothing. Raises anIsNothingError.- The rules of
benchmark_relative_returns. - The rules of the returns-matrix method.
Returns
pr::LowOrderPrior: The prior the returns-matrix method fitted.
Related
PortfolioOptimisers.AbstractSpannedShrinkageCalibrationAlgorithm — Type
abstract type AbstractSpannedShrinkageCalibrationAlgorithm <: AbstractCalibrationAlgorithmComputes the Spanned Shrinkage, the lambda of a Cross-Sectional Factor Prior, from the fit.
The Spanned Shrinkage blends the expected factor returns of the nested factor prior with the spanned coefficients g of the Return Forecast, and it lies in [0, 1]. A rule of this family stands in the lambda slot of CrossSectionalFactorPrior, and Num_SpanShrinkCal is the bound of that slot, so a rule of another family is refused at construction.
A rule is called as alg(key, pr, w, slv, ctx), as resolve_calibration_slot states, and returns the shrinkage. The prior refuses a returned number outside [0, 1].
Interfaces
In order to implement a new concrete type that works seamlessly with the library, subtype AbstractSpannedShrinkageCalibrationAlgorithm and implement the following methods:
The functor
(alg::AbstractSpannedShrinkageCalibrationAlgorithm)(key::Symbol, pr::AbstractPriorResult, w, slv, ctx::CalibrationContext) -> Number: Returns the Spanned Shrinkage.
Arguments
key: Name of the slot that is being resolved,:lambda.pr: The factor moments of the nested factor prior, on the reduced factor axis of the estimated factors and the observed factors.w: The observation weights ofpr, ornothing.slv:nothing.ctx: ACalibrationContextwhosecscarries the fit of the prior.
Returns
lambda::Number: The Spanned Shrinkage, in[0, 1].
reads_forecast_history
reads_forecast_history(alg::MySpannedShrinkage) -> Bool: Returnstruewhen the rule readsctx.cs.hist. Add it only for a rule that reads the history, because the default answersfalse.
Related
AbstractCalibrationAlgorithmAbstractOrthogonalForecastScaleCalibrationAlgorithmNum_SpanShrinkCalCrossSectionalFactorPriorcross_sectional_calibrationPrecisionBlend: the default rule of the slot.SteinShrinkage: the family's rule that reads the intensity of a shrunk expected-returns estimator.
PortfolioOptimisers.AbstractOrthogonalForecastScaleCalibrationAlgorithm — Type
abstract type AbstractOrthogonalForecastScaleCalibrationAlgorithm <: AbstractCalibrationAlgorithmComputes the Orthogonal Forecast Scale, the c of a Cross-Sectional Factor Prior, from the fit.
The Orthogonal Forecast Scale multiplies the part of the Return Forecast that the latest Factor Exposures do not span, and it lies in [0, ∞). A rule of this family stands in the c slot of CrossSectionalFactorPrior, and Num_OrthFcScaleCal is the bound of that slot, so a rule of another family is refused at construction.
A rule is called on the same terms as a rule of AbstractSpannedShrinkageCalibrationAlgorithm, and returns the scale. The prior refuses a returned number that is negative or not finite.
Interfaces
In order to implement a new concrete type that works seamlessly with the library, subtype AbstractOrthogonalForecastScaleCalibrationAlgorithm and implement the following methods:
The functor
(alg::AbstractOrthogonalForecastScaleCalibrationAlgorithm)(key::Symbol, pr::AbstractPriorResult, w, slv, ctx::CalibrationContext) -> Number: Returns the Orthogonal Forecast Scale.
Arguments
key: Name of the slot that is being resolved,:c.pr: The factor moments of the nested factor prior, on the reduced factor axis of the estimated factors and the observed factors.w: The observation weights ofpr, ornothing.slv:nothing.ctx: ACalibrationContextwhosecscarries the fit of the prior.
Returns
c::Number: The Orthogonal Forecast Scale, finite and>= 0.
reads_forecast_history
reads_forecast_history(alg::MyOrthogonalForecastScale) -> Bool: Returnstruewhen the rule readsctx.cs.hist. Add it only for a rule that reads the history, because the default answersfalse.
Related
AbstractCalibrationAlgorithmAbstractSpannedShrinkageCalibrationAlgorithmNum_OrthFcScaleCalCrossSectionalFactorPriorcross_sectional_calibrationForecastCalibrationSlope: the family's rule that reads the calibration slope of the orthogonal forecast.
PortfolioOptimisers.reads_forecast_history — Function
reads_forecast_history(rule) -> BoolAnswer whether the occupant of a slot of a Cross-Sectional Factor Prior reads the Return Forecast history.
A member that computes no history fits the Return Forecast again at each observation to make one, so the prior makes the history only when a slot asks for it, and it makes it once for both slots. forecast_history makes it, and the CalibrationContext carries it in cs.hist. The default answers false, so a stated number and a plain function never get the history. A rule type that reads it adds a method that answers true.
Arguments
rule: The occupant of the slot.
Returns
flag::Bool: Whether the occupant reads the Return Forecast history.
Examples
julia> PortfolioOptimisers.reads_forecast_history(0.5)falseRelated