Cross-Sectional Factor Prior (a)

A cross-sectional factor prior fits a factor model at each observation from the exposures in an AssetPanel, and turns the model into moments of the assets. It returns a LowOrderPrior whose rr field holds the fitted CrossSectionalFactorModel.

PortfolioOptimisers.CrossSectionalFactorPrior — Type
struct CrossSectionalFactorPrior{__T_factors, __T_neutralise, __T_families, __T_unseen, __T_cre, __T_wa, __T_pe, __T_ve, __T_ce, __T_pdm, __T_f_mp, __T_mp, __T_srep, __T_th, __T_bp, __T_mcap, __T_bw, __T_lag, __T_minra, __T_rfe, __T_lambda, __T_c, __T_ofit, __T_l1mask, __T_lx, __T_mtx_sqrt, __T_ex, __T_choice, __T_carry, __T_cache, __T_x_src} <: AbstractLowOrderPriorEstimator_A

Estimates a point-in-time cross-sectional factor model from an Asset Panel, and lifts it onto the assets.

The estimator reads per-asset Panel Fields, builds a Factor Exposure from each one, regresses the returns of every observation on the lagged exposures across the assets, and returns the asset moments beside a CrossSectionalFactorModel block. It is the cross-sectional counterpart of FactorPrior, which regresses the returns of each asset on a factor-return series over time.

The factor list can also hold observed factors, whose returns the caller observes and the fit does not estimate. A member of AbstractObservedExposureEstimator declares them: CurrencyExposure gives one Currency Factor per currency, and ObservedExposure gives one factor, such as a macro series or an observed market return. Each reads its return from the Exogenous Series rd.E by name. A member that gives no factor on the data, such as a CurrencyExposure on a universe of its base currency alone, reads no series, and the fit is the fit without it. The regression reads the returns net of the observed factors: by default the fit derives them as X - Z_obs * r_obs, and lx can name a Panel Field of net returns the caller measured. cross_sectional_local_returns states both paths. Under Currency Factors the net returns are the local returns, and rd.X stays in the base currency. The observed factors then join the factor model after the estimated factors, whatever their place in the list, so the factor covariance, mu, the scenarios, the factor attribution and the exposure constraints see them, and the Neutralisation and the Factor Family Basis pass them through. An observed series can start after the first fitted observation. Its NaN rows drop the assets that hold the factor from the regression at those observations, and the factor prior pe states the moments of the factor by its own rule: the default, mask-aware, estimates them from the observations the series has, and EmpiricalPrior() leaves them NaN, so the assets that hold the factor leave the Investable Mask.

The warm-ups of a fit add up, so a caller who sizes a window must sum three of them rather than take the longest. The longest warm-up of the Descriptors sets the first observation of the factor-return history, and lag adds lag observations to it. pe then warms up over that history, and ve over the idiosyncratic returns beside it. The default pe and the default ve each need 40 observations, their min_obs, so a default fit needs 40 observations after the Descriptor warm-up and the lag. A window that covers the Descriptors alone can leave pe too few observations to state a factor covariance, and the fit then refuses with a message that names the cause. Cross-validation meets this most often. A fold gives the estimator its own rows alone, so the Descriptors warm up again in every fold, and a rolling train window never grows past the warm-up. Size the train window against the sum of the three warm-ups.

On the online step the prior takes one of two routes. Unwrapped, it folds as a carry: its first step of partial_fit! seeds a CrossSectionalCarryState, and each step computes the exposures, the regression and the idiosyncratic variance of the new observations alone. Wrapped in Online, it refits: each step records the returns, both masks and the Panel Fields in a sample buffer, the call with no data prior(pe) is the batch fit over the rows of the buffer, and a max_history on the wrapper bounds them. When a member reads the Exogenous Series, as reads_exogenous_series answers, the buffer records every column of the series too, so an observed factor takes the refit. The carry fold records the series of the rows it carries, so an observed factor takes the carry too. The automatic choice of a dropped member reads every row, so choice states whether each fit chooses again or the first fit pins it.

The Carry Rule carry states what the carry fold does with a part that has no fold of its own. Under FoldOrRefit, the default, the carry fits such a part again over every row that it reads. The call with no data stays equal to the batch fit, but the cost of the refit grows with the stream:

  • A factor or a Return Forecast with no finite lookback makes the carry keep every panel row, and each step computes the Factor Exposures over all of them.
  • A Return Forecast with no fold of its rows fits again over every row at each call with no data.
  • A ve that does not fold, as supports_partial_fit answers, fits every idiosyncratic return again at each step.
  • A factor prior pe that does not fold, as carry_folds answers, fits every factor return again at each call with no data.
  • The idiosyncratic correlation under th > 0 with a ce that does not fold is estimated again over every standardised idiosyncratic return at each call with no data. A ce that folds, as the default ExpWeightedCovariance does, folds the rows of each step.

FoldOnly makes the constructor refuse each of these parts, and names it. A batch fit and the refit under Online ignore the rule.

A Scenario Cap is a route through pe. The prior builds its asset scenarios from the last rows of the factor scenarios and of the standardised idiosyncratic returns, as cross_sectional_scenarios states. So pe = EmpiricalPrior(; …, max_scenarios = w) gives the last w asset scenarios, and mu and sigma stay fitted over every observation. The max_history of Online is a different knob: it is a window on the whole fit, so the moments move with it.

ofit is the Orthogonal Forecast Fit. A fitted Return Forecast regresses the forward idiosyncratic return, which the regression makes orthogonal to the Factor Exposures, so the part of its forecast that the exposures do not span is under-scaled. The default ScoreNeutralisation neutralises the scores of such a member against every estimated factor of the block before the fit, OrthogonalPartCalibration scales the orthogonal part by a slope fitted on it, and UnadjustedForecast reads the member as it stands.

l1mask is the Leverage-One Mask. The residual of a Leverage-One Pair is zero by construction, and the default MaskedLeverageOne reads it as a gap in the standardised idiosyncratic returns, as the forecast fit, the idiosyncratic diagnostics and the attribution do. UnmaskedLeverageOne standardises it as it stands.

lambda and c are Calibration Slots. Each holds a number, or a rule that computes the number from the fit: lambda takes a rule of AbstractSpannedShrinkageCalibrationAlgorithm, and c takes a rule of AbstractOrthogonalForecastScaleCalibrationAlgorithm. Each fit, each refit and each online step resolves the rules with cross_sectional_calibration, after the split of the Return Forecast and before the mean. The block records the resolved numbers in its fields lambda and c.

Fields

  • factors: Pairs of factor name => Exposure Estimator. The estimated factors take the factor axis in the order they are written, and the observed factors follow them in the order they are written. A one-hot member contributes one factor per level of its categorical Panel Field, and so does a CurrencyExposure, so a Pair is not always one factor. The list holds at least one estimated member, because the cross-sectional regression fits the estimated factors, and the constructor refuses a list of observed members alone.
  • neutralise: Neutralisation, as Pairs of key => targets run in order, or nothing. A key names a factor or a Factor Family, and so does each target.
  • families: Constrained Factor Families, as Pairs of family label => dropped member, or nothing. When the dropped member is nothing, factor_family_basis chooses it. unseen states the return of a member that no asset of an observation loads on.
  • unseen: Unseen Member rule of the constrained Factor Families, a member of AbstractUnseenMemberRule. An Unseen Member is a member that no asset of positive regression weight loads on at one observation. Under ZeroUnseenMember, the default, it has a return of zero there, and the zero-sum condition of the observation holds over the other members. Under SolvedUnseenMember it stays in the condition, and the solve algorithm of cre gives its return, which then depends on the dropped member.
  • cre: Cross-Sectional Regression Estimator of the fit, and of the Neutralisation. It must fit no intercept, because the prior states the moments through the factor returns alone, and the fit refuses an intercept. A "market" => ConstantExposure() factor states the common return instead.
  • wa: Weight policy of the cross-sectional fit. Its p is the power the regression weights raise the market capitalisation to.
  • pe: Prior estimator. The fit gives it the reduced factor-return series, so a constrained Factor Family gives it a full-rank covariance. The default is an EmpiricalPrior of an ExpWeightedExpectedReturns and a RegimeAdjustedExpWeightedCovariance with centring = PreCentred(). Its half-life is the 40 observations of the default ve, so the systematic part and the specific part of sigma answer on one horizon, and after a change of regime both of them move. The factor covariance is centred, which is the convention of ce and of ve.
  • ve: Variance estimator. variance_series on it gives the idiosyncratic variance history, whose last row is the idiosyncratic risk of the latest observation. The default is a RegimeAdjustedExpWeightedVariance with centring = PreCentred(), which measures the second moment of each idiosyncratic series about zero. The prior states the mean of an asset through the factors alone, so the model sets the mean of the idiosyncratic return to zero, and the specific risk is its second moment. A variance about a running mean would understate it, and would also move the regime multiplier. A PortfolioOptimisersCovariance with a step in its mp, as its default mp holds, raises on this history, as its docstring states.
  • ce: Covariance estimator. It estimates the covariance of the standardised idiosyncratic returns, and the fit reads it only when th is positive. gap_fill_value on it gives the value of the cell of an inactive asset. The default gives NaN, so the fit passes the gap and the active mask of the panel to the estimator. A plain moment estimator gives zero, and the fit writes zero into the cell.
  • pdm: Positive definite matrix estimator. It repairs the idiosyncratic covariance alone, after the threshold, over the assets with a finite variance, and the fit reads it only when th is positive. mp.pdm repairs a different matrix: the asset covariance, which is the sum of the systematic block and this one. The default clips the correlation eigenvalues with ClippedNearestCorrelation, because the carry fold runs this repair at each call with no data, and the clip costs less than the nearest correlation matrix. pdm = Posdef() gives the nearest correlation matrix by NearestCorrelationMatrix.Newton.
  • f_mp: Factor matrix processing estimator. It processes the factor covariance that pe states, which is a different matrix from the asset covariance that mp processes. The factor covariance is on the factor axis, pe estimates it from the factor-return series, and a Factor Family that drops a member can leave it singular. cross_sectional_lift takes its square root under mtx_sqrt for the low-rank square root, so under the default mtx_sqrt a factor covariance that is not positive definite fails there rather than in the asset block.
  • mp: Matrix processing estimator.
  • srep: Systematic Repair rule of the lift, a member of AbstractSystematicRepair, or nothing. nothing runs every step of mp on the systematic block except the :pdm step, and SystematicRepair runs every step. Under each rule the lift then adds the idiosyncratic block and repairs the sum under the estimator that matrix_processing_posdef gives for mp. The default is nothing. The batch fit and the carry fold run the same lift when they read the prior, so the carry fold equals the batch fit under each rule.
  • th: Idiosyncratic correlation threshold. A value of zero leaves the idiosyncratic covariance diagonal, and a positive value keeps every correlation above it and zeroes the rest, so the block becomes a matrix.
  • bp: Power the benchmark weights raise the market capitalisation to. A value of zero gives every asset of the estimation universe the same benchmark weight, and reads no market capitalisation.
  • mcap: Name of the numeric Panel Field holding the market capitalisation.
  • bw: Name of the numeric Panel Field the prior writes its benchmark weights onto, and the one every Exposure Estimator reads them from. The Asset Panel must not already hold a field of this name.
  • lag: Number of observations by which the exposures lag the returns.
  • minra: Smallest eligible asset count an observation may carry, or nothing for max(2K, 30) over the reduced factor count K.
  • rfe: Return Forecast Estimator whose forecast enters mu, or nothing. It is fitted on the coverage universe, after the factor model, and its forecast is split against the latest Factor Exposures into the part they span and the part they do not.
  • lambda: Spanned Shrinkage of the expected factor returns towards the spanned part of the Return Forecast, in [0, 1], or a rule that computes it. A value of one keeps the fitted factor mean, and a value of zero takes the spanned forecast alone. With no Return Forecast Estimator the spanned part is zero, so the value shrinks the factor mean towards zero. The bound is Num_SpanShrinkCal. The default is PrecisionBlend, which weighs the factor mean and the spanned part by the error of each.
  • c: Orthogonal Forecast Scale of the Return Forecast, in [0, ∞), or a rule that computes it. It scales the part of the forecast the factors do not span, which the block carries in b. A value of zero discards it where it is finite, and a value above one scales it up. The bound is Num_OrthFcScaleCal. An asset whose forecast is not finite carries NaN in b whatever c is, because 0 * NaN is NaN, so the prior states no expected return for it and it leaves the Investable Mask. A forecast that is not finite at every asset splits into two zero parts, and changes no moment.
  • ofit: Orthogonal Forecast Fit, the rule that makes the orthogonal part of a fitted Return Forecast carry the scale of its fit. ScoreNeutralisation, the default, neutralises the scores of a member that answers true to fits_idiosyncratic_target against every estimated factor of the block, under the regression weights of the block, before the fit. OrthogonalPartCalibration fits the calibration slope of TargetReturnForecast on the orthogonal part of its prediction too, and scales the orthogonal part by the ratio of the two slopes. UnadjustedForecast reads the member as it stands. The rule acts before c, which multiplies the part that it gives.
  • l1mask: Leverage-One Mask, the rule by which the prior reads the residual of a Leverage-One Pair when it standardises the idiosyncratic returns. MaskedLeverageOne, the default, reads it as a gap, so the scenarios give the pair the average standardised return of its observation and the idiosyncratic correlation leaves it out. UnmaskedLeverageOne reads the zero residual as it stands: zero over a positive variance, and the average over a zero variance.
  • lx: Name of a numeric Panel Field of the returns net of the observed factors, or nothing to derive them as X - Z_obs * r_obs. The regression reads the named field unchanged. Under Currency Factors the net returns are the local returns. A caller names a field when some asset has no currency label, which the derived path gives a NaN net return, or when their own local returns are not X - Z_obs * r_obs, which includes every panel of simple returns because of the cross term, see currency_excess_index.
  • mtx_sqrt: Square-root algorithm of the factor covariance, and of the idiosyncratic covariance when that block is a full matrix, whose square roots make up chol. The default EigenFallbackSquareRoot takes the Cholesky factor where it exists, and the eigen square root of a singular positive semidefinite matrix otherwise. RidgeCholeskySquareRoot takes the Cholesky factor of the matrix with a small ridge added, and nothing takes the plain Cholesky factor, which raises a LinearAlgebra.PosDefException on a matrix that is not positive definite. matrix_square_root states each algorithm.
  • ex: Parallel execution strategy. It computes the Factor Exposures of one dependency layer, whose members read no exposure of each other. Each member writes its own columns of the exposure history, so every executor gives the same prior. The regression runs under the executor of cre, and a Return Forecast under the executor of its own Descriptor Scores.
  • choice: Choice Rule of the dropped member of each Factor Family whose member is nothing in families. Under BatchChoice, the default, each fit chooses again over every observation. Under PinnedChoice, the online step writes the choice of the first fit into families of the estimator that it returns. The two agree in a batch fit. On the carry fold, a step that moves the member under BatchChoice solves no observation of full rank again: it selects the columns of the raw factor returns that the new member keeps, and folds the factor prior again over every carried factor return, as cross_sectional_fold_move states. So FoldOnly refuses BatchChoice with an automatic member.
  • carry: Carry Rule of the carry fold, a member of AbstractCarryRule. FoldOrRefit, the default, folds each part that has a fold and fits every other part again at each step, so the carry fold equals the batch fit at a cost that can grow with the stream. FoldOnly makes the constructor refuse a part whose step cost grows, as carry_growing_parts lists it. A batch fit and the refit under Online ignore the rule.
  • cache: Optional partial-fit state. It is nothing until partial_fit! writes one. When the caller gives no data matrix, the batch verb of the estimator reads it. Each propagation channel does one thing with it. factory keeps it unchanged, because a factory call resolves the configuration, not the sample. port_opt_view copies the entries of the selected assets into a new state, so the viewed estimator holds the state of those assets alone. obs_weights_view removes it, because a state has no slice on the observation axis. Only a type with an observation-weights field has that channel. A type with no such field falls through to the identity method, so it keeps its state. A family whose state has no exact asset slice removes it on both axes and names the reason.
  • x_src: The source of the returns that the prior fits on, nothing or an AbstractReturnsSource. nothing and DataSource read the asset returns rd.X. ExcessSource reads the returns in excess of the benchmark, rd.X - rd.B, the Excess Frame, and refuses returns data with no benchmark. nothing means that the caller stated no source, so an optimiser that holds a reader of the Excess Frame warns about a prior at nothing, and not about a prior at DataSource(). A prior reads no prior, so it refuses PriorSource. A prior inside another estimator, such as the pe of a FactorPrior, keeps nothing, and the outer estimator states the source.

Constructors

CrossSectionalFactorPrior(; factors::Dict_VecPair,                          neutralise::Option{<:Dict_VecPair} = nothing,                          families::Option{<:Dict_VecPair} = nothing,                          unseen::AbstractUnseenMemberRule = ZeroUnseenMember(),                          cre::AbstractCrossSectionalRegressionEstimator = CrossSectionalLinearRegression(),                          wa::AbstractCrossSectionalWeightsAlgorithm = MarketCapWeights(),                          pe::AbstractLowOrderPriorEstimator_A_AF = EmpiricalPrior(; me = ExpWeightedExpectedReturns(), ce = RegimeAdjustedExpWeightedCovariance(; centring = PreCentred())),                          ve::AbstractCovarianceEstimator = RegimeAdjustedExpWeightedVariance(; centring = PreCentred()),                          ce::StatsBase.CovarianceEstimator = ExpWeightedCovariance(; centring = PreCentred()),                          pdm::AbstractPosdefEstimator = Posdef(; alg = ClippedNearestCorrelation()),                          f_mp::AbstractMatrixProcessingEstimator = MatrixProcessing(),                          mp::AbstractMatrixProcessingEstimator = MatrixProcessing(),                          srep::Option{<:AbstractSystematicRepair} = nothing,                          th::Real = 0.0, bp::Real = 1.0,                          mcap::AbstractString = "market_cap",                          bw::AbstractString = "benchmark_weights", lag::Integer = 1,                          minra::Option{<:Integer} = nothing,                          rfe::Option{<:AbstractReturnForecastEstimator} = nothing,                          lambda::Num_SpanShrinkCal = PrecisionBlend(),                          c::Num_OrthFcScaleCal = 1.0,                          ofit::AbstractOrthogonalForecastFit = ScoreNeutralisation(),                          l1mask::AbstractLeverageOneMask = MaskedLeverageOne(),                          lx::Option{<:AbstractString} = nothing,                          mtx_sqrt::Option{<:AbstractMatrixSquareRootAlgorithm} = EigenFallbackSquareRoot(),                          ex::FLoops.Transducers.Executor = ThreadedEx(),                          choice::AbstractChoiceRule = BatchChoice(),                          carry::AbstractCarryRule = FoldOrRefit(),                          cache::Option{<:AbstractPartialFitState} = nothing, x_src::Option{<:AbstractReturnsSource} = nothing) -> CrossSectionalFactorPrior

Keywords correspond to the struct's fields. factors, neutralise and families also take a dictionary, and the constructor collects each one into a vector of Pairs.

Validation

Propagated parameters

factory passes its arguments on to these @fprop fields:

  • cre: Recursively updated through factory.
  • wa: Recursively updated through factory.
  • pe: Recursively updated through factory.
  • ve: Recursively updated through factory.
  • ce: Recursively updated through factory.
  • f_mp: Recursively updated through factory.
  • mp: Recursively updated through factory.
  • rfe: Recursively updated through factory.
  • cache: Carried unchanged through factory.

View parameters

port_opt_view takes the view of these @vprop fields at the selected indices:

Examples

julia> CrossSectionalFactorPrior(; factors = ["mkt" => ConstantExposure()], lag = 2).lag2

Related

source
PortfolioOptimisers.prior — Function
prior(pe::CrossSectionalFactorPrior, X::MatNum, F::Option{<:MatNum} = nothing,
      pnl::Option{<:AssetPanel} = nothing; dims::Int = 1, ne::Option{<:VecStr} = nothing,
      E::Option{<:MatNum} = nothing, iv::Option{<:MatNum} = nothing,
      ivpa::Option{<:Num_VecNum} = nothing, ts::Option{<:VecDate} = nothing,
      strict::Bool = false, cache_reg::Bool = false, kwargs...) -> LowOrderPrior

Fit a cross-sectional factor model on an Asset Panel, and return the asset prior it lifts.

This is the returns-matrix method that every prior estimator implements, and the fit runs in it. The panel is the third positional argument, so a wrapping prior composes this estimator when it forwards the panel it received. The method below that takes a ReturnsResult passes its fields to it.

Mathematical definition

\[\begin{align} \boldsymbol{f}_{t} &= \underset{\boldsymbol{f}}{\arg\min} \left(\boldsymbol{x}^{\mathrm{loc}}_{t} - \mathbf{Z}_{t - \ell} \boldsymbol{f}\right)^\intercal \mathbf{Q}_{t} \left(\boldsymbol{x}^{\mathrm{loc}}_{t} - \mathbf{Z}_{t - \ell} \boldsymbol{f}\right)\,, \\ \boldsymbol{\varepsilon}_{t} &= \boldsymbol{x}^{\mathrm{loc}}_{t} - \mathbf{Z}_{t - \ell} \boldsymbol{f}_{t}\,, \\ \boldsymbol{g} &= \underset{\boldsymbol{g}}{\arg\min} \left(\boldsymbol{\alpha}_{T} - \mathbf{Z}_{T} \boldsymbol{g}\right)^\intercal \mathbf{Q}_{T} \left(\boldsymbol{\alpha}_{T} - \mathbf{Z}_{T} \boldsymbol{g}\right)\,, \\ \tilde{\boldsymbol{\mu}}_{f} &= \lambda \hat{\boldsymbol{\mu}}_{f} + (1 - \lambda) \boldsymbol{g}\,, \\ \boldsymbol{\mu}_{\mathcal{S}} &= \mathbf{Z}_{T, \mathcal{S}} \tilde{\boldsymbol{\mu}}_{f} + c \left(\boldsymbol{\alpha}_{T} - \mathbf{Z}_{T} \boldsymbol{g}\right)_{\mathcal{S}}\,, \\ \mathbf{\Sigma}_{\mathcal{S} \mathcal{S}} &= \mathbf{Z}_{T, \mathcal{S}} \hat{\mathbf{\Sigma}}_{f} \mathbf{Z}_{T, \mathcal{S}}^\intercal + \mathbf{D}_{\mathcal{S} \mathcal{S}}\,. \end{align}\]

Where:

  • $\boldsymbol{f}_{t}$: Factor returns of observation $t$, $K \times 1$, with entry $f_{tk}$ for factor $k$.
  • $\boldsymbol{x}_t$: Asset returns for observation $t$, the $t$-th row of the returns matrix.
  • $\boldsymbol{x}^{\mathrm{loc}}_{t}$: Returns the regression reads at observation $t$, $N \times 1$. They are $\boldsymbol{x}_{t}$ without Currency Factors, and the local returns of cross_sectional_local_returns with them.
  • $\mathbf{Z}_{t}$: Factor Exposures of observation $t$, $N \times K$, after the Neutralisation and on the reduced factor axis. $\mathbf{Z}_{T, \mathcal{S}}$ holds the rows of the assets of $\mathcal{S}$. Under Currency Factors the regression reads the estimated exposures alone, and every other line reads $\mathbf{Z}_{t}$ with the currency exposures appended as its last columns, $\boldsymbol{f}_{t}$ with the Currency Excess Returns appended, and $\boldsymbol{g}$ with zeros appended. The shrinkage $\lambda$ then reaches the estimated factors alone, and the expected return of a Currency Factor is the mean the nested factor prior states for it.
  • $\ell$: Exposure lag, the count of observations by which the exposures lag the returns.
  • $\mathbf{Q}_{t}$: Regression weight matrix of observation $t$, diagonal, holding the cross-sectional weight of each asset. An asset that is not eligible at observation $t$ takes a weight of zero.
  • $\boldsymbol{\varepsilon}_{t}$: Idiosyncratic returns of observation $t$, $N \times 1$.
  • $\boldsymbol{g}$: Coefficients of the part of the Return Forecast that the latest exposures span, $K \times 1$. It is zero when the prior states no Return Forecast Estimator.
  • $\boldsymbol{\alpha}_{t}$: Return Forecast cross-section at observation $t$. The weight of an asset whose forecast or whose exposures are not finite is zero in the fit of $\boldsymbol{g}$.
  • $\hat{\boldsymbol{\mu}}_{f}$, $\hat{\mathbf{\Sigma}}_{f}$: Expected factor returns and factor covariance that the nested factor prior states over the factor returns $\boldsymbol{f}_{t}$.
  • $\tilde{\boldsymbol{\mu}}_{f}$: Blended expected factor returns, $K \times 1$.
  • $\lambda \in [0, 1]$: Spanned Shrinkage of the expected factor returns towards $\boldsymbol{g}$. A rule in the slot gives the number it returns.
  • $c \in [0, \infty)$: Orthogonal Forecast Scale, the scale of the part of the Return Forecast that the latest exposures do not span. A rule in the slot gives the number it returns.
  • $\boldsymbol{\mu}$: Expected returns vector $N \times 1$.
  • $\mathbf{\Sigma}$: Asset covariance matrix, $N \times N$.
  • $\mathbf{D}$: Covariance of the idiosyncratic returns that the loadings block carries, $N \times N$. It is diagonal when the block stores a variance vector.
  • $\mathcal{S}$: Assets whose latest exposures are finite, whose systematic part the model states. An inactive asset has no finite exposure, so every asset of $\mathcal{S}$ is active at the latest observation.
  • $\mathcal{I}$: Investable assets, those that the Asset Panel activates at the latest observation and whose idiosyncratic variance and latest exposures are finite.
  • $T$: Number of observations.
  • $N$: Number of assets.
  • $K$: Number of factors.

The prior states an entry exactly when the model determines it. An asset of $\mathcal{S}$ outside $\mathcal{I}$, for example one in the warm-up of its idiosyncratic variance, has a finite expected return, and under a diagonal $\mathbf{D}$ a finite covariance with each other asset of $\mathcal{S}$, because the model sets the idiosyncratic covariance of two assets to zero. Its variance is NaN, and under a positive th so is each of its covariances, which read its variance. Every entry of $\boldsymbol{\mu}$ and every row and column of $\mathbf{\Sigma}$ outside $\mathcal{S}$ is NaN. The Investable Mask needs a finite mean and a finite variance, so an optimiser still reads $\mathcal{I}$ alone. An Empty Factor, an estimated factor whose exposure is zero at every pair of positive weight, has $f_{tk} = 0$ at every observation, and $\hat{\mathbf{\Sigma}}_{f}$ holds a zero row and column for it. It keeps its place on every factor axis, so a sub-universe with no asset in one level of a one-hot factor still fits. Without a Return Forecast Estimator the forecast terms are zero, so $\boldsymbol{\mu}_{\mathcal{S}} = \lambda \mathbf{Z}_{T, \mathcal{S}} \hat{\boldsymbol{\mu}}_{f}$. At $\lambda = 0$ and $c = 1$ the two parts of the forecast add up to $\boldsymbol{\alpha}_{T}$, so the expected return of an asset of $\mathcal{S}$ with a finite forecast is that forecast.

Algorithm

  1. Orient X, F and E by dims, rebuild the returns data that the Descriptors read, from X, F, ne, E, pnl, iv and ivpa, and take the two universe masks off pnl with cross_sectional_panel_masks. Split the factor list into estimated and observed members with cross_sectional_factor_partition.
  2. Build the benchmark weights BW with cross_sectional_cap_weights, over the assets of the estimation universe whose return and market capitalisation are finite, and write them onto a copy of the Asset Panel with cross_sectional_benchmark_returns. The universe and the warm-up read X, or the named net returns under lx, so a gap in an observed series does not move them. A benchmark power of zero reads no market capitalisation.
  3. Read the observed factors from the copy with cross_sectional_observed, so an observed member that wraps a CompositeExposure reads the benchmark weights. Take the returns the regression reads, Xl, with cross_sectional_local_returns. Build every estimated Factor Exposure with cross_sectional_exposure_history, one dependency layer after another and the members of a layer under ex, giving Ms, nf and fam. Under observed factors the estimated members read Xl in place of X, so a Descriptor of the returns measures the returns the regression explains. The first pe.lag rows of the derived Xl have no lagged exposure, so the derivation takes the exposure of the same observation there. An observed member cannot read Xl, because Xl is derived from its exposures. So the observed members that can read returns read X net of the observed members that read none, as cross_sectional_observed states.
  4. Drop the leading observations the Descriptors warm up over, with cross_sectional_warmup on the estimated and the observed exposures together, giving the rows rw. cross_sectional_exposure_stage runs steps 1 to 4.
  5. Neutralise the exposures with neutralise_exposures!, under the benchmark weights and the prior's own regression estimator.
  6. Build the Factor Family Basis fb with cross_sectional_family_basis, and reduce the exposures through it.
  7. Lag the reduced exposures and the market capitalisation by pe.lag, giving Zl and mcl. Trim the observed factors to the fitted observations with cross_sectional_observed_block, which refuses an infinite observed return among them and keeps a NaN as a gap. Take the eligibility mask msk of the fit with cross_sectional_eligible on Xl, and drop from it every pair whose lagged market capitalisation is not finite.
  8. Take the design of the regression under the Unseen Member rule unseen with unseen_member_design. Regress each observation's Xl on the design it gives with cross_sectional_live_regression, and map the coefficients back with unseen_member_returns, giving csr and the mask lv of the factors that are not empty, under the weights W of cs_weights_initial. Refuse a csr that carries an intercept with assert_cross_sectional_no_intercept. When needs_second_pass answers true, refine the weights with cs_weights_refine and regress again. The refinement reads the idiosyncratic variance under the two universe masks, as step 9 does.
  9. Take the idiosyncratic variance history vs with variance_series, under the active mask and the estimation mask of the fitted observations. An estimator that reads the masks resets an asset that the active mask turns off, and measures its regime over the estimation universe alone. An estimator that reads no mask ignores them. Standardise the idiosyncratic returns by vs with cross_sectional_standardised_residuals, giving S, and take the latest idiosyncratic covariance esigma with cross_sectional_idiosyncratic_covariance, from the same residuals with no fill, so a gap stays a gap for the correlation. Record the degrees of freedom and the divisor of each variance with variance_count and cross_sectional_variance_counts.
  10. Append the observed factors after the estimated ones with cross_sectional_observed_append: the observed returns after the factor returns, the observed exposures after the loadings and the exposure history, the names and the family labels, and pass-through factors on the Factor Family Basis. Fit pe.pe on the combined reduced factor returns of the factors that are not empty with cross_sectional_factor_moments, giving f_pr, which refuses a non-finite factor moment, divides the scenario weights by their sum, and processes the factor covariance under pe.f_mp, the matrix processing estimator of the factor axis and not the asset one. An Observed Factor is never empty. The method passes strict to pe.pe, as FactorPrior does, because the slot admits BlackLittermanPrior and EntropyPoolingPrior, which resolve view names against a universe. cross_sectional_assemble runs the standardisation and the counts of step 9 and the steps 11 to 17, and the call with no data of the carry fold runs it too.
  11. Build the CrossSectionalFactorModel block csfm, with the raw exposures of the latest observation in M, the reduced ones L beside them, a zero b, and the observed returns in fx. Record the rows of X that the fit covers, rw[r], in idx, and their timestamps in ts when ts is given. Under a family re-basis, expand the combined factor returns onto the raw axis with cross_sectional_expand, each row with the basis of its lagged exposures, and store them in fr.
  12. Fit the Return Forecast with cross_sectional_return_forecast, on the full returns data that the estimated members read, so that a Descriptor of the forecast warms up over every observation the panel has, giving the spanned coefficients g and the unscaled orthogonal part ap. Under observed factors the forecast thus reads Xl, and it forecasts the net return of each asset, which the split measures against loadings of the same returns. Resolve lambda and c with cross_sectional_calibration, and write b = c ap, the Result and the two numbers onto the block rr with cross_sectional_forecast_block. Blend the spanned part into the mean of the estimated factors with cross_sectional_forecast_mu, and keep the mean of the observed factors, giving f_mu.
  13. Expand the blended factor moments onto the raw factor axis with cross_sectional_expand, so fpr states the distribution of the factors the caller named. Its factor returns are fr, or the combined factor returns when no family is constrained.
  14. Take the investable assets idx with cross_sectional_investable.
  15. Rebuild the asset return scenarios Xs with cross_sectional_scenarios.
  16. Lift the reduced factor distribution onto the assets with cross_sectional_lift, and add b to the expected return it answers. The lift states every entry the model determines: the investable block, and the mean and the systematic covariances of an asset with finite exposures and no idiosyncratic variance.
  17. Assemble a LowOrderPrior over Xs, with the fitted base-currency returns under o_X, the three lifted moments, the factor prior's w divided by its sum, its ens, kld and ow, the block under rr, and the expanded factor prior under fpr. The factor returns of fpr and the returns of o_X keep the last rows alone, as many as Xs holds, so a factor prior with a Scenario Cap pairs each scenario with its own observation.

Arguments

  • pe: Cross-Sectional Factor Prior estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • F: Data matrix observations × factors if the dims keyword does not exist or dims = 1, factors × observations when dims = 2. The fit does not read it, because this estimator builds its own factors from the panel. The method writes it onto the rebuilt returns data only so that the returns data states what the caller held.
  • pnl: Optional AssetPanel, the panel that the returns data held. A wrapping prior forwards it unchanged, so that it can compose an estimator that is fitted on a panel. An estimator that reads no panel ignores it. This estimator reads it, and refuses without it.
  • dims: Dimension along which the observations lie.
  • ne: Names of the Exogenous Series, written onto the rebuilt returns data.
  • E: The Exogenous Series, oriented by dims as X is. The observed factors read their returns from it.
  • iv: Implied volatilities, written onto the rebuilt returns data.
  • ivpa: Implied-volatility risk-premium adjustment, written onto the rebuilt returns data.
  • ts: Timestamp vector. It has one entry for each observation of X, and the block records the entries of the rows it fits, or nothing.
  • strict: Whether to throw an error if sets does not contain the desired value in sets.dict[key]. It is forwarded to the nested factor prior pe.pe.
  • cache_reg: If true, a prior stores on its result the cache that its refit under new scenario weights reads: a CrossSectionalFactorPrior the active mask and the Return Forecast history on its block, and an OpinionPoolingPrior the rows of pe1, its pooled weights and the cache of pe2 in pc. The refit of an entropy pooling prior around it then runs no regression and no opinion. Every other prior forwards it to the prior that it wraps. Default is false.
  • kwargs...: Additional keyword arguments passed to the verbs of the algorithm.

Validation

  • pnl is not nothing. Raises an IsNothingError.
  • If provided, length(ts) equals the number of observations of X. Raises a DimensionMismatch.
  • The Asset Panel is time-varying. Raises an ArgumentError.
  • The Asset Panel holds no Panel Field named pe.bw, because the fit writes its benchmark weights onto that name. Raises an ArgumentError.
  • The history is longer than the exposure lag. Raises an ArgumentError.
  • At least two observations are left after the Descriptor warm-up and the exposure lag, because a covariance of one observation is not a number. Raises an ArgumentError.
  • Every fitted observation carries at least minra eligible assets. Raises an ArgumentError.
  • Under observed factors, E carries a column for every observed series, and no observed return of the fitted observations is infinite. Raises an IsNothingError, an ArgumentError or an IsNonFiniteError, from cross_sectional_observed and cross_sectional_observed_block.
  • At least one estimated factor is not empty. Raises an ArgumentError.
  • The regression fits no intercept. The prior states the moments through the factor returns alone, so an intercept would leave its mean out of mu and its variance out of sigma. Raises an ArgumentError.
  • The factor prior determines the factor moments of at least one asset: the loadings of the asset read only factors with a finite mean and a finite variance. Raises an IsNonFiniteError, from assert_cross_sectional_factor_moments.
  • The scenario weights of the factor prior, when it carries any, sum to a positive number. Raises a DomainError, from cross_sectional_scenario_weights.
  • At least one asset is investable at the latest observation. Raises an IsEmptyError.
  • The rules of every verb the algorithm names.

Returns

  • pr::LowOrderPrior: The prior on the full asset universe. An entry that the model does not determine is NaN: the whole entry of mu and row and column of sigma of an asset with no finite latest exposure, each entry that reads a factor moment the factor prior does not state, and the variance of an asset with no idiosyncratic variance, with its covariances under a positive th. An asset that loads on a factor whose mean or variance the factor prior does not state leaves the Investable Mask, and every other asset keeps its moments. rr is a CrossSectionalFactorModel, and fpr is the factor prior on the raw factor axis. w holds the scenario weights of the factor prior divided by their sum, so they sum to one, and fpr holds the same w. It is nothing when the factor prior carries no weights.

Related

source
PortfolioOptimisers.prior — Method
prior(pe::CrossSectionalFactorPrior, rd::ReturnsResult; strict::Bool = false,
      dims::Int = 1, cache_reg::Bool = false) -> LowOrderPrior

Fit a Cross-Sectional Factor Prior from a ReturnsResult.

The method passes the fields of rd to the returns-matrix method above, which runs the fit. The file defines this method instead of using prior(pe::AbstractPriorEstimator, rd::ReturnsResult), so that the refusal of an rd with no Asset Panel names rd.pnl, the field of rd that the caller built.

Algorithm

  1. Check that rd carries asset returns.
  2. Drop the observations of a gap of a vector benchmark under pe.x_src = ExcessSource() with shared_series_view. The estimator does not read the factor returns, so a gap in them drops nothing.
  3. Read the asset returns with benchmark_relative_returns under pe.x_src, giving X.
  4. Call the returns-matrix method with X, rd.F and rd.pnl, forwarding rd.ne, rd.E, rd.iv, rd.ivpa, rd.ts, strict and cache_reg as keyword arguments, and dims = 1, because a ReturnsResult holds its observations along the rows.
  5. Record the observations that step 2 dropped with shared_series_record, and pe.x_src with stamp_returns_source.

Arguments

  • pe: Cross-Sectional Factor Prior estimator.
  • rd: The returns result to use. It must carry asset returns in rd.X, and a time-varying Asset Panel in rd.pnl when an exposure reads a Panel Field. With no panel, the fit reads the mask-only panel of returns_with_universe.
  • strict: Passed to the returns-matrix method of the prior, where it decides if a scenario fill or a coverage peel raises in place of a warning.
  • dims: Accepted and ignored. A ReturnsResult holds its observations along the rows, so the call passes dims = 1. The method takes no other keyword that it does not list: an unknown keyword raises the MethodError of the language, and no estimator swallows it.
  • cache_reg: If true, a prior stores on its result the cache that its refit under new scenario weights reads: a CrossSectionalFactorPrior the active mask and the Return Forecast history on its block, and an OpinionPoolingPrior the rows of pe1, its pooled weights and the cache of pe2 in pc. The refit of an entropy pooling prior around it then runs no regression and no opinion. Every other prior forwards it to the prior that it wraps. Default is false.

Validation

Returns

  • pr::LowOrderPrior: The prior the returns-matrix method fitted.

Related

source
PortfolioOptimisers.AbstractSpannedShrinkageCalibrationAlgorithm — Type
abstract type AbstractSpannedShrinkageCalibrationAlgorithm <: AbstractCalibrationAlgorithm

Computes the Spanned Shrinkage, the lambda of a Cross-Sectional Factor Prior, from the fit.

The Spanned Shrinkage blends the expected factor returns of the nested factor prior with the spanned coefficients g of the Return Forecast, and it lies in [0, 1]. A rule of this family stands in the lambda slot of CrossSectionalFactorPrior, and Num_SpanShrinkCal is the bound of that slot, so a rule of another family is refused at construction.

A rule is called as alg(key, pr, w, slv, ctx), as resolve_calibration_slot states, and returns the shrinkage. The prior refuses a returned number outside [0, 1].

Interfaces

In order to implement a new concrete type that works seamlessly with the library, subtype AbstractSpannedShrinkageCalibrationAlgorithm and implement the following methods:

The functor

  • (alg::AbstractSpannedShrinkageCalibrationAlgorithm)(key::Symbol, pr::AbstractPriorResult, w, slv, ctx::CalibrationContext) -> Number: Returns the Spanned Shrinkage.

Arguments

  • key: Name of the slot that is being resolved, :lambda.
  • pr: The factor moments of the nested factor prior, on the reduced factor axis of the estimated factors and the observed factors.
  • w: The observation weights of pr, or nothing.
  • slv: nothing.
  • ctx: A CalibrationContext whose cs carries the fit of the prior.

Returns

  • lambda::Number: The Spanned Shrinkage, in [0, 1].

reads_forecast_history

  • reads_forecast_history(alg::MySpannedShrinkage) -> Bool: Returns true when the rule reads ctx.cs.hist. Add it only for a rule that reads the history, because the default answers false.

Related

source
PortfolioOptimisers.AbstractOrthogonalForecastScaleCalibrationAlgorithm — Type
abstract type AbstractOrthogonalForecastScaleCalibrationAlgorithm <: AbstractCalibrationAlgorithm

Computes the Orthogonal Forecast Scale, the c of a Cross-Sectional Factor Prior, from the fit.

The Orthogonal Forecast Scale multiplies the part of the Return Forecast that the latest Factor Exposures do not span, and it lies in [0, ∞). A rule of this family stands in the c slot of CrossSectionalFactorPrior, and Num_OrthFcScaleCal is the bound of that slot, so a rule of another family is refused at construction.

A rule is called on the same terms as a rule of AbstractSpannedShrinkageCalibrationAlgorithm, and returns the scale. The prior refuses a returned number that is negative or not finite.

Interfaces

In order to implement a new concrete type that works seamlessly with the library, subtype AbstractOrthogonalForecastScaleCalibrationAlgorithm and implement the following methods:

The functor

  • (alg::AbstractOrthogonalForecastScaleCalibrationAlgorithm)(key::Symbol, pr::AbstractPriorResult, w, slv, ctx::CalibrationContext) -> Number: Returns the Orthogonal Forecast Scale.

Arguments

  • key: Name of the slot that is being resolved, :c.
  • pr: The factor moments of the nested factor prior, on the reduced factor axis of the estimated factors and the observed factors.
  • w: The observation weights of pr, or nothing.
  • slv: nothing.
  • ctx: A CalibrationContext whose cs carries the fit of the prior.

Returns

  • c::Number: The Orthogonal Forecast Scale, finite and >= 0.

reads_forecast_history

  • reads_forecast_history(alg::MyOrthogonalForecastScale) -> Bool: Returns true when the rule reads ctx.cs.hist. Add it only for a rule that reads the history, because the default answers false.

Related

source
PortfolioOptimisers.reads_forecast_history — Function
reads_forecast_history(rule) -> Bool

Answer whether the occupant of a slot of a Cross-Sectional Factor Prior reads the Return Forecast history.

A member that computes no history fits the Return Forecast again at each observation to make one, so the prior makes the history only when a slot asks for it, and it makes it once for both slots. forecast_history makes it, and the CalibrationContext carries it in cs.hist. The default answers false, so a stated number and a plain function never get the history. A rule type that reads it adds a method that answers true.

Arguments

  • rule: The occupant of the slot.

Returns

  • flag::Bool: Whether the occupant reads the Return Forecast history.

Examples

julia> PortfolioOptimisers.reads_forecast_history(0.5)false

Related

source