Cross-Sectional Prior Fold: private API

struct CrossSectionalCarryState{__T_buf, __T_win, __T_der, __T_nf, __T_fam, __T_Ms, __T_X, __T_Xl, __T_obs, __T_bw, __T_mcap, __T_amsk, __T_emsk, __T_sums, __T_families, __T_fcb, __T_Z, __T_csr, __T_lv1, __T_lv, __T_W, __T_vs, __T_ve, __T_ve1, __T_pe, __T_seed, __T_hist, __T_S, __T_Sc, __T_ce, __T_fsc, __T_fh, __T_fst, __T_xf, __T_fds, __T_tip} <: AbstractPartialFitState

Carries the carry fold of a Cross-Sectional Factor Prior between two online steps.

A CrossSectionalFactorPrior with no cache seeds this state at its first partial_fit!, as the carry of EmpiricalPrior seeds a PriorCarryState. The state applies the rule of the carry fold: it folds what folds and refits the rest. A new observation changes no past Factor Exposure, no past regression and no past idiosyncratic variance, so a step computes them for the new observations alone. The idiosyncratic variance and the factor prior fold exactly. So does the idiosyncratic correlation under th > 0 when its ce folds, as supports_partial_fit answers. The Return Forecast, and a ce that does not fold, refit at the call with no data from the carried rows. A Return Forecast that computes its history one observation at a time, as folds_forecast_rows answers, folds too: the state carries its Descriptor scores and its rows, and a step computes the rows of its new observations alone.

Two choices read every fitted observation: the automatic dropped member of a Factor Family under a BatchChoice, and the mark of the Empty Factors. A step that moves the dropped member solves no observation of full rank again: cross_sectional_fold_move selects the columns of the raw factor returns that the new member keeps, and folds the factor prior again over them. That fold reads every fitted factor return, so its cost grows with the stream. The move solves each rank-deficient observation again with cross_sectional_move_solve: an observation with an Unseen Member under SolvedUnseenMember, or one with a dependent factor set. Under a CrossSectionalTargetRegression whose target answers false to is_basis_invariant, a move fits every carried observation again, as cross_sectional_move_basis states. Under a GeneralisedLinearModel target the move folds too, because its iterative fit takes the same iterates in every basis. A step where an Empty Factor comes alive solves no fitted observation again: the factor has a return of zero at each of them, and cross_sectional_step_factors folds the factor prior again over every fitted factor return. That fold reads every fitted observation, so its cost grows with the stream, and it happens at most once for each factor. Under a regression estimator that changes the answer of an observation whose design gains a zero column, as cross_sectional_alive_folds answers, that step fits every carried observation again.

A prior whose tree reads the Exogenous Series, as reads_exogenous_series answers, folds an observed factor too. The buffer and the carried rows hold the series. The returns net of the observed factors are derived series: the batch fit derives each row from the observed exposures of the row lag observations before it, and the first lag rows of the sample from the exposure of the same row. So the state derives each row one time, and carries it.

A slot that reads the Return Forecast history, as reads_forecast_history answers, makes the state carry that history, unless the member publishes its own, as cross_sectional_carries_history answers. The row of a block observation reads that observation and the observations before it, so a new observation changes no old row. A step that fits the new observations alone therefore fits the Return Forecast once for each new row and appends the row. A step that fits every observation again can change every row, so it makes every row again. The state keeps a row for every observation of the block, because the batch fit gives a rule every row. A Scenario Cap on the factor prior cuts its scenarios and no observation of the block, so it cuts no row of the history.

Fields

• buf: Buffer of every folded observation: the returns, both masks, the implied volatilities and ivpa when the steps carry them, and the Exogenous Series when the prior reads it. The online step of an optimiser rebuilds its returns data from it with returns_buffer.

• win: The carried rows of the Asset Panel, as a ReturnsResult of the returns, the panel, the implied volatilities and ivpa when the steps carry them, and the Exogenous Series when the prior reads it, or nothing before the first step. It holds the last rows that the Factor Exposures of a new observation read, as cross_sectional_carry_rows states, or every row.

• der: The derived series of the carried rows, (; Xn, Xl, Zo), or nothing without an observed factor. Xn holds the returns net of the observed members that read no returns, which the other observed members read, or nothing when no member needs them. Xl holds the returns net of every observed factor, which the estimated members read. Zo holds the observed exposures of the last lag rows, observations × assets × factors, which the derivation of the rows of the next step reads.

• nf: Names of the raw factor axis, or nothing before the first step.

• fam: Factor Family label of each raw factor, or nothing before the first step.

• Ms: Neutralised exposures of every observation after the Descriptor warm-up, observations × assets × raw factors, or nothing before the warm-up ends.

• X: Returns of every observation after the warm-up, observations × assets.

• Xl: Returns net of the observed factors of every observation after the warm-up, which the regression reads, or nothing without an observed factor, when the regression reads X.

• obs: Observed factors of every observation after the warm-up, (; Z, R, lv, nf, fam) as cross_sectional_observed states them, or nothing without an observed factor. The factor prior reads the observed returns R beside the factor returns of the regression. The observed exposures Z and the exposure history Ms are views of one backing, as cross_sectional_fold_join appends them, so the call with no data reads the joined history with no copy.

• bw: Benchmark weights of every observation after the warm-up.

• mcap: Market capitalisation of every observation after the warm-up, or nothing when the prior reads none.

• amsk: Active mask of every observation after the warm-up.

• emsk: Estimation mask of every observation after the warm-up.

• sums: Sum over the observations after the warm-up of the absolute benchmark-weighted exposure of each raw factor, which the automatic choice of a dropped member reads, or nothing without a constrained Factor Family.

• families: The constrained Factor Families with the dropped member of each one named, as the last fit chose them, or nothing.

• fcb: Factor Family Basis of every observation after the warm-up, or nothing.

• Z: Reduced exposures of the last lag observations, which the regression of the next observations reads.

• csr: The regression of every fitted observation, or nothing before the first fit.

• lv1: Mark of the factors that are not empty in the first pass of the regression.

• lv: Mark of the factors that are not empty in the last pass of the regression.

• W: Regression weights of every fitted observation.

• vs: Idiosyncratic variance history of every fitted observation.

• ve: The variance estimator ve of the prior, folded over the residuals of the last pass.

• ve1: The variance estimator ve of the prior, folded over the residuals of the first pass, or nothing when the weight policy needs no second pass.

• pe: The factor prior pe of the prior, folded over the factor returns, or the factor prior as it is when it does not fold.

• seed: Number of fitted observations of the first fit. A fit of every carried observation folds them as one block again, so a SeedWindow cuts the rows that it cut at the first fit.

• hist: The Return Forecast history at every fitted observation but the last, observations × assets, or nothing when the state carries none. The row of an observation is the forecast of the member fitted on that observation and the observations before it, as forecast_history_refit fits it. The call with no data appends the forecast of the last observation.

• S: The standardised idiosyncratic returns of every fitted observation, with the fill, as cross_sectional_standardised_residuals states them, or nothing before the first fit. A row reads its own observation alone, so a step appends the rows of its new observations, and the call with no data reads them as they are.

• Sc: The standardised idiosyncratic returns of every fitted observation with no fill, which the idiosyncratic correlation reads, or nothing before the first fit, when the threshold th of the prior is zero, and when ce folds them.

• ce: The covariance estimator ce of the prior, folded over the standardised idiosyncratic returns with no fill of every fitted observation, or nothing before the first fit, when the threshold th of the prior is zero, and when ce does not fold, as supports_partial_fit answers. A step folds the rows of its new observations into it, so the call with no data reads the correlation from its state.

• fsc: The Descriptor scores of the Return Forecast at every observation after the warm-up, (; S, w, g) as descriptor_panel_scores states them, or nothing when the forecast does not compute its history one observation at a time, as folds_forecast_rows answers. The scores of the last rows of the warm-up that cross_sectional_forecast_lead counts come first. A step appends the scores of its new observations, with cross_sectional_carry_scores.

• fh: The Return Forecast history at every fitted observation, observations × assets, or nothing when the forecast does not fold. The call with no data reads the Result of the forecast off it, with return_forecast_result.

• fst: The fold state of the Return Forecast after the fitted observations whose target is known, as return_forecast_step gives it, or nothing when the forecast does not fold or carries no state. A step passes it to the forecast, and the call with no data builds the Result of the forecast with it.

• xf: The factor list, as cross_sectional_descriptor_carry and cross_sectional_observed folded its members, so that each Descriptor that carries a state holds its state after the folded observations, or nothing before the first step. A step folds its new observations into them, and reads the Descriptors of those observations off them.

• fds: The Descriptor Scores of the Return Forecast, as descriptor_carry folded their Descriptors, so that each Descriptor that carries a state holds its state after the folded observations. It is nothing before the first step, and when the forecast does not compute its history one observation at a time, as folds_forecast_rows answers. A step folds its new observations into them, and scores those observations with the Descriptors that it reads off them, in cross_sectional_carry_scores.

• tip: The number of observations that the newest state of this lineage folded. Every state that a step derives from this one shares the counter. A history appends in place only when the state is the newest one, as cross_sectional_carry_own checks.

Constructors

CrossSectionalCarryState(; buf::SampleBufferState = SampleBufferState(),                         win::Option{<:ReturnsResult} = nothing, der = nothing,                         nf = nothing, fam = nothing, Ms = nothing, X = nothing,                         Xl = nothing, obs = nothing, bw = nothing, mcap = nothing,                         amsk = nothing, emsk = nothing, sums = nothing,                         families = nothing, fcb = nothing, Z = nothing, csr = nothing,                         lv1 = nothing, lv = nothing, W = nothing, vs = nothing,                         ve = nothing, ve1 = nothing, pe = nothing,                         seed::Integer = 0, hist = nothing, S = nothing,                         Sc = nothing, ce = nothing, fsc = nothing, fh = nothing,                         fst = nothing, xf = nothing, fds = nothing,                         tip::Base.RefValue{Int} = Ref(0)) -> CrossSectionalCarryState

Keywords correspond to the struct's fields, and every field but buf and tip defaults to nothing, or to 0 for seed. tip defaults to a new counter at 0. The default is the empty state that a first step builds.

Related

• CrossSectionalFactorPrior
• partial_fit!
• cross_sectional_carry_fold
• PriorCarryState
• SampleBufferState

copy(
    x::CrossSectionalCarryState
) -> Union{CrossSectionalCarryState{__T_buf, Nothing, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6, Base.RefValue{Int64}} where {__T_buf<:SampleBufferState, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed<:Integer, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6}, CrossSectionalCarryState{__T_buf, __T_win, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6, Base.RefValue{Int64}} where {__T_buf<:SampleBufferState, __T_win<:ReturnsResult, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed<:Integer, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6}}

Copies a CrossSectionalCarryState, so that the copy shares no array and no folded state with the original. The copy starts a lineage of its own, so its first step appends to its histories in place.

Arguments

• x: The state to copy.

Returns

• state::CrossSectionalCarryState: A new state, equal to x.

Related

• CrossSectionalCarryState
• partial_fit

cross_sectional_forecast_reads_panel(rfe::Nothing)
cross_sectional_forecast_reads_panel(rfe::CustomValueReturnForecast)
cross_sectional_forecast_reads_panel(rfe::AbstractReturnForecastEstimator)

Answers whether the Return Forecast of a Cross-Sectional Factor Prior reads the Asset Panel at the call with no data of the carry fold.

A Return Forecast that reads the panel refits at each call with no data from the carried rows. Its Result keeps a value at every fitted observation, and return_forecast_rows aligns the fitted observations with the last rows of the returns data, so the carry fold carries every panel row under it. A CustomValueReturnForecast and an absent forecast read no row. A forecast that computes its history one observation at a time, as folds_forecast_rows answers, reads no row at the call with no data either: a step computes its rows, and the state carries them.

Arguments

• rfe: The Return Forecast Estimator, or nothing.

Returns

• reads::Bool: true when the forecast reads the panel.

Related

• cross_sectional_carry_rows
• CrossSectionalCarryState

cross_sectional_carry_rows(
    pe::CrossSectionalFactorPrior
) -> Union{Nothing, var"#s2313"} where var"#s2313"<:Integer

Returns the number of panel rows that the carry fold of a Cross-Sectional Factor Prior carries, or nothing for every row.

The Factor Exposures of a new observation read the last rows of the panel that carry_lookback of the prior counts, so the carry keeps them. A Descriptor that carries a state reads the new observation alone, as descriptor_carry folds it, so it adds no row. Under a Return Forecast that reads the panel at the call with no data, as cross_sectional_forecast_reads_panel answers, the carry keeps every row. A forecast that computes its history one observation at a time reads its last carry_lookback rows, which carry_lookback of the prior counts: a Descriptor of its scores that carries a state counts one row. An unbounded look-back keeps every row too.

Arguments

• pe: Cross-Sectional Factor Prior estimator.

Returns

• n::Option{<:Integer}: The number of rows, or nothing.

Related

• carry_lookback
• CrossSectionalCarryState

cross_sectional_descriptor_carry(est::AbstractVector, rd::ReturnsResult, n::Nothing)
cross_sectional_descriptor_carry(est::AbstractVector, rd::ReturnsResult, n::Integer)

Folds the last n observations of a ReturnsResult into the Descriptors of the estimated members of a factor list that carry a state, on the carry fold of a CrossSectionalFactorPrior.

Each member folds with descriptor_carry. The batch fit gives n = nothing, and gets the members as they are twice.

Arguments

• est: The estimated members, pairs of factor name => Exposure Estimator, as the carry folded them so far.
• rd: The returns result to use. It holds the panel rows that the carry carries, followed by the new observations.
• n: The number of new observations, or nothing.

Returns

• carry::NamedTuple: xf, the members with the state after the new observations, and xv, the members whose stateful Descriptors are CarriedDescriptor of the new observations.

Related

• descriptor_carry
• cross_sectional_exposure_series

cross_sectional_window_append(
    _::Nothing,
    rd::ReturnsResult,
    xk::NamedTuple
) -> ReturnsResult

Appends the observations of a step to the carried panel rows, and returns the rows that the exposures of the step read.

The Descriptors of a batch fit read iv and ivpa off the returns data, so the rows carry them. A row of iv belongs to the observation of the same row, so it is appended with the returns. ivpa has no observation axis, so the first step pins it.

Arguments

• win: The carried rows, or nothing before the first step.
• rd: The returns data of the step.
• xk: The Exogenous Series of the step, (; ne, E) as exogenous_step_kwargs answers when the prior reads it, or an empty NamedTuple.

Validation

• win.iv and rd.iv are both nothing, or neither is, as assert_buffer_presence_agreement checks. An ArgumentError is thrown otherwise.
• rd.ivpa equals win.ivpa, as assert_pinned_value checks. An ArgumentError is thrown otherwise.

Returns

• rd::ReturnsResult: The carried rows followed by the rows of the step, with the returns, the Asset Panel, iv, ivpa and the series of xk alone.

Related

• cross_sectional_carry_fold

cross_sectional_window_trim(
    rd::ReturnsResult,
    _::Nothing
) -> ReturnsResult

Keeps the last n rows of the panel rows of a step, of their derived series, or of the Descriptor scores of a Return Forecast, or every row when n is nothing.

Arguments

• rd: The panel rows, an array whose first dimension is the rows, the derived series (; Xn, Xl, Zo) of the rows, the scores (; S, w, g), or nothing. The carry keeps more rows than lag, so the cut keeps every row of Zo.
• n: The number of rows to keep, or nothing.

Returns

• rd: The rows that the state carries, of the type of the input.

Related

• cross_sectional_carry_rows

cross_sectional_fold_variance(
    ve,
    E::AbstractMatrix{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}},
    em::AbstractMatrix{Bool},
    am::AbstractMatrix{Bool},
    seed::Integer
) -> NamedTuple{(:V, :ve), <:Tuple{Any, Any}}

Folds a variance estimator over a block of residuals, and returns its variance after each row.

The first seed rows fold as one block: the series of those rows is the batch series, and a SeedWindow cuts the block. Each later row folds alone, and the variance after it is read from the folded state, so a row reads no residual after its own. An estimator that does not fold, as supports_partial_fit answers, gives the batch series of E, and the carry fold then gives it every fitted row at each step.

Arguments

• ve: The variance estimator, folded so far.
• E: Residuals, observations × assets.
• em: Estimation mask of the rows of E.
• am: Active mask of the rows of E.
• seed: Number of leading rows to fold as one block, 0 when ve already folded.

Returns

• V::Matrix: The variance after each row of E.
• ve: The estimator after the last row.

Related

• variance_series
• partial_fit!
• cross_sectional_carry_fold

cross_sectional_fold_factors(
    pe::AbstractPriorEstimator,
    f::AbstractMatrix{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}}
) -> Any

Folds the factor returns of new observations into the factor prior of the carry fold.

A factor prior for which carry_folds answers true folds the rows with partial_fit!. Any other factor prior stays as it is, and the call with no data refits it over the factor returns that the state carries.

Arguments

• pe: The factor prior, folded so far.
• f: The factor returns of the new observations, on the factors that are not empty.

Returns

• pe: The factor prior after the new observations.

Related

• carry_folds
• cross_sectional_factor_prior
• CrossSectionalCarryState

cross_sectional_factor_prior(
    pe::AbstractPriorEstimator,
    f::AbstractMatrix{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}};
    strict
) -> Any

Reads the factor prior of the carry fold out of its state.

A factor prior for which carry_folds answers true answers from the state that cross_sectional_fold_factors folded. Any other factor prior refits over the factor returns f.

Arguments

• pe: The factor prior that the state carries.
• f: The factor returns of every fitted observation, on the factors that are not empty.
• strict: Forwarded to the factor prior.

Returns

• pr::LowOrderPrior: The factor prior.

Related

• cross_sectional_fold_factors
• cross_sectional_factor_moments

cross_sectional_fold_regression(
    pe::CrossSectionalFactorPrior,
    blk::NamedTuple,
    prev::NamedTuple
) -> Union{Nothing, NamedTuple{(:csr, :W, :lv1, :lv, :ve1), <:Tuple{CrossSectionalRegression, AbstractMatrix{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}}, BitVector, BitVector, Any}}}

Runs the regression passes of a Cross-Sectional Factor Prior over a block of observations.

The batch fit runs the same passes over every observation. Each observation is fitted on its own row, so a block gives the rows that the batch fit gives, when the masks of the Empty Factors are the masks of every fitted observation. A first fit or a fit of every carried observation states no mark, and the function takes the mark of the block. A step states the marks of the observations it fitted, and regresses the block over those marks joined to the mark of the block. A factor that the block marks and those marks leave empty comes alive at the block. The function answers nothing when the regression estimator does not fold such a factor, as cross_sectional_alive_folds answers, because every observation must then be fitted again.

Algorithm

1. Take the eligibility mask msk with cross_sectional_eligible, drop every pair whose lagged market capitalisation is not finite, and refuse an observation below cross_sectional_minra.
2. Take the first-pass weights with cs_weights_initial, and run the first pass with cross_sectional_fold_pass.
3. When needs_second_pass answers true, fold ve1 over the first-pass residuals with cross_sectional_fold_variance. Invert the variance of the row before each observation with cross_sectional_lagged_inverse, blend the weights with cs_weights_blend, and run the second pass with cross_sectional_fold_pass.

Arguments

• pe: Cross-Sectional Factor Prior estimator.
• blk: The observations: Zl, the lagged reduced exposures; Xr, the returns; em and am, the masks; mcl, the lagged market capitalisation or nothing; cb, the observed factors of the block or nothing; fcb, the Factor Family Basis of the lagged exposures or nothing; B, the lagged exposures on the raw axis; and t0, the number of fitted observations before the block, which a refusal adds to the row of the block it names.
• prev: The marks lv1 and lv, the folded estimator ve1 and the number seed of rows to fold as one block.

Validation

• The rules of every verb the algorithm names.

Returns

• fit::Option{<:NamedTuple}: csr, W, lv1, lv and ve1, or nothing when a factor comes alive and the regression estimator does not fold it.

Related

• cross_sectional_carry_fold
• prior

cross_sectional_previous_variance(ve::Nothing) -> nothing
cross_sectional_previous_variance(ve::AbstractCovarianceEstimator) -> VecNum

Reads the variance of the observation before a block out of the folded first-pass variance estimator.

A first fit holds no folded estimator, and its first observation reads no earlier variance, so the method over nothing answers nothing. A step reads the variance of its folded estimator, and copies it, because the fold of the block writes into the state.

Arguments

• ve: The folded first-pass variance estimator, or nothing.

Returns

• v0::Option{<:VecNum}: The variance of each asset before the block, or nothing.

Related

• cross_sectional_fold_regression
• cross_sectional_lagged_inverse

cross_sectional_fold_pass(
    pe::CrossSectionalFactorPrior,
    blk::NamedTuple,
    W::AbstractMatrix{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}},
    old::Union{Nothing, BitVector}
) -> Union{Nothing, NamedTuple{(:csr, :lv), <:Tuple{CrossSectionalRegression, BitVector}}}

Runs one regression pass of the carry fold of a Cross-Sectional Factor Prior over a block of observations.

Algorithm

1. Take the design of the regression under the Unseen Member rule pe.unseen with unseen_member_design, giving Z.
2. Join the mark of the factors that are not empty over Z to old with cross_sectional_fold_mark. Answer nothing when a factor comes alive at the block and the regression estimator does not fold it.
3. Regress Z on the marked factors with cross_sectional_live_regression, after the t0 observations fitted before the block, map its coefficients back with unseen_member_returns, and refuse an intercept with assert_cross_sectional_no_intercept.

Arguments

• pe: Cross-Sectional Factor Prior estimator.
• blk: The observations, as cross_sectional_fold_regression states them. The pass reads Zl, the lagged reduced exposures; Xr, the returns; fcb, the Factor Family Basis of the lagged exposures or nothing; B, the lagged exposures on the raw axis; and t0, the number of fitted observations before the block.
• W: Regression weights of the block.
• old: The mark of the fitted observations, or nothing for a first fit.

Returns

• pass::Option{<:NamedTuple}: csr, the regression, and lv, the mark, or nothing.

Related

• cross_sectional_fold_regression

assert_cross_sectional_no_intercept(
    csr::CrossSectionalRegression
) -> CrossSectionalRegression

Refuses a cross-sectional regression that fitted an intercept, for a Cross-Sectional Factor Prior.

The prior states its moments through the factor returns alone, so an intercept would leave its mean out of mu and its variance out of sigma. The batch fit and the carry fold both call it after each regression pass.

Arguments

• csr: The regression.

Validation

• csr.b is nothing. An ArgumentError is thrown otherwise.

Returns

• csr::CrossSectionalRegression: The regression, unchanged.

Related

• CrossSectionalFactorPrior
• prior

cross_sectional_fold_mark(
    _::Nothing,
    new::BitVector,
    _
) -> BitVector

Joins the mark of the factors that are not empty over a block to the mark of the fitted observations.

A factor that the block marks and the fitted observations leave empty comes alive at the block. Its exposure is zero at every pair of positive weight of each fitted observation, so the batch fit regresses each such observation over a design with a zero column. The regression estimator decides if that column changes the answer of the observation, as cross_sectional_alive_folds answers. When it does not, the join keeps the fitted observations, and the step folds. When it does, the function answers nothing, and every observation must be fitted again.

Arguments

• old: The mark of the fitted observations, or nothing for a first fit.
• new: The mark of the block.
• cre: The cross-sectional regression estimator of the prior.

Validation

• Without old, at least one factor is not empty. Raises an ArgumentError.

Returns

• lv::Option{BitVector}: new for a first fit, old .| new when no factor comes alive or cre folds it, and nothing otherwise.

Related

• cross_sectional_fold_regression
• cross_sectional_live_factors
• cross_sectional_alive_folds

cross_sectional_fold_factor_returns(f::MatNum, lv::AbstractVector{Bool}, cb::Nothing)
cross_sectional_fold_factor_returns(f::MatNum, lv::AbstractVector{Bool},
                                    cb::NamedTuple)

Returns the factor returns of a block that the factor prior of the carry fold folds: the factor returns of the regression on the factors that are not empty, and the observed returns after them.

The batch fit appends the observed factors after the estimated ones with cross_sectional_observed_append, and fits the factor prior on the factors that are not empty, so the columns are in the same order.

Arguments

• f: The factor returns of the regression of the block, on the reduced axis.
• lv: The mark of the factors that are not empty.
• cb: The observed factors of the block from cross_sectional_observed_block, or nothing.

Returns

• f::Matrix: The factor returns that the factor prior folds.

Related

• cross_sectional_fold_factors
• cross_sectional_carry_fold

cross_sectional_dropped_names(
    _::Nothing,
    _::Nothing,
    _::AbstractVector{<:AbstractString}
)

Names the dropped member of each constrained Factor Family of a basis.

Arguments

• families: The constrained Factor Families, or nothing.
• fcb: The Factor Family Basis that a fit chose for them, or nothing.
• nf: Names of the raw factor axis.

Returns

• families::Option{<:Vector{<:Pair}}: Pairs of family label => dropped member, or nothing.

Related

• cross_sectional_pinned_families
• cross_sectional_carry_fold

cross_sectional_fold_choice(choice::PinnedChoice, pe::CrossSectionalFactorPrior,
                            st::CrossSectionalCarryState)
cross_sectional_fold_choice(choice::BatchChoice, pe::CrossSectionalFactorPrior,
                            st::CrossSectionalCarryState)

Applies the Choice Rule of a Cross-Sectional Factor Prior at a step of its carry fold.

A pinned choice keeps the dropped members of the first fit, which the state records. A batch choice chooses each automatic member again over every observation after the warm-up. It reads the sums that the state carries, with the rule of factor_family_basis: the member with the largest sum of absolute benchmark-weighted exposures, and the first such member on a tie. The state adds the rows to the sums in the order of the batch fit, so the two choose the same member. When the member moves, cross_sectional_fold_move folds the move: it solves no observation of full rank again, and folds the factor prior again over every fitted factor return.

Arguments

• choice: The Choice Rule of pe.
• pe: Cross-Sectional Factor Prior estimator.
• st: The state after the rows of the step.

Returns

• families::Option{<:Vector{<:Pair}}: Pairs of family label => dropped member, or nothing.

Related

• PinnedChoice
• BatchChoice
• cross_sectional_carry_fold

cross_sectional_fold_sums(
    _::Nothing,
    _,
    _::AbstractArray{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}, 3},
    _::AbstractMatrix{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}}
)

Adds the absolute benchmark-weighted exposures of new observations to the sums that the automatic choice of a dropped member reads.

Arguments

• families: The constrained Factor Families of the prior, or nothing.
• sums: The sums so far, or nothing.
• Ms: Neutralised exposures of the new observations.
• bw: Benchmark weights of the new observations.

Returns

• sums::Option{<:Vector}: One sum per raw factor, or nothing without a constrained Factor Family.

Related

• cross_sectional_fold_choice
• factor_family_basis

cross_sectional_fold_rows(
    pe::CrossSectionalFactorPrior,
    st::CrossSectionalCarryState,
    win::ReturnsResult,
    n::Integer,
    g0::Integer
) -> NamedTuple{(:nf, :fam, :Ms, :X, :Xl, :obs, :bw, :mcap, :amsk, :emsk, :der, :xf), <:Tuple{Vector{String}, Vector{String}, Any, Any, Any, Union{Nothing, NamedTuple{(:Z, :R, :lv, :nf, :fam), <:NTuple{5, Any}}}, Any, Any, Any, Any, Union{Nothing, NamedTuple{(:Xn, :Xl, :Zo), <:Tuple{Any, AbstractMatrix{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}}, Any}}}, Any}}

Computes the neutralised Factor Exposures of the new observations of a step, and drops the rows of the Descriptor warm-up.

The exposures of an observation read the last lookback rows of the panel, so the rows that the state carries give the exposures of the batch fit. The derived series of an observation read the observed exposures of the row pe.lag observations before it. A Descriptor that carries a state cannot give that row again, so the state carries the observed exposures of the last pe.lag rows in der.Zo.

Algorithm

1. Run cross_sectional_exposure_series over the rows, with the derived series of the carried rows that the state holds. It gives the benchmark weights, the observed factors, the returns net of them and the exposures of the estimated factors. It derives the rows of the step alone, and it computes the exposures of the n rows of the step alone, each from the last lookback rows that the member reads.
2. Take the last n rows, of the observed exposures with cross_sectional_observed_carry too. Before the warm-up ends, drop the rows before the first one with an eligible asset on the estimated and the observed exposures together, as cross_sectional_warmup does.
3. Neutralise the exposures of the rows that remain with cross_sectional_neutralise!.

Arguments

• pe: Cross-Sectional Factor Prior estimator.
• st: The state before the step.
• win: The carried rows followed by the rows of the step.
• n: Number of rows of the step.
• g0: Number of observations before the first row of win.

Returns

• rows::NamedTuple: nf and fam, the raw factor axis; the rows after the warm-up: Ms, X, Xl, obs, bw, mcap, amsk and emsk, with Xl and obs nothing without an observed factor; der, the derived series of every row of win and the observed exposures of its last pe.lag rows, or nothing; and xf, the factor list with the state of each Descriptor after the rows of the step.

Related

• cross_sectional_carry_fold
• cross_sectional_exposure_stage

cross_sectional_fold_refit(
    pe::CrossSectionalFactorPrior,
    st::CrossSectionalCarryState,
    families::Union{Nothing, AbstractVector{<:Pair}},
    seed::Integer
) -> Union{CrossSectionalCarryState{__T_buf, Nothing, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6, Base.RefValue{Int64}} where {__T_buf<:SampleBufferState, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed<:Integer, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6}, CrossSectionalCarryState{__T_buf, __T_win, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6, Base.RefValue{Int64}} where {__T_buf<:SampleBufferState, __T_win<:ReturnsResult, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed<:Integer, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6}}

Fits every observation that the carry fold of a Cross-Sectional Factor Prior carries after the warm-up.

This is the first fit, and the fit after a step where a factor comes alive and the regression estimator does not fold it, as cross_sectional_alive_folds answers, or that moves the automatic choice of a dropped member where cross_sectional_fold_move does not fold it. It runs the steps of the batch fit over the carried histories. The idiosyncratic variance and the factor prior fold the first seed observations as one block, and every later observation alone, so a SeedWindow cuts the rows that it cut at the first fit.

Algorithm

1. Build the Factor Family Basis of every observation with cross_sectional_family_basis, under families, or under the families of pe at the first fit, and name its dropped members with cross_sectional_dropped_names.
2. Take the observed factors of every fitted observation with cross_sectional_observed_block, which refuses an infinite observed return.
3. Run the regression passes over every fitted observation with cross_sectional_fold_regression, on the returns net of the observed factors.
4. Fold ve over the residuals with cross_sectional_fold_variance, and the factor prior over the factor returns of cross_sectional_fold_factor_returns with cross_sectional_fold_factors.

Arguments

• pe: Cross-Sectional Factor Prior estimator.
• st: The state, whose histories hold every observation after the warm-up.
• families: The families with their dropped members named, or nothing at the first fit.
• seed: Number of fitted observations to fold as one block.

Returns

• st::CrossSectionalCarryState: The state with every fitted output replaced.

Related

• cross_sectional_carry_fold
• prior

cross_sectional_fold_step(
    pe::CrossSectionalFactorPrior,
    st::CrossSectionalCarryState,
    families::Union{Nothing, AbstractVector{<:Pair}},
    m::Integer
) -> Union{Nothing, CrossSectionalCarryState{__T_buf, Nothing, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6, Base.RefValue{Int64}} where {__T_buf<:SampleBufferState, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed<:Integer, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6}, CrossSectionalCarryState{__T_buf, __T_win, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6, Base.RefValue{Int64}} where {__T_buf<:SampleBufferState, __T_win<:ReturnsResult, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed<:Integer, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6}}

Fits the new observations of a step of the carry fold of a Cross-Sectional Factor Prior, and folds them into its state.

Each output of a fitted observation reads no later observation, so the step fits the new observations alone and appends their outputs. A factor that the new observations mark and the fitted observations leave empty comes alive at the step. Its exposure is zero at every pair of positive weight of each fitted observation, so its column adds nothing to the fit of that observation, and the answer of least norm gives it a return of zero there, as cross_sectional_live_regression states. So the step keeps every fitted output, and regresses the new observations over the joined marks. It folds the factor prior again over every fitted factor return, with cross_sectional_step_factors, because the factor prior folds the factors that are not empty. The step answers nothing when the regression estimator does not fold a factor that comes alive, as cross_sectional_alive_folds answers.

Algorithm

1. Reduce the exposures of the new observations with cross_sectional_family_basis under families, and lag them behind the reduced exposures of the last lag observations.
2. Take the observed factors of the new observations with cross_sectional_observed_block, which refuses an infinite observed return.
3. Run the regression passes over the new observations with cross_sectional_fold_regression, under the marks of the fitted observations, on the returns net of the observed factors.
4. Fold ve over the new residuals with cross_sectional_fold_variance, and fold the factor prior with cross_sectional_step_factors.
5. Append every output to the state with cross_sectional_fold_append, and record the marks of the step.

Arguments

• pe: Cross-Sectional Factor Prior estimator.
• st: The state, whose histories hold the new observations as their last m rows.
• families: The families with their dropped members named, or nothing.
• m: Number of new observations.

Returns

• st::Option{CrossSectionalCarryState}: The state after the step, or nothing.

Related

• cross_sectional_carry_fold
• cross_sectional_fold_refit

cross_sectional_fold_histories(
    pe::CrossSectionalFactorPrior,
    st::CrossSectionalCarryState,
    rows::NamedTuple
) -> Union{CrossSectionalCarryState{__T_buf, Nothing, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6, Base.RefValue{Int64}} where {__T_buf<:SampleBufferState, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed<:Integer, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6}, CrossSectionalCarryState{__T_buf, __T_win, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6, Base.RefValue{Int64}} where {__T_buf<:SampleBufferState, __T_win<:ReturnsResult, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed<:Integer, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6}}

Appends the rows of a step to the histories that the carry fold of a Cross-Sectional Factor Prior carries.

Arguments

• pe: Cross-Sectional Factor Prior estimator.
• st: The state before the step.
• rows: The rows of the step after the warm-up, from cross_sectional_fold_rows.

Returns

• st::CrossSectionalCarryState: The state with the rows appended to its histories and its sums.

Related

• cross_sectional_carry_fold
• cross_sectional_fold_join

cross_sectional_carry_fold(
    pe::CrossSectionalFactorPrior,
    st::CrossSectionalCarryState,
    rd::ReturnsResult,
    gaps::Union{Nothing, AbstractVector{Bool}}
) -> Union{CrossSectionalCarryState{__T_buf, Nothing, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6, Base.RefValue{Int64}} where {__T_buf<:SampleBufferState, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed<:Integer, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6}, CrossSectionalCarryState{__T_buf, __T_win, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6, Base.RefValue{Int64}} where {__T_buf<:SampleBufferState, __T_win<:ReturnsResult, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, _Q, _R, _S, _T, _U, _V, _W, __T_seed<:Integer, _X, _Y, _Z, _Z1, _Z2, _Z3, _Z4, _Z5, _Z6}}

Folds the observations of a ReturnsResult into the carry fold of a Cross-Sectional Factor Prior.

This is the step of a prior that carries a CrossSectionalCarryState. It applies the rule of the carry fold: the exposures, the observed factors, the returns net of them, the regression and the idiosyncratic variance of a new observation read no later observation, so the step computes them for the new observations alone.

A batch fit drops an observation with a Shared Series gap at its entry, see shared_series_view. So the carry computes nothing for an observation that gaps marks: its panel rows, its histories and its lags read the observations that the batch fit keeps. The buffer buf keeps every observation with its mark, because the Fold Context of an optimiser holds one row for each observation, and the histories count their offset with buffer_fit_count.

Algorithm

1. Refuse returns data that no fold takes with assert_prior_fold_returns, and a panel that is absent or static. Take the Exogenous Series of the step with exogenous_step_kwargs, which refuses a step with no series when the prior reads it.
2. Append the returns, both masks, the implied volatilities, the series and the marks gaps of every observation of the step to the buffer buf. Take the observations with no mark with shared_series_step. When there is none, return the state with the new buffer.
3. Append the rows of those observations, their series and their implied volatilities to the carried panel rows with cross_sectional_window_append.
4. Compute the neutralised exposures, the observed factors and the derived series of the new observations with cross_sectional_fold_rows, and append the rows after the warm-up to the histories with cross_sectional_fold_histories, and their Descriptor scores with cross_sectional_carry_scores. Keep the panel rows and their derived series that cross_sectional_carry_rows names.
5. Before the first fit, fit every observation with cross_sectional_fold_refit once lag + 2 observations follow the warm-up, as the batch fit needs.
6. After it, apply the Choice Rule with cross_sectional_fold_choice. When the choice moves, fold the move into the state with cross_sectional_fold_move. Then fit the new observations with cross_sectional_fold_step, which folds a factor that comes alive. Fit every observation again with cross_sectional_fold_refit when either one answers nothing. A variance estimator pe.ve that does not fold, as supports_partial_fit answers, carries no state, so every step fits every observation again.
7. Bring the outputs that read the fit up to the fitted observations with cross_sectional_carry_outputs: the Return Forecast history that a slot reads, the rows of a Return Forecast that folds, and the standardised idiosyncratic returns. Each keeps the carried rows after a step of cross_sectional_fold_step, and makes every row again after a fit of every observation.

Arguments

• pe: Cross-Sectional Factor Prior estimator.
• st: The state before the step.
• rd: The returns result to use. Its pnl is a time-varying Asset Panel.
• gaps: One mark for each observation of rd, true at an observation with a Shared Series gap, or nothing.

Validation

• The rules of assert_prior_fold_returns.
• The rules of cross_sectional_window_append: every step carries iv or none does, under the ivpa of the first step.
• rd.pnl is a time-varying Asset Panel. An ArgumentError is thrown otherwise.
• The rules of exogenous_step_kwargs and of the block method of partial_fit! for a SampleBufferState: a prior that reads the Exogenous Series gets it at every step, under the names of the first step.
• The rules of every verb the algorithm names.

Returns

• st::CrossSectionalCarryState: The state after the step.

Related

• CrossSectionalCarryState
• partial_fit!
• prior

fold_prior_step(pe::CrossSectionalFactorPrior{<:Any, …, <:Option{<:CrossSectionalCarryState}},
                rd::ReturnsResult, gaps::Option{<:AbstractVector{Bool}})

Folds the observations of a ReturnsResult into a Cross-Sectional Factor Prior on its carry route.

This is the carry form of fold_prior_step, the fold of every prior that partial_fit! over a ReturnsResult takes.

An unwrapped prior follows the rule of the carry fold: it folds what folds and refits the rest. Its first step seeds a CrossSectionalCarryState, as the carry of EmpiricalPrior seeds its own state. Each step computes the Factor Exposures and the observed factors of the new observations from the carried panel rows, runs the regression on the new observations alone, and folds the idiosyncratic variance and the factor prior. A prior whose tree reads the Exogenous Series, as reads_exogenous_series answers, records it, so each step must bring it. cross_sectional_carry_fold states the step. The call with no data prior(pe) equals the batch fit over the folded observations up to rounding, until a SeedWindow or a PinnedChoice keeps a value of the first fit. A pinned choice is recorded in the state, and families stays as the caller wrote it.

Online seeds the cache with a SampleBufferState instead, and the prior refits through the refit method of fold_prior_step. The method narrows cache to the state of the carry fold, so a buffer reaches that route. The matrix form refuses through the generic method, because a matrix carries no Asset Panel, and the prior reads its Factor Exposures off one.

Algorithm

1. Seed an empty CrossSectionalCarryState when pe.cache is nothing, after the refusal of a member that holds a state, with assert_cold_carry_entry. Copy a state that is not the newest one of its lineage with cross_sectional_carry_own, because the step appends to its histories in place.
2. Fold rd and gaps with cross_sectional_carry_fold. The step succeeded, so record the state it returns as the newest one of its lineage in tip. A step that throws leaves the counter, and the next step of the same state writes the same spare rows.
3. Rebuild the prior with the state.

Arguments

• pe: Cross-Sectional Factor Prior estimator with no state, or with the state of its carry fold.
• rd: The returns result to use. Its pnl is a time-varying Asset Panel.
• gaps: One mark for each observation of rd, true at an observation with a Shared Series gap, or nothing.

Validation

• With pe.cache at nothing, no member of pe holds a partial-fit state. The carry reads every member as configuration, and keeps each state that it folds in its own state, so a state that the caller folded into a member would enter the first step, and the prior would no longer equal the batch fit. An ArgumentError is thrown otherwise, and it names the path of the state, see assert_cold_carry_entry.
• The rules of cross_sectional_carry_fold.

Returns

• pe::CrossSectionalFactorPrior: The prior, with its cache field set to the state after the last observation.

Related

• CrossSectionalFactorPrior
• CrossSectionalCarryState
• cross_sectional_carry_fold
• fold_prior_step
• Online
• prior
• AbstractChoiceRule