GitHub

Cross-Sectional Forecast History Carry: private API

The carry fold of a CrossSectionalFactorPrior carries the return forecast history when a slot of the prior reads it. The functions below decide when the state carries the history, bring it up to the fitted observations at each step, and give it to the slots. The fifth function brings the standardised idiosyncratic returns, which the call with no data reads, up to the fitted observations by the same rule, and the sixth brings the input of the idiosyncratic correlation up with them. The next two give the rows that a covariance estimator reads from the standardised returns with no fill, and the idiosyncratic covariance from the covariance of those rows. The batch fit and the carry fold share both. The next functions carry a return forecast that computes its history one observation at a time: its returns data over a set of panel rows, its Descriptor scores, its rows, and the rows that a step hands it. The step under the fit of the orthogonal part of the forecast is public, and the public page documents it. The last two bring every output that reads the fit up to the fitted observations.

PortfolioOptimisers.cross_sectional_forecast_refits — Function
cross_sectional_forecast_refits(rfe::AbstractReturnForecastEstimator)
cross_sectional_forecast_refits(rfe::Union{FixedWeightedReturnForecast,
                                           ExpWeightedReturnForecast})

Answers whether the history of a Return Forecast Estimator comes from a refit at each observation.

FixedWeightedReturnForecast and ExpWeightedReturnForecast publish their history in the Result of one fit, so the call with no data of the carry fold reads it there and the state carries no row. Every other member answers true. A member that publishes a history and answers true costs a fit at each step unless it folds its rows, as folds_forecast_rows states, and the call with no data still reads the history of its Result.

Arguments

  • rfe: Return Forecast Estimator.

Returns

  • refits::Bool: true when the history comes from a refit.

Related

source
PortfolioOptimisers.cross_sectional_carries_history — Function
cross_sectional_carries_history(
    pe::CrossSectionalFactorPrior
) -> Bool

Answers whether the carry fold of a Cross-Sectional Factor Prior carries the Return Forecast history.

It does when a slot of the prior answers true to reads_forecast_history, and the history of its Return Forecast comes from a refit, as cross_sectional_forecast_refits answers.

Arguments

  • pe: Cross-Sectional Factor Prior estimator.

Returns

  • carries::Bool: true when the state carries the history.

Related

source
PortfolioOptimisers.cross_sectional_forecast_history — Function
cross_sectional_forecast_history(
    rf::AbstractReturnForecastResult,
    rfe::AbstractReturnForecastEstimator,
    rd::ReturnsResult,
    csfm::CrossSectionalFactorModel,
    H::Union{Nothing, AbstractMatrix{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}}}
) -> AbstractMatrix{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}}

Returns the Return Forecast history that a Cross-Sectional Factor Prior hands a slot that reads it.

The batch fit and the call with no data of the carry fold both take the history here, from the Result rf that the prior fitted on the whole sample. A Result that carries a history gives it, so the member is fitted once. Otherwise the batch fit refits the member at each observation with forecast_history_refit, and the carry fold appends the forecast of the last observation to the rows H that it carried.

Arguments

  • rf: The Return Forecast Result of the whole sample.
  • rfe: Return Forecast Estimator.
  • rd: The returns result to use. It is the returns data the member was fitted on.
  • csfm: The factor-model block the member was fitted on.
  • H: The rows that the carry fold carries, at every observation of the block but the last, or nothing.

Validation

Returns

  • hist::MatNum: Return Forecast history, observations × assets, on the rows of the block.

Related

source
PortfolioOptimisers.cross_sectional_carry_history — Function
cross_sectional_carry_history(
    pe::CrossSectionalFactorPrior,
    st::CrossSectionalCarryState,
    keep::Bool
) -> CrossSectionalCarryState

Brings the Return Forecast history of the carry fold of a Cross-Sectional Factor Prior up to its fitted observations.

The row of a block observation t is the forecast of the member fitted on the returns data through that observation and on the block rows 1:t, as forecast_history_refit fits it. No row reads a later observation, so a step that fits the new observations alone keeps the rows it carried and fits one row for each new observation. The state keeps no row for the last observation: the call with no data fits the member there anyway, and appends that forecast with cross_sectional_forecast_history.

Algorithm

  1. Return st unchanged when the state carries no history, as cross_sectional_carries_history answers. When the member folds, as folds_forecast_rows answers, its rows at every fitted observation but the last are the history, so return the state with them. The row of an observation is the forecast that the member publishes there, which is the forecast of a fit through that observation.
  2. Keep the carried rows when keep is true, and none otherwise.
  3. Build the block of the fitted observations from the state, with cross_sectional_carry_append and the returns data of cross_sectional_carry_forecast_returns. The member reads no idiosyncratic covariance, so the block carries none. Take the member of the Orthogonal Forecast Fit pe.ofit with orthogonal_forecast_member, the member whose history the batch fit reads.
  4. Fit that member at each block row after the kept rows and before the last, on the returns data and the block that forecast_history_block cuts to that row.
  5. Append the new rows to the kept rows.

Arguments

  • pe: Cross-Sectional Factor Prior estimator.
  • st: The state after the step, after its first fit.
  • keep: Whether the step fitted the new observations alone, so that the carried rows stay.

Validation

Returns

  • st::CrossSectionalCarryState: The state with its history brought up to its fitted observations.

Related

source
PortfolioOptimisers.cross_sectional_carry_standardised — Function
cross_sectional_carry_standardised(
    pe::CrossSectionalFactorPrior,
    st::CrossSectionalCarryState,
    stepped::Bool
) -> CrossSectionalCarryState

Brings the standardised idiosyncratic returns of the carry fold of a Cross-Sectional Factor Prior up to its fitted observations.

The standardised return of an observation reads the residual, the variance and the active mask of that observation alone, and the fill reads the other assets of the same observation. A step that fits the new observations alone changes no past residual and no past variance, so the state keeps its rows and appends the rows of the new observations. A step that fits every observation again can change every row, so it makes every row again. The call with no data then reads the rows as they are, and its cost does not grow with the history. The residuals of the step take the Leverage-One Mask of the prior through leverage_one_residuals before the standardisation, as the batch fit does, so the carry fold equals the batch fit.

Under a threshold th above zero, the idiosyncratic correlation reads the rows with no fill, and cross_sectional_carry_correlation brings them up. A ce that folds, as supports_partial_fit answers, folds the rows of the step with partial_fit!, under the rule of cross_sectional_correlation_rows for the gaps. So the state keeps the folded ce and no row with no fill, and the call with no data reads the correlation from the state of ce. A step that fits every observation again folds every row into the ce of the prior again. A ce that does not fold makes the state keep the rows with no fill, and the call with no data estimates the correlation over them again.

Arguments

  • pe: Cross-Sectional Factor Prior estimator.
  • st: The state after the fit of the step.
  • stepped: true when the step fitted the new observations alone, false when it fitted every observation.

Returns

  • st::CrossSectionalCarryState: The state with S at every fitted observation. When the threshold th of the prior is not zero, it also holds the folded ce, or Sc at every fitted observation when ce does not fold. A state with no fit is returned as it is.

Related

source
PortfolioOptimisers.cross_sectional_carry_correlation — Function
cross_sectional_carry_correlation(
    pe::CrossSectionalFactorPrior,
    Sc0,
    ce0,
    E::AbstractMatrix{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}},
    V::AbstractMatrix{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}},
    A::AbstractMatrix{<:Bool}
) -> Union{NamedTuple{(:Sc, :ce), <:Tuple{Any, Nothing}}, NamedTuple{(:Sc, :ce), <:Tuple{Nothing, Any}}}

Brings the input of the idiosyncratic correlation of the carry fold of a Cross-Sectional Factor Prior up to its fitted observations.

cross_sectional_carry_standardised calls it with the rows of a step. Under a threshold th above zero, the correlation reads the standardised idiosyncratic returns with no fill.

Algorithm

  1. Return no rows and no estimator when th is zero, because the correlation reads no residual.
  2. Standardise the rows of the step with no fill with cross_sectional_standardised_residuals.
  3. When pe.ce does not fold, as supports_partial_fit answers, append the rows to Sc0. The call with no data estimates the correlation over them again.
  4. Otherwise, fold the rows into ce0 with partial_fit!, under the rule of cross_sectional_correlation_rows for the gaps, and keep no row. The call with no data reads the correlation from the state of ce.

Arguments

  • pe: Cross-Sectional Factor Prior estimator.
  • Sc0: The rows with no fill that the state carries, or nothing.
  • ce0: The folded ce that the state carries, or pe.ce at a fit of every observation.
  • E: The idiosyncratic returns of the rows of the step, under the Leverage-One Mask of the prior.
  • V: The idiosyncratic variances of the rows of the step.
  • A: The active mask of the rows of the step.

Returns

Related

source
PortfolioOptimisers.cross_sectional_correlation_rows — Function
cross_sectional_correlation_rows(
    ce::CovarianceEstimator,
    S::AbstractMatrix{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}},
    amsk::AbstractMatrix{<:Bool}
) -> Union{NamedTuple{(:X, :kw), <:Tuple{AbstractMatrix{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}}, NamedTuple{(:active_mask,), <:Tuple{AbstractMatrix{<:Bool}}}}}, NamedTuple{(:X, :kw), <:Tuple{Any, @NamedTuple{}}}}

Return the rows and the keyword arguments with which a covariance estimator reads the standardised idiosyncratic returns.

The batch fit passes them to Statistics.cov, and the carry fold passes the rows of each step to partial_fit!. So the two routes read the gaps of the rows by one rule.

Algorithm

  1. Get fv from ce with gap_fill_value. fv is the value that ce gives a gapped cell of S. A cell of S is non-finite where the asset is inactive, and where it is active with no finite standardised return: a missing return, or a variance in its warm-up.
  2. If fv is finite, write it over every non-finite cell of a copy of S, and give no keyword argument. The fallback fv is zero, which is the mean of a standardised series.
  3. If fv is not finite, give S as it stands, with amsk as the active_mask. A gap-aware ce then freezes the block of an inactive asset and does not decay it, and it takes an active non-finite cell as a holiday.

Arguments

  • ce: Covariance estimator of the standardised idiosyncratic returns.
  • S: Standardised idiosyncratic returns with no fill, observations × assets.
  • amsk: The active mask of the rows of S, observations × assets.

Returns

  • (; X, kw)::NamedTuple: The rows X that ce reads, and the keyword arguments kw that go with them.

Related

source
PortfolioOptimisers.cross_sectional_thresholded_covariance — Function
cross_sectional_thresholded_covariance(
    th::Real,
    pdm::Union{Nothing, AbstractPosdefEstimator},
    C::AbstractMatrix{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}},
    ev::AbstractVector{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}}
) -> Any

Return the idiosyncratic covariance of the latest observation from the covariance of the standardised idiosyncratic returns.

cross_sectional_idiosyncratic_covariance states the definition. The batch fit and the carry fold estimate C by different routes, and both reach this function.

Algorithm

  1. Convert C to the correlation R.
  2. Set to zero every entry of R off the diagonal whose magnitude does not exceed th, and set the diagonal to one. This step also sets a non-finite correlation to zero.
  3. Rescale R by the latest idiosyncratic volatilities, giving D. Copy the lower triangle of D into the upper one, so D is exactly symmetric.
  4. Make the block of D over the assets with a finite variance positive definite with posdef!.

Arguments

  • th: The correlation threshold, above zero.
  • pdm: Positive definite matrix estimator, or nothing.
  • C: The covariance of the standardised idiosyncratic returns, assets × assets.
  • ev: The latest idiosyncratic variances, one per asset.

Returns

  • D::MatNum: The idiosyncratic covariance.

Related

source
PortfolioOptimisers.cross_sectional_forecast_window — Function
cross_sectional_forecast_window(
    pe::CrossSectionalFactorPrior,
    win::ReturnsResult,
    der::Union{Nothing, NamedTuple}
) -> Union{ReturnsResult{_A, _B, Nothing, Nothing, Nothing, Nothing, _C, _D, Nothing, _E, _F, AssetPanel{Vector{T}, Nothing, Nothing}} where {_A, _B, _C, _D, _E, _F, T<:AbstractPanelField}, ReturnsResult{_A, _B, Nothing, Nothing, Nothing, Nothing, _C, _D, Nothing, _E, _F, AssetPanel{Vector{T}, var"#s185", var"#s1851"}} where {_A, _B, _C, _D, _E, _F, T<:AbstractPanelField, var"#s185"<:AbstractMatrix{Bool}, var"#s1851"<:AbstractMatrix{Bool}}}

Returns the returns data that the Return Forecast of the carry fold of a Cross-Sectional Factor Prior scores its Descriptors over.

The batch fit gives the forecast the returns data that the estimated members read: the returns with the benchmark weights on the panel, and under observed factors the returns net of them. The benchmark weights of a row read that row alone, so the function builds them over the rows that it is given.

Arguments

  • pe: Cross-Sectional Factor Prior estimator.
  • win: Panel rows, with their returns, Asset Panel and Exogenous Series.
  • der: The derived series of the rows of win, (; Xn, Xl), or nothing without an observed factor.

Returns

  • rd::ReturnsResult: The returns data of the forecast over the rows of win.

Related

source
PortfolioOptimisers.cross_sectional_carry_scores — Function
cross_sectional_carry_scores(
    pe::CrossSectionalFactorPrior,
    st::CrossSectionalCarryState,
    win::ReturnsResult,
    rows::NamedTuple,
    n::Integer
) -> CrossSectionalCarryState

Appends the Descriptor scores of the new observations of a step to the carry fold of a Cross-Sectional Factor Prior.

A Return Forecast that computes its history one observation at a time, as folds_forecast_rows answers, reads its scores at each fitted observation. A score reads the last lookback panel rows of its observation and no factor-model block, so a step computes the scores of its new observations alone, from the rows that it carries. A Descriptor that carries a state reads the new observations alone, as descriptor_carry folds them, so it adds no row: the state carries the Descriptor Scores with the state of each such Descriptor. A fit of every observation then reads the scores that the state carries, and no panel row before the carried ones. A member that trains on rows before the block also reads the last rows of the warm-up that cross_sectional_forecast_lead counts, so the state keeps their scores before the scores of the first row of the histories.

Algorithm

  1. Return st unchanged when the forecast does not fold.
  2. Take the Descriptor Scores that the state carries, or the ones of the forecast at the first step. Keep the last carry_lookback(ds.descriptors) + n - 1 rows of win and of its derived series, as carry_lookback counts them, and build the returns data of the forecast over them with cross_sectional_forecast_window.
  3. Fold the n rows of the step into the Descriptors that carry a state with descriptor_carry.
  4. Count the rows of the step to score: the m observations after the warm-up and the last a rows of the warm-up before them, as cross_sectional_forecast_lead counts them, up to n. When the histories hold a row before the step, every row of the step follows the warm-up, so the count is m. When the count is zero, return the state with the folded Descriptor Scores alone.
  5. Score the Descriptors of the forecast over those rows with descriptor_panel_scores, with the Descriptors that the fold reads off the states, and keep the counted rows.
  6. Append them to the scores that the state carries with cross_sectional_fold_append, and keep the last a rows before the histories and every row of the histories with cross_sectional_window_trim. Keep the folded Descriptor Scores.

Arguments

  • pe: Cross-Sectional Factor Prior estimator.
  • st: The state, with the rows of the step appended to its histories.
  • win: The carried rows followed by the rows of the step.
  • rows: The rows of the step, as cross_sectional_fold_rows returns them. Its Ms holds the m new observations after the warm-up, the last rows of win, and its der the derived series of every row of win, or nothing without an observed factor.
  • n: Number of rows of the step, the last rows of win.

Returns

  • st::CrossSectionalCarryState: The state with the scores of the new observations appended, and the Descriptor Scores after the rows of the step in fds.

Related

source
PortfolioOptimisers.cross_sectional_carry_forecast — Function
cross_sectional_carry_forecast(
    pe::CrossSectionalFactorPrior,
    st::CrossSectionalCarryState,
    stepped::Bool
) -> CrossSectionalCarryState

Brings the Return Forecast rows of the carry fold of a Cross-Sectional Factor Prior up to its fitted observations.

A Return Forecast that computes its history one observation at a time, as folds_forecast_rows answers, computes the rows of its new observations from its fold state and from the scores and the block at those observations and at the forecast_target_gap observations before them. A step that fits the new observations alone changes no past row of the block, so the state keeps its rows and its fold state, and computes the rows of the new observations. A step that fits every observation again can change every row of the block, so it computes every row again from an empty fold state, from the scores that the state carries.

Algorithm

  1. Return st unchanged when the forecast does not fold, or before the first fit.
  2. Keep the carried rows and the fold state when stepped is true, and none otherwise. Take the fitted observations after the kept rows, and the last forecast_target_gap(pe.rfe) kept rows before them, or every kept row when there are fewer.
  3. Build the block of those observations from the state, with cross_sectional_carry_append. Take the member of the Orthogonal Forecast Fit pe.ofit with orthogonal_forecast_member.
  4. Compute the rows of those observations and the new fold state with orthogonal_forecast_step, from a copy of the scores that the state carries at the rows of cross_sectional_forecast_rows and from the kept fold state. A member that trains on the rows before the block reads them too.
  5. Append the rows of the new observations to the kept rows, and keep the new fold state.

Arguments

  • pe: Cross-Sectional Factor Prior estimator.
  • st: The state after the fit of the step.
  • stepped: true when the step fitted the new observations alone, false when it fitted every observation.

Validation

Returns

  • st::CrossSectionalCarryState: The state with its Return Forecast rows at every fitted observation, and the fold state of the forecast after them.

Related

source
PortfolioOptimisers.cross_sectional_forecast_rows — Function
cross_sectional_forecast_rows(rfe, r::AbstractUnitRange, k::Integer, g::Integer,
                              a::Integer)
cross_sectional_forecast_rows(rfe::TargetReturnForecast, r::AbstractUnitRange,
                              k::Integer, g::Integer, a::Integer)

Returns the rows of the carried Descriptor scores that a step of the carry fold of a Cross-Sectional Factor Prior hands its Return Forecast.

The step hands the member its new observations, and before them the observations whose target matures at the step: the last g observations that the member read, or all of them when it read fewer. The scores start a rows before the histories after the warm-up, as cross_sectional_forecast_lead counts them, so row t of the histories is row a + t of the scores.

Algorithm

The method that Julia selects is the algorithm.

  1. A member that reads the block alone: the fitted observations after the first k, and the last g of the first k, as rows of the scores.
  2. TargetReturnForecast under whole_history: the member also trains on a row before the block whose forward window reaches into the block, so its observations are the rows of the scores. Return the rows after the first k fitted observations, and the g rows before them, or every row before them when there are fewer. Without whole_history, method 1.

Arguments

  • rfe: The Return Forecast Estimator that the step fits.
  • r: The fitted observations, as rows of the histories after the warm-up.
  • k: Number of fitted observations that the state keeps.
  • g: The observations that a target takes to mature, as forecast_target_gap answers.
  • a: Number of rows of the warm-up that the scores hold before the histories.

Returns

  • rows::AbstractUnitRange: Rows of the carried scores.

Related

source
PortfolioOptimisers.cross_sectional_forecast_lead — Function
cross_sectional_forecast_lead(rfe, lag::Integer)
cross_sectional_forecast_lead(rfe::TargetReturnForecast, lag::Integer)

Returns the number of rows of the Descriptor warm-up whose scores the carry fold of a Cross-Sectional Factor Prior keeps for its Return Forecast.

The histories of the carry start lag rows before the first fitted observation. A member that trains on a row before the block trains on the last forecast_target_gap rows before it, because their forward windows reach into the block. When the gap is longer than lag, the first of those rows are rows of the warm-up, and the histories hold no such row. So the state keeps their scores, once, before the scores of the first row of the histories.

Algorithm

The method that Julia selects is the algorithm.

  1. A member that reads the block alone trains on no row before it. Return zero.
  2. TargetReturnForecast under whole_history: return forecast_target_gap(rfe) - lag, or zero when the gap is not longer than lag. Without whole_history, method 1.

Arguments

  • rfe: The Return Forecast Estimator that the step fits.
  • lag: The lag of the exposures of the prior, pe.lag.

Returns

  • a::Integer: The number of rows of the warm-up.

Related

source
PortfolioOptimisers.cross_sectional_carry_outputs — Function
cross_sectional_carry_outputs(
    pe::CrossSectionalFactorPrior,
    st::CrossSectionalCarryState,
    stepped::Bool
) -> CrossSectionalCarryState

Brings the outputs of the carry fold of a Cross-Sectional Factor Prior that read the fit up to its fitted observations.

Algorithm

  1. Bring the Return Forecast rows up with cross_sectional_carry_forecast.
  2. Bring the Return Forecast history that a slot reads up with cross_sectional_carry_history. It reads the rows of step 1 when the member folds.
  3. Bring the standardised idiosyncratic returns up with cross_sectional_carry_standardised.

Arguments

  • pe: Cross-Sectional Factor Prior estimator.
  • st: The state after the fit of the step.
  • stepped: true when the step fitted the new observations alone, false when it fitted every observation.

Returns

  • st::CrossSectionalCarryState: The state with the three outputs at every fitted observation.

Related

source
PortfolioOptimisers.cross_sectional_carry_forecast_returns — Function
cross_sectional_carry_forecast_returns(
    pe::CrossSectionalFactorPrior,
    st::CrossSectionalCarryState
) -> Any

Returns the returns data that the Return Forecast of the carry fold of a Cross-Sectional Factor Prior reads at the call with no data.

The batch fit gives the forecast the returns data that the estimated members read: the returns with the benchmark weights on the panel, and under observed factors the returns net of them. A forecast that reads the panel at the call with no data makes the state carry every row, so the carried rows and their derived series give that returns data, through cross_sectional_forecast_window. Any other forecast reads no row, and gets the carried rows.

Arguments

  • pe: Cross-Sectional Factor Prior estimator.
  • st: The state of its carry fold.

Returns

  • rd::ReturnsResult: The returns data of the forecast.

Related

source