Prior partial fit (a)

A prior updates with partial_fit! in one of two ways, and the type of the state it carries shows which.

A prior that updates its moments exactly carries a PortfolioOptimisers.PriorCarryState. Its moments come from the incremental fits of its members. It stores the rows only because a LowOrderPrior holds the returns X for the scenario risk measures, and it reads no row to compute its result. A prior with no exact update carries a PortfolioOptimisers.SampleBufferState, which Online gives it, and it computes its result with its batch fit over the rows that the buffer kept. The same buffer stores the factor returns. PortfolioOptimisers.needs_factor_returns reads the members of the prior to decide whether the update stores them, so the update takes the same arguments as the batch fit.

A prior that stores the observations updates each member that has an incremental fit, and refits each member that has none over its own rows. You write the same estimator as for a batch fit.

A CrossSectionalFactorPrior that Online does not wrap carries a PortfolioOptimisers.CrossSectionalCarryState. A new observation changes no past exposure, regression or idiosyncratic variance, so each update computes them for the new observations alone, and its factor prior and idiosyncratic variance update exactly.

partial_fit!(pe, rd) updates a prior from a ReturnsResult, as prior(pe, rd) fits it. It reads the returns, the factor returns and the two masks of the panel of rd. It also reads the per-asset data of that panel when the prior reads them, as PortfolioOptimisers.reads_panel_fields answers. A prior with no exact update stores all of these, so it takes a narrower estimation universe and the per-asset data. A prior that updates its moments exactly takes the active mask alone, so it refuses an estimation mask that differs from the active mask.

PortfolioOptimisers.partial_fit! — Method
partial_fit!(state::PriorCarryState, x::VecNum; kwargs...)
partial_fit!(state::PriorCarryState, X::MatNum; dims::Int = 1, kwargs...)

Folds observations into the buffer that a PriorCarryState carries.

The method forwards to the fold of the buffer, with its keyword arguments. So a CoveragePolicy mask goes into the buffer beside the rows it describes, as it does for a SampleBufferState that an Online seeded. No factor observation reaches the buffer, because EmpiricalPrior, the one carrying prior, never reads one and drops it before the carry. The fold leaves the named-asset set as it is, because only prior(pe) with no data finds an asset to name.

Arguments

  • state: The carry state to fold into.
  • x: One observation, with one entry per asset.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.
  • kwargs...: Additional keyword arguments, such as active_mask, forwarded to the fold of the buffer.

Returns

  • state::PriorCarryState: The state after the last observation.

Related

source
PortfolioOptimisers.port_opt_view — Method
port_opt_view(
    x::PriorCarryState,
    i,
    args...
) -> PriorCarryState{_A, Set{Int64}} where _A

Slices a PriorCarryState to the selected assets.

The method slices the buffer as usual. The entries of the named-asset set are asset indices, and the slice renumbers the assets, so the method renumbers the set too. An asset that the state had already named stays named at its new index, and an asset that the slice removes leaves the set.

Algorithm

  1. Index the asset axis 1:N with i, giving sel, the old index of each asset that the slice keeps.
  2. Keep each new index j whose old index sel[j] is in x.named, giving named.
  3. Slice the buffer with port_opt_view, and return the new state.

Arguments

  • x: The state to slice.
  • i: Index, indices or Bool mask of the assets to keep.
  • args...: Additional positional arguments, forwarded to the slice of the buffer.

Returns

  • state::PriorCarryState: The state of the same observations over the selected assets.

Related

source
PortfolioOptimisers.merge_states — Method
merge_states(
    a::PriorCarryState,
    b::PriorCarryState
) -> PriorCarryState

Merges two PriorCarryState fitted on disjoint blocks into the state of the concatenated block.

The buffers merge by concatenation. The named-asset sets merge by union, because an asset that either state has already named needs no second notice.

Arguments

  • a: The state of the first block of observations.
  • b: The state of the second block of observations.

Returns

  • state::PriorCarryState: The state that a fold of the two blocks as one block gives.

Related

source
PortfolioOptimisers.prior — Method
prior(pe::AbstractPriorEstimator; kwargs...)

Reads a prior out of the state that its estimator carries, with no data matrix.

This is the method of prior with no data, and the entry point of every prior that folds. It reads the state and dispatches on its type:

  • A SampleBufferState means refit. The estimator has no recursion of its own, so the method runs the batch verb over the rows that the buffer kept. These are all the rows folded so far when the buffer has no cap, and the last max_history rows when an Online set one. The method drops the rows that the buffer marks with a Shared Series gap, with shared_series_buffer, as the batch fit over the same rows drops them at its entry. The batch verb receives the factor rows that the buffer recorded, through factor_buffer, or nothing where the buffer recorded none. The fold fixed that choice from the estimator tree.
  • A PriorCarryState means fold and carry. Each family that takes it has its own method of prior with no data.

Every method with no data records the source that pe states in the prior result, with stamp_returns_source, as the ReturnsResult method does. A consumer that receives that Result in place of the estimator then reads its frame.

Mathematical definition

Under a SampleBufferState:

\[\begin{align} \mathcal{P}_T &= \mathcal{P}(\mathbf{X}_{S},\, \mathbf{F}_{S},\, \mathbf{E}_{S},\, \mathcal{A}_{S})\,, \\ S &= \{t \in \{T-m+1, \ldots, T\} : g_t = 0\}\,, \\ m &= \min(T,\, M)\,. \end{align}\]

Where:

  • $\mathcal{P}_T$: Folded prior, the result that the estimator's batch verb gives with no data after the estimator folds the observations $\boldsymbol{x}_1, \ldots, \boldsymbol{x}_T$ in order, one at a time or in blocks of any size.
  • $\mathcal{P}(\cdot)$: Batch prior, the result that the estimator's batch verb gives on the matrices in the parentheses.
  • $\mathbf{X}_{S}$: The rows in $S$ of the returns matrix of the folded observations.
  • $\mathbf{F}_{S}$: The rows in $S$ of the factor returns matrix of the folded factor observations, absent when the fold recorded none.
  • $\mathbf{E}_{S}$: The rows in $S$ of the folded Exogenous Series, absent when the fold recorded none.
  • $\mathcal{A}_{S}$: The Asset Panel of the rows in $S$, with the folded Panel Fields and both folded masks, absent when the fold recorded no Panel Field.
  • $S$: The rows that the buffer holds and that have no Shared Series gap.
  • $g_t$: $1$ when the fold marked observation $t$ with a Shared Series gap, and $0$ otherwise.
  • $m$: Number of rows that the buffer holds.
  • $M$: Cap on the buffer, the max_history of the wrapping Online, and $\infty$ when it is nothing.
  • $T$: Number of observations.

Algorithm

  1. Read the state out of pe.cache with partial_fit_cache.
  2. Dispatch on the type of the state. Under a SampleBufferState, drop the rows of a Shared Series gap with shared_series_buffer, and run the batch verb over the sample_buffer and the factor_buffer of the result with buffer_prior, which gives the masks, the Panel Fields and the Exogenous Series that the buffer records, and the caller's keyword arguments.
  3. Under a SampleBufferState, record the Shared Series cut with shared_series_record: the marks of the dropped observations of the buffer, and the row key of the factor model block. The batch verb counts the positions among the rows that it kept, so the record maps them back through the rows of buffer_kept_rows. The batch verb also counts them from the first row of the buffer, and a capped buffer does not record how many rows it dropped, so buffer_row_key removes them under a cap.

Arguments

  • pe: Prior estimator carrying a state.
  • kwargs...: Additional keyword arguments, forwarded to the batch verb.

Validation

  • pe.cache is not nothing. An ArgumentError that names the verb which fills the field is thrown otherwise.
  • The rules of shared_series_buffer.

Returns

  • pr::AbstractPriorResult: The prior that the state of the estimator gives.

Related

source
PortfolioOptimisers.partial_fit! — Function
partial_fit!(
    pe::AbstractPriorEstimator,
    X::Union{AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}, AbstractVector{<:Union{Number, AbstractJuMPScalar}}};
    ...
) -> Any
partial_fit!(
    pe::AbstractPriorEstimator,
    X::Union{AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}, AbstractVector{<:Union{Number, AbstractJuMPScalar}}},
    F::Union{Nothing, AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}, AbstractVector{<:Union{Number, AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask
) -> Any

Folds observations, and the factor observations beside them, into the sample buffer that a prior carries.

This is the refit route of the prior family. Every prior that carries a SampleBufferState reaches this method, which has the arity of the prior's own batch verb, prior(pe, X, F). The estimator tree decides what the fold does with F, through needs_factor_returns, and the fold does what the batch verb does with the same argument:

  • true. The tree holds a factor leaf. The fold refuses a call without F with the IsNothingError of assert_factor_returns, which names F, before it appends a row. A call with F records it.
  • false. The tree never reads F, so the fold drops it, as prior(EmpiricalPrior(), X, F) drops it, and the buffer records the rows alone.
  • nothing. The tree does not say, so the buffer records F when the call gives it and not otherwise, as the batch verb of an optional-argument prior does.

The buffer fixes what it records at its first append and refuses a mixture. So a run that gives F at one step and not at the next is refused at that step with an error.

A prior that reads Panel Fields, as reads_panel_fields answers, refuses this form, because a matrix carries no Asset Panel. It folds a ReturnsResult instead.

Algorithm

  1. Refuse a prior that reads Panel Fields. Answer needs_factor_returns for the tree, and apply the answer to F with fold_factor_argument, giving the F that the buffer records.
  2. Read the buffer out of pe.cache with assert_sample_buffer, giving state.
  3. Fold a matrix, its factor block and its masks into state through the block method of partial_fit!, or a vector, its factor observation and its masks through the single-observation method.
  4. Rebuild the prior with its cache set to the new state.
  5. Apply the Choice Rule of the regression of the prior over the rows of state with pin_buffer_choice, and return the prior.

Arguments

  • pe: The prior whose buffer the method folds forward.
  • X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is one observation across the assets, and dims then has no effect.
  • F: The factor observations, in the orientation of X, or nothing.
  • dims: Dimension along which the computation runs.
  • active_mask: The active mask of the block, of the shape of X, or with one entry per asset when X is one observation, or nothing.
  • estimation_mask: The estimation mask, on the same terms as active_mask.

Validation

  • reads_panel_fields(pe) is false. An ArgumentError is thrown otherwise.
  • F is not nothing when needs_factor_returns(pe) === true. An IsNothingError is thrown otherwise.
  • pe carries a SampleBufferState. An ArgumentError is thrown otherwise.
  • Every condition that the fold of the buffer checks.

Returns

  • pe: The prior, with its cache field set to the buffer after the last observation.

Related

source
PortfolioOptimisers.partial_fit! — Method
partial_fit!(pe::AbstractPriorEstimator, rd::ReturnsResult)

Folds the observations of a ReturnsResult into a prior. This is the fold of every prior.

It mirrors prior(pe, rd). It reads the returns, the factor returns and both masks of rd, and the Panel Fields of rd.pnl when the tree of the prior reads them. The online step of an optimiser folds its prior through this method. The matrix form, partial_fit!(pe, X, F; active_mask, estimation_mask), stays for a prior that reads no Panel Field.

The type of the prior selects the route, and each route states what it can honour. This method is the refit route, which every prior that carries a SampleBufferState takes. The buffer records what the batch fit reads, so the route honours all of it:

  • The estimation mask, beside the active mask. No panel, or a static panel, gives no mask.
  • The Panel Fields, when reads_panel_fields answers true. The buffer records them, and the call with no data gives the batch verb the panel that sample_buffer_panel rebuilds. For any other prior the call refuses a Panel Field with an error that names it, because the buffer has no slot that the prior reads.
  • The Exogenous Series, when reads_exogenous_series answers true. The buffer records every column of it with its names, and the call with no data gives them to the batch verb. For any other prior the Fold Context keeps the series, and the prior's buffer records none.

The carry of EmpiricalPrior has a route of its own, because its exact folds take no estimation mask. HighOrderPriorEstimator and BlackLittermanPrior forward rd to the prior they embed. CrossSectionalFactorPrior has a route of its own, which takes the carry fold when the prior has no buffer and applies its Choice Rule after the fold of a refit. fold_prior_step holds each route.

A batch fit drops an observation with a gap in a Shared Series that it reads, with shared_series_view. A prior reads the factor returns as a Shared Series when reads_factor_returns answers true. The step keeps the observation and marks it, because the buffer of a prior and the Fold Context of its optimiser hold one row each for every observation. Every fit through the buffer then drops it, and a carry fold leaves it out of its running moments. So the call with no data gives the batch fit over the same observations.

Algorithm

  1. Read the source that the prior states with returns_source, giving x_src.
  2. Mark the observations with a Shared Series gap with shared_series_gaps: a gap of a vector benchmark under x_src = ExcessSource(), and a gap of the factor returns when the prior reads them. Mark a gap of a matrix benchmark in the estimation mask under ExcessSource() with benchmark_estimation_mask, as the batch fit marks it.
  3. Take the observations that the prior folds with prior_step_returns under x_src: rd, or rd with X - B in place of X under ExcessSource().
  4. Fold them with their marks with fold_prior_step.

Arguments

  • pe: The prior whose buffer the method folds forward.
  • rd: The returns result to use.

Validation

Returns

  • pe: The prior, with its cache field set to the buffer after the last observation.

Related

source
PortfolioOptimisers.partial_fit! — Function
partial_fit!(pe::EmpiricalPrior{<:Any, <:Any, Nothing, <:Any, <:Any,
                                <:Option{<:PriorCarryState}}, X, F = nothing; kwargs...)
partial_fit!(pe::EmpiricalPrior{<:Any, <:Any, <:Number, <:Any, <:Any,
                                <:Option{<:PriorCarryState}}, X, F = nothing; kwargs...)

Folds observations into an EmpiricalPrior, which folds its moments and carries its rows.

pe.me and pe.ce fold exactly, so a step of the prior is their two steps and an append to the carried rows. The cost of a step grows with the square of the number of assets, and does not grow with the number of observations already folded. The cost of a refit from a buffer grows with that number too. This difference is the reason the carry route exists. A call with no data that is written as prior(pe, sample_buffer(pe.cache)) passes every parity test and discards both exact folds, so a review must refuse that form.

The two methods differ in one line. The method with no horizon folds the observation as it is. The horizon method folds log1p of the observation, because the batch horizon method fits its moments on log returns. It carries the observation itself, because the LowOrderPrior it returns carries the arithmetic returns that the caller passed. A buffer of log rows would change the matrix that every scenario risk measure reads.

A matrix of observations can carry the marks gaps of its Shared Series gaps, see shared_series_gaps. A batch fit drops a marked observation, so the two members fold the unmarked observations alone, and the carry state keeps every observation with its mark, because the Fold Context of an optimiser holds one row for each observation. The call with no data reads the unmarked rows, see shared_series_buffer.

A member that does not fold, such as a SemiMoment covariance or a composite whose mp.alg reads the sample, stays as it is under fold_member, and prior(pe) with no data refits it over the carried rows. So a caller writes the estimator that they would write in batch, and no member keeps a second copy of the sample.

The fold takes a factor observation and drops it, because prior(EmpiricalPrior(), X, F) declares F and never reads it. The fold has the arity of the batch verb, so an outer estimator that passes F down its tree meets no MethodError here. needs_factor_returns answers false for this estimator, so no estimator above it keeps a factor row for it.

Algorithm

  1. Orient a matrix X and a matrix iv to observations × assets with dims_oriented, transposing them when dims == 2. Under a matrix X, take its unmarked observations and their active mask with shared_series_member_rows.
  2. Under the horizon method, take log1p of the observations of step 1.
  3. Fold pe.me and pe.ce through fold_carry_members, on the observations of step 2 under the horizon method and on the observations of step 1 otherwise. At the first step, when pe.cache is nothing, it first refuses a member that holds a state.
  4. Append every arithmetic observation, with iv, ivpa and the marks gaps, to the carry state through fold_carry, and return the estimator with its members and its cache field rebound. No member that folds reads iv, and a member that does not fold reads it from the carried buffer at the call with no data.

Arguments

  • pe: Empirical prior estimator.
  • X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is one observation across the assets.
  • F: The factor observations beside X, or nothing. The fold drops them, as the batch verb does.
  • dims: Dimension along which the computation runs.
  • iv: The implied volatilities of the observations, oriented as X is, or nothing. The carry state records them.
  • ivpa: The implied volatility risk premium adjustment, or nothing. The carry state pins it.
  • gaps: Under a matrix X, one mark for each observation, true at an observation with a Shared Series gap, or nothing. The carry state records them.
  • kwargs...: Additional keyword arguments, forwarded to the two members and to the carry state.

Validation

  • dims in (1, 2).
  • Every condition that the fold of pe.me or pe.ce checks. A member that holds observation weights refuses the fold with an ArgumentError that names the member, because a weight vector reweights every past observation when a new one arrives.
  • At the first step, no member holds a partial-fit state, see assert_cold_carry_entry. The carried rows start empty, so a warm member would read rows that the carried rows, the refitted members and X do not read. An ArgumentError that names the path of the state is thrown otherwise.

Returns

  • pe: The estimator, with its members folded and its cache field rebound.

Related

source