Prior partial fit (a)
A prior updates with partial_fit! in one of two ways, and the type of the state it carries shows which.
A prior that updates its moments exactly carries a PortfolioOptimisers.PriorCarryState. Its moments come from the incremental fits of its members. It stores the rows only because a LowOrderPrior holds the returns X for the scenario risk measures, and it reads no row to compute its result. A prior with no exact update carries a PortfolioOptimisers.SampleBufferState, which Online gives it, and it computes its result with its batch fit over the rows that the buffer kept. The same buffer stores the factor returns. PortfolioOptimisers.needs_factor_returns reads the members of the prior to decide whether the update stores them, so the update takes the same arguments as the batch fit.
A prior that stores the observations updates each member that has an incremental fit, and refits each member that has none over its own rows. You write the same estimator as for a batch fit.
A CrossSectionalFactorPrior that Online does not wrap carries a PortfolioOptimisers.CrossSectionalCarryState. A new observation changes no past exposure, regression or idiosyncratic variance, so each update computes them for the new observations alone, and its factor prior and idiosyncratic variance update exactly.
partial_fit!(pe, rd) updates a prior from a ReturnsResult, as prior(pe, rd) fits it. It reads the returns, the factor returns and the two masks of the panel of rd. It also reads the per-asset data of that panel when the prior reads them, as PortfolioOptimisers.reads_panel_fields answers. A prior with no exact update stores all of these, so it takes a narrower estimation universe and the per-asset data. A prior that updates its moments exactly takes the active mask alone, so it refuses an estimation mask that differs from the active mask.
PortfolioOptimisers.partial_fit! — Method
partial_fit!(state::PriorCarryState, x::VecNum; kwargs...)
partial_fit!(state::PriorCarryState, X::MatNum; dims::Int = 1, kwargs...)Folds observations into the buffer that a PriorCarryState carries.
The method forwards to the fold of the buffer, with its keyword arguments. So a CoveragePolicy mask goes into the buffer beside the rows it describes, as it does for a SampleBufferState that an Online seeded. No factor observation reaches the buffer, because EmpiricalPrior, the one carrying prior, never reads one and drops it before the carry. The fold leaves the named-asset set as it is, because only prior(pe) with no data finds an asset to name.
Arguments
state: The carry state to fold into.x: One observation, with one entry per asset.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.dims: Dimension along which the computation runs.kwargs...: Additional keyword arguments, such asactive_mask, forwarded to the fold of the buffer.
Returns
state::PriorCarryState: The state after the last observation.
Related
PortfolioOptimisers.port_opt_view — Method
port_opt_view(
x::PriorCarryState,
i,
args...
) -> PriorCarryState{_A, Set{Int64}} where _A
Slices a PriorCarryState to the selected assets.
The method slices the buffer as usual. The entries of the named-asset set are asset indices, and the slice renumbers the assets, so the method renumbers the set too. An asset that the state had already named stays named at its new index, and an asset that the slice removes leaves the set.
Algorithm
- Index the asset axis
1:Nwithi, givingsel, the old index of each asset that the slice keeps. - Keep each new index
jwhose old indexsel[j]is inx.named, givingnamed. - Slice the buffer with
port_opt_view, and return the new state.
Arguments
x: The state to slice.i: Index, indices orBoolmask of the assets to keep.args...: Additional positional arguments, forwarded to the slice of the buffer.
Returns
state::PriorCarryState: The state of the same observations over the selected assets.
Related
PortfolioOptimisers.merge_states — Method
merge_states(
a::PriorCarryState,
b::PriorCarryState
) -> PriorCarryState
Merges two PriorCarryState fitted on disjoint blocks into the state of the concatenated block.
The buffers merge by concatenation. The named-asset sets merge by union, because an asset that either state has already named needs no second notice.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Returns
state::PriorCarryState: The state that a fold of the two blocks as one block gives.
Related
PortfolioOptimisers.prior — Method
prior(pe::AbstractPriorEstimator; kwargs...)Reads a prior out of the state that its estimator carries, with no data matrix.
This is the method of prior with no data, and the entry point of every prior that folds. It reads the state and dispatches on its type:
- A
SampleBufferStatemeans refit. The estimator has no recursion of its own, so the method runs the batch verb over the rows that the buffer kept. These are all the rows folded so far when the buffer has no cap, and the lastmax_historyrows when anOnlineset one. The method drops the rows that the buffer marks with a Shared Series gap, withshared_series_buffer, as the batch fit over the same rows drops them at its entry. The batch verb receives the factor rows that the buffer recorded, throughfactor_buffer, ornothingwhere the buffer recorded none. The fold fixed that choice from the estimator tree. - A
PriorCarryStatemeans fold and carry. Each family that takes it has its own method ofpriorwith no data.
Every method with no data records the source that pe states in the prior result, with stamp_returns_source, as the ReturnsResult method does. A consumer that receives that Result in place of the estimator then reads its frame.
Mathematical definition
Under a SampleBufferState:
\[\begin{align} \mathcal{P}_T &= \mathcal{P}(\mathbf{X}_{S},\, \mathbf{F}_{S},\, \mathbf{E}_{S},\, \mathcal{A}_{S})\,, \\ S &= \{t \in \{T-m+1, \ldots, T\} : g_t = 0\}\,, \\ m &= \min(T,\, M)\,. \end{align}\]
Where:
- $\mathcal{P}_T$: Folded prior, the result that the estimator's batch verb gives with no data after the estimator folds the observations $\boldsymbol{x}_1, \ldots, \boldsymbol{x}_T$ in order, one at a time or in blocks of any size.
- $\mathcal{P}(\cdot)$: Batch prior, the result that the estimator's batch verb gives on the matrices in the parentheses.
- $\mathbf{X}_{S}$: The rows in $S$ of the returns matrix of the folded observations.
- $\mathbf{F}_{S}$: The rows in $S$ of the factor returns matrix of the folded factor observations, absent when the fold recorded none.
- $\mathbf{E}_{S}$: The rows in $S$ of the folded Exogenous Series, absent when the fold recorded none.
- $\mathcal{A}_{S}$: The Asset Panel of the rows in $S$, with the folded Panel Fields and both folded masks, absent when the fold recorded no Panel Field.
- $S$: The rows that the buffer holds and that have no Shared Series gap.
- $g_t$: $1$ when the fold marked observation $t$ with a Shared Series gap, and $0$ otherwise.
- $m$: Number of rows that the buffer holds.
- $M$: Cap on the buffer, the
max_historyof the wrappingOnline, and $\infty$ when it isnothing. - $T$: Number of observations.
Algorithm
- Read the state out of
pe.cachewithpartial_fit_cache. - Dispatch on the type of the state. Under a
SampleBufferState, drop the rows of a Shared Series gap withshared_series_buffer, and run the batch verb over thesample_bufferand thefactor_bufferof the result withbuffer_prior, which gives the masks, the Panel Fields and the Exogenous Series that the buffer records, and the caller's keyword arguments. - Under a
SampleBufferState, record the Shared Series cut withshared_series_record: the marks of the dropped observations of the buffer, and the row key of the factor model block. The batch verb counts the positions among the rows that it kept, so the record maps them back through the rows ofbuffer_kept_rows. The batch verb also counts them from the first row of the buffer, and a capped buffer does not record how many rows it dropped, sobuffer_row_keyremoves them under a cap.
Arguments
pe: Prior estimator carrying a state.kwargs...: Additional keyword arguments, forwarded to the batch verb.
Validation
pe.cacheis notnothing. AnArgumentErrorthat names the verb which fills the field is thrown otherwise.- The rules of
shared_series_buffer.
Returns
pr::AbstractPriorResult: The prior that the state of the estimator gives.
Related
PortfolioOptimisers.partial_fit! — Function
partial_fit!(
pe::AbstractPriorEstimator,
X::Union{AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}, AbstractVector{<:Union{Number, AbstractJuMPScalar}}};
...
) -> Any
partial_fit!(
pe::AbstractPriorEstimator,
X::Union{AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}, AbstractVector{<:Union{Number, AbstractJuMPScalar}}},
F::Union{Nothing, AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}, AbstractVector{<:Union{Number, AbstractJuMPScalar}}};
dims,
active_mask,
estimation_mask
) -> Any
Folds observations, and the factor observations beside them, into the sample buffer that a prior carries.
This is the refit route of the prior family. Every prior that carries a SampleBufferState reaches this method, which has the arity of the prior's own batch verb, prior(pe, X, F). The estimator tree decides what the fold does with F, through needs_factor_returns, and the fold does what the batch verb does with the same argument:
true. The tree holds a factor leaf. The fold refuses a call withoutFwith theIsNothingErrorofassert_factor_returns, which namesF, before it appends a row. A call withFrecords it.false. The tree never readsF, so the fold drops it, asprior(EmpiricalPrior(), X, F)drops it, and the buffer records the rows alone.nothing. The tree does not say, so the buffer recordsFwhen the call gives it and not otherwise, as the batch verb of an optional-argument prior does.
The buffer fixes what it records at its first append and refuses a mixture. So a run that gives F at one step and not at the next is refused at that step with an error.
A prior that reads Panel Fields, as reads_panel_fields answers, refuses this form, because a matrix carries no Asset Panel. It folds a ReturnsResult instead.
Algorithm
- Refuse a prior that reads Panel Fields. Answer
needs_factor_returnsfor the tree, and apply the answer toFwithfold_factor_argument, giving theFthat the buffer records. - Read the buffer out of
pe.cachewithassert_sample_buffer, givingstate. - Fold a matrix, its factor block and its masks into
statethrough the block method ofpartial_fit!, or a vector, its factor observation and its masks through the single-observation method. - Rebuild the prior with its
cacheset to the newstate. - Apply the Choice Rule of the regression of the prior over the rows of
statewithpin_buffer_choice, and return the prior.
Arguments
pe: The prior whose buffer the method folds forward.X: Observations to fold. A matrix holds one observation per row whendims == 1, and one per column whendims == 2. A vector is one observation across the assets, anddimsthen has no effect.F: The factor observations, in the orientation ofX, ornothing.dims: Dimension along which the computation runs.active_mask: The active mask of the block, of the shape ofX, or with one entry per asset whenXis one observation, ornothing.estimation_mask: The estimation mask, on the same terms asactive_mask.
Validation
reads_panel_fields(pe)isfalse. AnArgumentErroris thrown otherwise.Fis notnothingwhenneeds_factor_returns(pe) === true. AnIsNothingErroris thrown otherwise.pecarries aSampleBufferState. AnArgumentErroris thrown otherwise.- Every condition that the fold of the buffer checks.
Returns
pe: The prior, with itscachefield set to the buffer after the last observation.
Related
PortfolioOptimisers.partial_fit! — Method
partial_fit!(pe::AbstractPriorEstimator, rd::ReturnsResult)Folds the observations of a ReturnsResult into a prior. This is the fold of every prior.
It mirrors prior(pe, rd). It reads the returns, the factor returns and both masks of rd, and the Panel Fields of rd.pnl when the tree of the prior reads them. The online step of an optimiser folds its prior through this method. The matrix form, partial_fit!(pe, X, F; active_mask, estimation_mask), stays for a prior that reads no Panel Field.
The type of the prior selects the route, and each route states what it can honour. This method is the refit route, which every prior that carries a SampleBufferState takes. The buffer records what the batch fit reads, so the route honours all of it:
- The estimation mask, beside the active mask. No panel, or a static panel, gives no mask.
- The Panel Fields, when
reads_panel_fieldsanswerstrue. The buffer records them, and the call with no data gives the batch verb the panel thatsample_buffer_panelrebuilds. For any other prior the call refuses a Panel Field with an error that names it, because the buffer has no slot that the prior reads. - The Exogenous Series, when
reads_exogenous_seriesanswerstrue. The buffer records every column of it with its names, and the call with no data gives them to the batch verb. For any other prior the Fold Context keeps the series, and the prior's buffer records none.
The carry of EmpiricalPrior has a route of its own, because its exact folds take no estimation mask. HighOrderPriorEstimator and BlackLittermanPrior forward rd to the prior they embed. CrossSectionalFactorPrior has a route of its own, which takes the carry fold when the prior has no buffer and applies its Choice Rule after the fold of a refit. fold_prior_step holds each route.
A batch fit drops an observation with a gap in a Shared Series that it reads, with shared_series_view. A prior reads the factor returns as a Shared Series when reads_factor_returns answers true. The step keeps the observation and marks it, because the buffer of a prior and the Fold Context of its optimiser hold one row each for every observation. Every fit through the buffer then drops it, and a carry fold leaves it out of its running moments. So the call with no data gives the batch fit over the same observations.
Algorithm
- Read the source that the prior states with
returns_source, givingx_src. - Mark the observations with a Shared Series gap with
shared_series_gaps: a gap of a vector benchmark underx_src = ExcessSource(), and a gap of the factor returns when the prior reads them. Mark a gap of a matrix benchmark in the estimation mask underExcessSource()withbenchmark_estimation_mask, as the batch fit marks it. - Take the observations that the prior folds with
prior_step_returnsunderx_src:rd, orrdwithX - Bin place ofXunderExcessSource(). - Fold them with their marks with
fold_prior_step.
Arguments
pe: The prior whose buffer the method folds forward.rd: The returns result to use.
Validation
- The rules of
fold_prior_step.
Returns
pe: The prior, with itscachefield set to the buffer after the last observation.
Related
PortfolioOptimisers.partial_fit! — Function
partial_fit!(pe::EmpiricalPrior{<:Any, <:Any, Nothing, <:Any, <:Any,
<:Option{<:PriorCarryState}}, X, F = nothing; kwargs...)
partial_fit!(pe::EmpiricalPrior{<:Any, <:Any, <:Number, <:Any, <:Any,
<:Option{<:PriorCarryState}}, X, F = nothing; kwargs...)Folds observations into an EmpiricalPrior, which folds its moments and carries its rows.
pe.me and pe.ce fold exactly, so a step of the prior is their two steps and an append to the carried rows. The cost of a step grows with the square of the number of assets, and does not grow with the number of observations already folded. The cost of a refit from a buffer grows with that number too. This difference is the reason the carry route exists. A call with no data that is written as prior(pe, sample_buffer(pe.cache)) passes every parity test and discards both exact folds, so a review must refuse that form.
The two methods differ in one line. The method with no horizon folds the observation as it is. The horizon method folds log1p of the observation, because the batch horizon method fits its moments on log returns. It carries the observation itself, because the LowOrderPrior it returns carries the arithmetic returns that the caller passed. A buffer of log rows would change the matrix that every scenario risk measure reads.
A matrix of observations can carry the marks gaps of its Shared Series gaps, see shared_series_gaps. A batch fit drops a marked observation, so the two members fold the unmarked observations alone, and the carry state keeps every observation with its mark, because the Fold Context of an optimiser holds one row for each observation. The call with no data reads the unmarked rows, see shared_series_buffer.
A member that does not fold, such as a SemiMoment covariance or a composite whose mp.alg reads the sample, stays as it is under fold_member, and prior(pe) with no data refits it over the carried rows. So a caller writes the estimator that they would write in batch, and no member keeps a second copy of the sample.
The fold takes a factor observation and drops it, because prior(EmpiricalPrior(), X, F) declares F and never reads it. The fold has the arity of the batch verb, so an outer estimator that passes F down its tree meets no MethodError here. needs_factor_returns answers false for this estimator, so no estimator above it keeps a factor row for it.
Algorithm
- Orient a matrix
Xand a matrixivtoobservations × assetswithdims_oriented, transposing them whendims == 2. Under a matrixX, take its unmarked observations and their active mask withshared_series_member_rows. - Under the horizon method, take
log1pof the observations of step 1. - Fold
pe.meandpe.cethroughfold_carry_members, on the observations of step 2 under the horizon method and on the observations of step 1 otherwise. At the first step, whenpe.cacheisnothing, it first refuses a member that holds a state. - Append every arithmetic observation, with
iv,ivpaand the marksgaps, to the carry state throughfold_carry, and return the estimator with its members and itscachefield rebound. No member that folds readsiv, and a member that does not fold reads it from the carried buffer at the call with no data.
Arguments
pe: Empirical prior estimator.X: Observations to fold. A matrix holds one observation per row whendims == 1, and one per column whendims == 2. A vector is one observation across the assets.F: The factor observations besideX, ornothing. The fold drops them, as the batch verb does.dims: Dimension along which the computation runs.iv: The implied volatilities of the observations, oriented asXis, ornothing. The carry state records them.ivpa: The implied volatility risk premium adjustment, ornothing. The carry state pins it.gaps: Under a matrixX, one mark for each observation,trueat an observation with a Shared Series gap, ornothing. The carry state records them.kwargs...: Additional keyword arguments, forwarded to the two members and to the carry state.
Validation
dims in (1, 2).- Every condition that the fold of
pe.meorpe.cechecks. A member that holds observation weights refuses the fold with anArgumentErrorthat names the member, because a weight vector reweights every past observation when a new one arrives. - At the first step, no member holds a partial-fit state, see
assert_cold_carry_entry. The carried rows start empty, so a warm member would read rows that the carried rows, the refitted members andXdo not read. AnArgumentErrorthat names the path of the state is thrown otherwise.
Returns
pe: The estimator, with its members folded and itscachefield rebound.
Related