Price ingestion: private API
PortfolioOptimisers.project_panel_clock — Function
project_panel_clock(pnl::Nothing, tsj, X::TimeSeries.TimeArray, ca::Tuple) -> nothing
project_panel_clock(pnl::AssetPanel, tsj, X::TimeSeries.TimeArray, ca::Tuple) -> AssetPanelPut a caller's AssetPanel on the clock the ingestion emits.
A caller states a Panel Field on the clock of the table they hold. The join and the collapse of price_ingestion move that clock, so price_ingestion makes the projection. After it, the PricesResult states one clock and everything it holds is on that clock. A static panel has no observation axis, so feature_row_indices returns Colon() for it and the panel passes through unchanged.
A collapse puts many rows into one period. Each period takes the Panel Field values of the last row of its group, as LastRow does. The timestamp function of ca does not change this row. So under (Dates.week, first, last) a week pairs the prices of its last day with the Panel Field values of the same day.
Algorithm
pnlisnothing: returnnothing.- Otherwise read the incoming clock
ts_oldand the asset names offX, and check the panel against them withcheck_asset_panel. - Find the timestamp of the row that each emitted observation takes, giving
ts_row. Without a collapse,ts_rowistsj. With a collapse, collapse the row positions oftsjwith the period and timestamp functions ofcaand the value functionlast. This gives the last row of each group, asTimeSeries.collapseforms the groups, andts_rowholds the timestamps of those rows. - Find the rows of
ts_rowin the incoming clock withfeature_row_indices, and view the panel over them withport_opt_view. The view takes the whole asset axis and its names, so that it also cuts a square tensor Panel Field on its label axis.
Arguments
pnl: The caller'sAssetPanel, ornothing.tsj: The timestamps of the joined table, before the collapse.X: The asset table the caller passed. Its timestamps are the incoming clock, and its column names are the asset names.ca: Thecollapse_argsof thePriceIngestion. An empty tuple states no collapse.
Validation
- The panel describes the assets of
X, and a time-varying panel also its observations. Raises aDimensionMismatch. - Every entry of
ts_rowis a timestamp ofX, when the panel is time-varying. Raises anArgumentError, frommatched_row_indices.
Returns
pnl′::Option{<:AssetPanel}: The panel on the emitted clock, ornothing.
Related
PortfolioOptimisers.unify_gaps — Function
unify_gaps(A::TimeSeries.TimeArray, ::Type{T} = absence_type(series_value_type(A))) -> TimeSeries.TimeArraySpell every absent price of one series the one way the ingestion layer carries, in the value type T.
A source spells an absent price one of two ways. An outer join of ragged per-asset histories pads with NaN, and a wide table built from a tidy one leaves missing. The layer unifies both as NaN, which no deletion step reads. This defines a gap for every later piece, so it runs before the Span Rule and before the join.
The caller does not name the target type. The default of T derives it from the series alone with absence_type, and the conversion uses this default for a PricesResult built by hand. price_ingestion first promotes the asset, factor and benchmark series to one type, because the join lays them in one table, and passes that type as T.
Algorithm
- Refuse an infinite value with
assert_no_infinite_price, which throws an error that names it. The Span Rule reads a non-finite cell as unpriced and the conversion reads it as a price, so the two would disagree about the cell. - The values are already of type
T, so their gaps are alreadyNaN: returnAuntouched. - At least one value is
missing: rebuild the values. Map eachmissingtoabsent_value(T)and convert every other entry toT. This is the one place where amissingbecomes an absence, so the refusal of a type that cannot hold one happens here, by name. - Otherwise convert every entry to
T. Nothing is absent, so the function spells nothing and does not ask whetherTcan spell an absence. This includes an element type that admitsmissingwhen no value ismissing.
Arguments
A: One price series.T: The value type to carry it in. Defaults toabsence_type(series_value_type(A)).
Validation
- Every value is
missing,NaN, or finite. An infinity raises aDomainErrornaming the column and the observation. Tcarries an absence when a value ismissing. Raises aDomainErrornamingT, fromabsent_value.- With the default
T: the element type of the series is notMissingalone. Raises aDomainError, fromabsence_type.
Returns
A′::TimeSeries.TimeArray: The same series in typeT, with every absent price spelledNaN.
Related
PortfolioOptimisers.assert_no_infinite_price — Function
assert_no_infinite_price(v::AbstractArray, names, ts) -> nothingRefuse an infinite value in a price series by name.
The layer spells an absent price NaN, and an infinity is not that spelling. Two pieces of the layer read an infinity differently. listing_span reads a non-finite cell as unpriced, so the Span Rule keeps an interior infinity inside the listing. The conversion reads the same cell as a price, and the return that follows it is a finite -100 %. This function refuses the value where the layer fixes the spelling, so every later piece reads one definition of a gap.
Algorithm
- Find the first value that is not
missing, notNaNand not finite, givingk. - No such value: return.
- Read the observation and the column of
kfrom its Cartesian index. A one-column series can hold its values in a vector, and then the column is the first. - Raise a
DomainErrorthat names the value, the column and the timestamp.
Arguments
v: The values of one series,observations × columns.names: The series' column names.ts: The series' timestamps.
Validation
- Every value is
missing,NaN, or finite. Raises aDomainErrornaming the first offending column and observation.
Returns
nothing.
Related
PortfolioOptimisers.series_value_type — Function
series_value_type(A::Nothing) -> Type{Union{}}
series_value_type(A::TimeSeries.TimeArray) -> TypeRead the value type of one optional price series, without the Missing a wide table may carry.
The ingestion layer derives the type it carries prices in from the series themselves. Each series contributes the type of its values without Missing, because the target type spells an absence and the source does not. A series the caller omitted contributes Union{}, the identity of promote_type, so the site that promotes needs no branch for an absent table.
Algorithm
The method that Julia selects is the algorithm.
Aisnothing: returnUnion{}.Ais a table: return the element type of its values withMissingremoved. A table whose element type isMissingalone givesUnion{}too, so it contributes nothing to the promotion.
Arguments
A: One price series, ornothing.
Returns
T::Type: The value type,Union{}for no series.
Related
PortfolioOptimisers.absence_type — Function
absence_type(::Type{T}) -> Type
absence_type(::Type{Union{}}) -> Union{}Derive the type the ingestion layer carries a series of value type T in.
The layer spells an absent price NaN, so the carried type must be able to hold one. The layer reads that type off the data and does not name it: the type is float_if_integer of T. A floating-point type gives itself, so a Float32 panel stays Float32. An integer type gives its floating-point type, Float64 for Int and BigFloat for BigInt. A Rational and a number type the library does not know give themselves.
This function does not decide whether the type can hold an absence, because a series with no gap and no padding never spells one. absent_value throws an error that names the type when the layer writes an absence in it.
Algorithm
The method that Julia selects is the algorithm.
TisUnion{}: refuse. Every price table holds onlymissing, so no value type exists to derive.- Otherwise return
float_if_integerofT.
Arguments
T: The value type of a series, asseries_value_typereads it.
Validation
T !== Union{}, whichseries_value_typegives for a table whose element type isMissingalone. Raises aDomainError.
Returns
S::Type:float_if_integerofT.
Related
PortfolioOptimisers.absent_value — Function
absent_value(::Type{T}) -> TReturn the one spelling of an absent value in type T, and throw an error that names a type that cannot hold one.
The layer spells an absence one way, NaN, so a type that cannot represent NaN cannot state a gap. This function refuses such a type, for example a Rational panel that holds a gap or that a join must pad. Without it, a conversion raises an InexactError that names nothing. The refusal names the type and what the caller can do. An integer panel never reaches the refusal, because absence_type widens it to a floating-point type first.
The layer stays open to number types it does not know. A type carries an absence when convert(T, NaN) returns a value for which isnan holds.
Algorithm
- Convert
NaNtoT. A conversion that throws givesnothing. - Refuse
Tunless the conversion gave a value andisnanholds for it. - Return that value.
Arguments
T: The type an absence must be spelled in.
Validation
convert(T, NaN)succeeds andisnanof it holds. Otherwise raises aDomainErrornamingT.
Returns
x::T: The absence,convert(T, NaN).
Related
PortfolioOptimisers.assert_pad_spellable — Function
assert_pad_spellable(::Type{T}, method::Symbol) -> nothingBefore a join that can pad, throw an error that names a value type that cannot spell the pad.
TimeSeries.merge pads with NaN converted to the table's value type, which has the same bits as absent_value(T). For a type that cannot hold an absence, merge raises an InexactError that names nothing. This function asks first, so the refusal names the type. It asks before every join that can pad, whether or not the join then pads a row. An inner join drops an observation and does not pad it, so it asks for nothing.
Algorithm
methodis:inner: return.- Otherwise ask
absent_value(T), which throws an error that names the type.
Arguments
T: The value type of the tables the layer joins, the layer's unification target.method: The join, asTimeSeries.mergetakes it.
Validation
Tcarries an absence whenmethodis not:inner. Raises aDomainErrornamingT, fromabsent_value.
Returns
nothing.
Related
PortfolioOptimisers.align_series — Function
align_series(A::TimeSeries.TimeArray{T}, ts::AbstractVector) -> TimeSeries.TimeArrayPut one carried series on the clock the ingestion layer emits, and pad the observations where it is silent.
The layer carries the implied volatilities beside the assets and does not join them into the asset table, because they carry the asset names and a join would rename them. The alignment works as a left join works for a factor. The emitted clock decides. The function keeps each observation the series states, and each observation the series does not state becomes an absence that the layer carries.
Algorithm
- Fill a table of
length(ts)rows and the series' columns withabsent_valueof its type. This asks for an absence even when the series covers the clock, as a join that can pad does. - Write each row of the series whose timestamp is in
tsat that timestamp's row.
Arguments
A: The series to align.ts: The emitted clock.
Validation
Tcarries an absence. Raises aDomainErrornamingT, fromabsent_value.
Returns
A′::TimeSeries.TimeArray: The series onts,NaNwhere it was silent.
Related
PortfolioOptimisers.padded_observations — Function
padded_observations(A::Nothing, ts::AbstractVector) -> Int
padded_observations(A::TimeSeries.TimeArray, ts::AbstractVector) -> IntCount the observations of the emitted clock at which one series is silent.
The join or the alignment padded each of those observations, and the layer carries each padded observation as an absence and names it. A series the caller omitted is silent nowhere.
Algorithm
The method that Julia selects is the algorithm.
Aisnothing:0.Ais a table: the number of timestamps intsabsent fromA's.
Arguments
A: One price series, ornothing.ts: The clock the layer emitted.
Returns
n::Int: The number of padded observations.
Related
PortfolioOptimisers.padding_report_line — Function
padding_report_line(name::String, A::Option{<:TimeSeries.TimeArray}, ts::AbstractVector) -> Option{String}Write the line of the padding report that one series owes, or nothing when the layer padded it nowhere.
Algorithm
- Count the padded observations with
padded_observations. - None: return
nothing. - Otherwise write a line that names the table, the count out of the clock's length, and every column. A series silent at an observation is silent in all its columns.
Arguments
name: How the caller spells the series,"X","F","B"or"iv".A: The series, ornothing.ts: The clock the layer emitted.
Returns
line::Option{String}: The report line, ornothing.
Related
PortfolioOptimisers.assert_join_padding — Function
assert_join_padding(lines::AbstractVector, strict::Bool) -> nothingName what the join and the alignment padded, warning by default and refusing under strict.
A carried absence raises no error later. A fit that reads a padded factor or a padded one-column benchmark drops that observation, and a fit that reads a padded implied volatility or a padded column of a per-asset benchmark drops that asset, see shared_series_view and coverage_mask. Neither drop names its cause. So this function names it through strict_diagnostic, which a Held Gap and PriceGapFill also use to report. A caller who runs a walk-forward and sets strict knows that the layer padded no covariate.
Algorithm
- Drop the
nothingentries. None is left, so the layer padded nothing: return. - Otherwise join the lines into one report that names the three join methods, and pass it to
strict_diagnostic.
Arguments
lines: The report linespadding_report_linewrote,nothingentries included.strict: Iftrue, throws anArgumentError; iffalse, issues a warning.
Validation
- The layer padded nothing. Raises an
ArgumentErrorunderstrict.
Returns
nothing.
Related
PortfolioOptimisers.series_names — Function
series_names(A::Nothing) -> Vector{Symbol}
series_names(A::TimeSeries.TimeArray) -> Vector{Symbol}Name the columns of one optional price table, and name none when the caller passed none.
A series the caller omitted contributes no column name. It contributes an empty list, so no site that reads the names needs an isnothing branch.
Algorithm
The method that Julia selects is the algorithm.
Aisnothing: return an empty name list.Ais a table: return its column names.
Arguments
A: One price table, ornothing.
Returns
n::Vector{Symbol}: The table's column names, empty when there is no table.
Related
PortfolioOptimisers.assert_disjoint_series_names — Function
assert_disjoint_series_names(a::AbstractVector{Symbol}, b::AbstractVector{Symbol}, na::String, nb::String) -> nothingRefuse a column name that two of the price tables both carry.
A column name says which series a column came from. The layer merges the asset, factor and benchmark tables onto one clock and then splits the blocks by name. TimeSeries.merge renames the second of two columns that share a name by appending _1. So a shared name makes one block take the other's column with no error, and the new name belongs to no block. The conversion also cannot choose between the two columns. X's AAPL and F's AAPL are different series, and to keep either one is worse than to refuse both.
Arguments
a: Column names of the first table.b: Column names of the second table.na: How the caller spells the first table.nb: How the caller spells the second table.
Validation
isdisjoint(a, b). Raises aConflictingArgumentErrornaming the shared columns.
Returns
nothing.
Related
PortfolioOptimisers.assert_unreserved_series_names — Function
assert_unreserved_series_names(n::AbstractVector{Symbol}, nn::String) -> nothingRefuse a series named after the observation clock.
The conversion writes the clock into a column named timestamp. A series with the same name then becomes timestamp_1, and the clock keeps the name. So a lookup of the series by its name reads the dates as prices. The layer owns the name, and a caller who holds a series with that name must rename it.
Arguments
n: Column names of one table.nn: How the caller spells that table.
Validation
:timestamp ∉ n. Raises aConflictingArgumentError.
Returns
nothing.
Related
PortfolioOptimisers.assert_distinct_series_names — Function
assert_distinct_series_names(X::TimeSeries.TimeArray, F::Option{<:TimeSeries.TimeArray} = nothing, B::Option{<:TimeSeries.TimeArray} = nothing, E::Option{<:TimeSeries.TimeArray} = nothing) -> nothingCheck that every price series reaching the layer can still be named after the join.
Both entry points of the layer run this check before they merge, price_ingestion and prices_to_returns. After it, every later piece can split the merged table by name. Each name belongs to exactly one of the asset, factor and benchmark blocks, and no name is the clock's.
The Exogenous Series E is converted beside the three blocks, and a consumer selects its columns by name. So its names are also disjoint from the three blocks, and no name is the clock's.
This function cannot check two series of one table. The TimeSeries.TimeArray constructor runs TimeSeries.replace_dupes! over its column names, so it renames the duplicates of a table before the table exists, and no duplicate reaches this function.
Algorithm
- Read the four name lists, an absent table naming none, with
series_names. - Refuse a name shared by two of them with
assert_disjoint_series_names, over all six pairs. - Refuse the clock's own name in any of the four with
assert_unreserved_series_names.
Arguments
X: Asset prices,observations × assets.F: Optional factor prices.B: Optional benchmark prices.E: Optional Exogenous Series.
Validation
- The asset, factor, benchmark and Exogenous Series names are pairwise disjoint. Raises a
ConflictingArgumentErrornaming the shared columns. - None of the four is
timestamp. Raises aConflictingArgumentError.
Returns
nothing.
Related
PortfolioOptimisers.assert_span_shape — Function
assert_span_shape(span::Nothing, nobs, na) -> nothing
assert_span_shape(span::AbstractMatrix{Bool}, nobs::Integer, na::Integer) -> nothingCheck that a Listing Span has the shape of the price panel it describes.
A span states which assets are listed at each observation of the price clock, so it has the shape of the asset prices. A PricesResult that holds no span states no universe, and there is nothing to check.
Algorithm
The method that Julia selects is the algorithm.
spanisnothing: thePricesResultstates no universe, so there is nothing to check.- Otherwise check the span's shape against the price panel's.
Arguments
span: The Listing Span on the price clock, ornothing.nobs: Observation count of the price panel.na: Asset count of the price panel.
Validation
size(span) == (nobs, na). Raises aDimensionMismatch.
Returns
nothing.
Related
PortfolioOptimisers.returns_universe_masks — Function
returns_universe_masks(span::Nothing, R::AbstractMatrix)
returns_universe_masks(span::AbstractMatrix{Bool}, R::AbstractMatrix)Derive the two universe masks that the AssetPanel of the ReturnsResult states.
A PricesResult from the layer arrives with a Listing Span. This function projects the span onto the returns clock and intersects it with finiteness. A PricesResult built outside the layer arrives without a span and states no universe, whether or not its prices hold a gap. The function does not derive a span for it, because a derivation from one window reads a delisting that straddles the window end as an asset that was never listed. The library still handles the gaps of such a PricesResult, because with no panel the Coverage Universe reads finiteness alone. So pnl === nothing means that the returns data states no universe beyond the finiteness of its returns: the layer did not build the PricesResult, or the prices held no gap.
Algorithm
The method that Julia selects is the algorithm.
spanisnothing: returnnothing, nothing.spanis given: project and intersect it withuniverse_masks, then compress the pair withcompress_all_true.
Arguments
span: The Listing Span over the surviving price rows and assets, ornothing.R: The returns panel the conversion produced,observations × assets.
Returns
(amsk, emsk): The two masks, ornothing, nothingwhen thePricesResultstates no universe.
Related
PortfolioOptimisers.compress_all_true — Function
compress_all_true(amsk::AbstractMatrix{Bool}, emsk::AbstractMatrix{Bool})Store a pair of universe masks that is true everywhere as the constant it is.
A gapless ingestion states a universe in which every asset is listed at every observation and every return is finite, so both masks are true everywhere. A PortfolioOptimisers.AllTrueMask stores that in two integers. So a panel that carries such masks beside its Panel Fields costs O(1) memory for them, not one bit per observation and asset. With no Panel Field, attach_universe_masks reads the type and emits no panel at all. The estimation mask is a subset of the active mask. When the estimation mask is all true, the active mask is all true too, so the function tests the estimation mask alone.
Algorithm
- Some entry of
emskisfalse: return the two masks unchanged. - Otherwise return one
PortfolioOptimisers.AllTrueMaskof the same size as both masks.
Arguments
amsk: The active mask.emsk: The estimation mask.
Returns
(amsk, emsk): The two masks, compressed when they admit it.
Related
PortfolioOptimisers.attach_universe_masks — Function
attach_universe_masks(pnl, amsk::Nothing, emsk::Nothing)
attach_universe_masks(pnl::Option{<:AssetPanel{Nothing}}, amsk::AllTrueMask, emsk::AllTrueMask) -> nothing
attach_universe_masks(pnl::Option{<:AssetPanel{Nothing}}, amsk::AbstractMatrix{Bool}, emsk::AbstractMatrix{Bool}) -> AssetPanel
attach_universe_masks(pnl::AssetPanel{<:AbstractVector}, amsk::AbstractMatrix{Bool}, emsk::AbstractMatrix{Bool}) -> AssetPanelPut the two universe masks onto the AssetPanel that the ReturnsResult holds.
The layer emits one ReturnsResult, and the masks go on it. A Pipeline step has one output, and PricesToReturns cannot emit the masks by another route without putting them on the price clock, where the masks are not stated. Because the ReturnsResult holds the masks, port_opt_view slices the universe together with the returns and needs no extra code.
The function keeps a caller's Panel Fields and replaces both masks with the layer's. The active mask cannot be a declaration that a panel brings into the layer. A time-varying panel holds an amsk by construction, and asset_panel writes an all-true one when the caller states none. So an all-true amsk beside a time-varying Panel Field looks the same as a declared calendar in which every asset is listed. The only way to state a listing is span on PriceIngestion. The estimation mask states a fact about the data and not about the instruments, so the conversion derives it again in every case with universe_masks, and emsk ⊆ amsk holds by construction.
Algorithm
The method that Julia selects is the algorithm.
- No masks: return the panel unchanged. The
PricesResultstates no universe. - All-true masks and no Panel Field: return
nothing. Every asset is listed and every return is finite, so the masks state nothing that the finiteness of the returns does not state, and the Coverage Universe of no panel is the same. - Other masks and no panel: return an
AssetPanelof the two masks andpf = nothing. This is the layer's common case when the prices hold a gap: a caller holding only prices has no feature data. - Masks and a panel with no Panel Field: apply steps 2 and 3. The function replaces both masks, so the panel brings nothing else.
- Masks and a panel with Panel Fields: keep its Panel Fields and replace both its masks. A static panel's fields carry no observation axis, so the function first lifts them onto the clock of the masks with
panel_field_lift.
Arguments
pnl: The Asset Panel that thePricesResultheld, ornothing.amsk: The active mask, ornothing.emsk: The estimation mask, ornothing.
Returns
pnl′::Option{<:AssetPanel}: The panel that theReturnsResultholds, ornothingwhen there is no panel or the panel states nothing.
Related
AssetPanelasset_panel: writes the all-trueamska time-varying build carries when the caller states none.PriceIngestion: itsspanis the only way to state a listing.returns_universe_maskspanel_field_liftprices_to_returns
PortfolioOptimisers.listing_span_view — Function
listing_span_view(span::Nothing, ts_new, ts_old, j) -> nothing
listing_span_view(span::AbstractMatrix{Bool}, ts_new, ts_old, j) -> SubArray
listing_span_view(span::ListingSpan, ts_new, ts_old, j) -> Union{ListingSpan, SubArray}View the Listing Span of a PricesResult over the surviving timestamps and the assets j, or return nothing when the PricesResult holds none.
The span states a fact about the instruments, so the span of a window is a view of the panel-wide span, never a span derived again from the window. A span derived from a window reads a delisting that straddles the window end as an asset that was never listed. A view is exact, and over a PortfolioOptimisers.ListingSpan it stores no boolean cell.
The span is parallel to the price clock by position. So the function finds its rows from the surviving timestamps with matched_row_indices, as it does for a time-varying AssetPanel.
Algorithm
The method that Julia selects is the algorithm.
spanisnothing: returnnothing, and match no timestamp.spanis a matrix: find its rows withmatched_row_indices, and view it at those rows and the assetsj.spanis aPortfolioOptimisers.ListingSpanand the rows are a contiguous window in clock order, for example a fold's window or the whole clock. Subtract the number of rows before the window from both bounds, keep the bounds of the assetsj, and return a newListingSpanof the window's length. The result keeps two integers per asset and does not expand them into a view of booleans. A bound outside the window states the same listings inside it as a bound at its edge. Any other row selection can split an interval in two, which one interval cannot state, so it takes step 2.
Arguments
span: The Listing Span on the price clock, ornothing.ts_new: Timestamps that survived the selection.ts_old: Timestamps of the price clock the span is parallel to.j: Asset indices, a vector orColon().
Returns
s::Option{<:AbstractMatrix{Bool}}: A newListingSpanfor a contiguous window of aListingSpan, a view ofspanfor any other selection, ornothing.
Related