Price ingestion
From raw prices to a prices result
PriceIngestion turns raw price series into the PricesResult that the conversion to returns reads. It runs once, on the whole history, and it is not a Pipeline step. It also spells every absent price, missing or a non-finite number, as NaN. Two of its steps change the observations themselves. It joins the factor and benchmark series to the prices, and it can collapse the data to a lower frequency. Cross-validation cuts its folds once, on the observations of the result, so a step that adds, removes or renumbers observations cannot run inside a fold. A hyperparameter that changed the test window would also give scores that you cannot compare. Because it sees the whole history, PriceIngestion also finds the listing span of each asset from all of its prices. A step inside a pipeline sees one window, which does not show whether an asset is listed before or after it.
The result holds the listing span of each asset in its span field. PricesToReturns moves the spans onto the dates of the returns and gives the ReturnsResult an AssetPanel that states the universe. When no price is missing and you give no Panel Field, every asset is listed and every return is finite. The universe masks then state nothing that the returns do not state, so the conversion gives pnl === nothing. When you give Panel Fields and no price is missing, both universe masks are a PortfolioOptimisers.AllTrueMask, which stores no cells.
PriceIngestion writes an absent price as NaN, the one value the library uses for a missing number. The conversion to returns keeps it as a NaN return, and deletes neither the observation nor the asset. A PricesResult that PriceIngestion did not build has no spans, so the conversion gives its returns no universe. The conversion does not guess a universe from one window.
The asset prices set the observation dates. With the default join_method = :left, PriceIngestion aligns the factor, benchmark and implied-volatility series to the dates of the asset prices, and writes NaN where a series has no value. The result then has the timestamps of the asset prices, timestamp(pr.X) == timestamp(X), unless collapse_args is non-empty. The :outer join keeps the union of the dates, and the :inner join keeps their intersection. PriceIngestion reports every observation it padded, with the table, the number of observations and the columns. It warns by default, and throws an error when strict is set.
A collapse puts the observations of one period into one row. A time-varying AssetPanel then takes the values of the last observation of each period, whatever timestamp function collapse_args names. With collapse_args = (week, first, last), the prices and the panel values of a week both come from its last day.
The element type of the result comes from the input series. A Float32 panel stays Float32, Float32 next to Float64 gives Float64. The ingestion layer is an entry point of integer data, so an integer panel converts here, once, to its floating-point type, and every kernel downstream reads floating-point data. A type that has no value for an absent number throws an error that names the type. The error comes at a gap, before a join that can pad, and when PriceIngestion aligns the implied volatilities, even if the join or the alignment then pads nothing. An absent implied volatility is kept like an absent price, and ImpliedVolatility then fits only on the columns whose implied volatilities have no gaps.
Types
PortfolioOptimisers.PriceIngestion — Type
struct PriceIngestion{__T_join_method, __T_collapse_args, __T_span, __T_strict} <: AbstractEstimatorEstimator that assembles raw price series into a PricesResult and its Listing Span, which the conversion then reads.
PriceIngestion runs once on the whole panel, and it is not a Pipeline step. The observation clock decides this. The factor and benchmark join can add or drop observations, and the frequency collapse renumbers them. Cross-validation cuts its folds once on the clock of the PricesResult. A hyperparameter that moves the clock changes the test set, and a score on one test set cannot be compared with a score on another. A step that changes only values leaves the clock alone, so it stays inside the Pipeline.
Because it runs outside the Pipeline, the Span Rule can read the whole panel. A step sees only a window, and a span derived from a window reads a delisting that straddles the window end as an asset that was never listed. A split forbids work on the data before it, but a listing calendar is a fact about the instruments and not an estimate from returns, so the panel-wide read is valid.
The PricesResult holds the derived Listing Span in its span field. The conversion projects it onto the returns clock and gives the ReturnsResult an AssetPanel that states the universe. A caller who holds a listing calendar passes it as span, and it replaces the Span Rule's answer.
The asset table states the clock. It states the universe, so it also states the clock. Under the default join_method = :left, the layer aligns the factor, benchmark and implied-volatility series to that clock and pads them where they are silent. The layer names what it padded: the table, the number of observations and the columns. It warns by default and refuses under strict, because a carried absence raises no error later: a fit drops the observation of a padded factor or one-column benchmark, and the asset of a padded implied volatility or per-asset benchmark, and does not name the cause.
Each field states a fact about the sources that the layer cannot derive. join_method states which clock decides, collapse_args the frequency the analysis wants, span the caller's listing calendar, and strict whether a padded covariate is acceptable.
Fields
join_method: How the layer joins the factor and benchmark series onto the asset clock, asTimeSeries.mergetakes it.:leftkeeps the asset clock and pads the others where they are silent.:outertakes the union of the clocks and can pad every table.:innertakes their intersection and pads none.
collapse_args: Arguments for collapsing the joined series to a lower frequency, asTimeSeries.collapsetakes them. An empty tuple leaves the clock alone.
span: Optional listing statement of the caller's own,price observations × assets. It replaces the answer of the Span Rule. AnyAbstractMatrix{Bool}is valid, for example a listing calendar, or a constituency that leaves and rejoins.nothingderives the span from the gaps.
strict: Whether a padded observation refuses the ingestion.falsenames what the join and the alignment padded in a warning, andtrueraises anArgumentErrorwith the same report.
Constructors
PriceIngestion(; join_method::Symbol = :left, collapse_args::Tuple = (), span::Option{<:AbstractMatrix{Bool}} = nothing, strict::Bool = false,) -> PriceIngestionKeywords correspond to the struct's fields.
Validation
join_method in (:left, :outer, :inner).
Examples
julia> X = TimeArray(Date(2020, 1, 1):Day(1):Date(2020, 1, 3), [NaN 101.0; 102.0 103.0; 104.0 105.0], ["A", "B"]);julia> pr = price_ingestion(PriceIngestion(), X);julia> pr.spanListingSpan(3 × 2)julia> Matrix(pr.span)3×2 Matrix{Bool}: 0 1 1 1 1 1Related
Functions
PortfolioOptimisers.price_ingestion — Function
price_ingestion(est::PriceIngestion, X::TimeSeries.TimeArray;
F::Option{<:TimeSeries.TimeArray} = nothing,
B::Option{<:TimeSeries.TimeArray} = nothing,
E::Option{<:TimeSeries.TimeArray} = nothing,
iv::Option{<:TimeSeries.TimeArray} = nothing,
ivpa::Option{<:Num_VecNum} = nothing,
pnl::Option{<:AssetPanel} = nothing) -> PricesResult
price_ingestion(est::PriceIngestion, pr::PricesResult) -> PricesResultAssemble raw price series into a PricesResult and its Listing Span.
The three steps that move the clock run here and not in a Pipeline, and the Span Rule reads the whole panel after them. The result is an ordinary PricesResult. Its span field states which assets are listed at each observation, and PricesToReturns projects it onto the returns clock.
The layer spells an absent price NaN, the library's one spelling for absence. The conversion carries it into the returns and does not delete the observation or the asset that holds it. Every axis the layer touches carries its absences. If a factor, benchmark or implied-volatility series is silent at an observation of the emitted clock, the layer pads it there and names the padding. It warns by default and refuses under strict, because a fit that reads a padded series drops the observation or the asset in silence: a gap in a factor or a one-column benchmark drops the observation, and a gap in an implied volatility or a column of a per-asset benchmark drops the asset.
The caller does not name the value type of the PricesResult. The layer promotes the asset, factor and benchmark types to one type, because the join lays them in one table. It then widens that type only as the return arithmetic would widen it. A Float32 panel stays Float32, Float32 beside Float64 joins in Float64, and an integer panel takes the floating-point type that represents it. The layer throws an error that names a type that cannot spell an absence wherever it might have to spell one: at a missing value, before a join that can pad, and when it aligns the implied volatilities. The join and the alignment ask before they know whether they pad a row.
Algorithm
- Check with
assert_distinct_series_namesthat the asset, factor and benchmark series keep their names through the join, and that no Exogenous Series takes the name of one of them. The join renames a name that two tables share, and a block would then take another block's column. - Promote the value types that the three price tables contribute through
series_value_type, and widen the result withabsence_type, giving the target type. Unify the absent-price spelling of the three tables in that type withunify_gaps, so a gap means one thing from here on. - Join the factor and the benchmark series onto the asset clock under
join_method. A left join keeps the asset clock and pads the others where they are silent. An outer join adds the rows that one series has and another does not, and can pad every table. An inner join keeps the rows that every table has. Before a join that can pad, askassert_pad_spellable, which throws an error that names a type that cannot spell the pad. - Write the padding report lines of the asset, factor and benchmark tables against the joined clock with
padding_report_line, before the collapse renumbers it. - Collapse the joined series to a lower frequency when
collapse_argsis not empty. This renumbers every observation. - Split the joined series back into the asset, factor and benchmark blocks, all on one clock.
- Align the implied volatilities to that clock with
align_series, in the type thatabsence_typederives from their own type, and pad the observations where they are silent. Write their report line, and carryivpathrough. An implied volatility is a volatility and not a price, so the layer carries it and never converts it to a return. Align the Exogenous SeriesEto the same clock in the same way, and write its report line.Eis aligned and never joined, so it adds and drops no observation of the assets: a series one consumer reads must not change the universe of every asset. It is a level series, soprices_to_returnsconverts it. - Name what the layer padded with
assert_join_padding, which warns, or refuses understrict. - Put a caller's
AssetPanelon the emitted clock withproject_panel_clock. A caller states a Panel Field on the clock of the table they hold, and the join and the collapse move that clock, so the projection happens here. Each period of a collapse takes the Panel Field values of the last row of its group, asLastRowdoes. The timestamp function ofcollapse_argsdoes not change this row, so under(Dates.week, first, last)the prices and the Panel Field values of a week come from one day. A static panel has no observation axis, and the layer carries it through unchanged. - Read the Listing Span off the asset block with
listing_span, unless the caller declared one. A declared span replaces the answer of the Span Rule. - Return the
PricesResultthat carries all of it.
Under the default join the emitted clock is the asset table's, so timestamp(pr.X) == timestamp(X) unless collapse_args is not empty. A caller who declares a listing calendar can therefore size it against the table they hold.
Arguments
est: ThePriceIngestionestimator.X: Asset prices,observations × assets.F: Optional factor prices.B: Optional benchmark prices, one column or one per asset.E: Optional Exogenous Series, level series on any clock, such as a Currency Excess Index. The layer aligns it to the emitted clock and padsNaNwhere it is silent.iv: Optional implied volatilities, one column per asset, on any clock. The layer aligns them to the emitted clock and padsNaNwhere they are silent.ivpa: Optional implied volatility adjustment.pnl: OptionalAssetPanelof Panel Fields the caller already holds. The layer keeps its Panel Fields and does not read its masks. A time-varying panel carries anamskby construction, so that field cannot carry a declaration. A listing statement goes inspan.pr: APricesResult, for the second form. The layer ingests its series again. It keeps a span thatprstates unlessest.spanoverrides it, and the Span Rule runs only where neither states one.
Validation
!isempty(X). Raises anIsEmptyError.- The asset, factor and benchmark column names are pairwise disjoint, and none of them is
timestamp. Raises aConflictingArgumentErrornaming the offending columns. - At least one of
X,FandBhas an element type that is notMissingalone. Raises aDomainError, fromabsence_type. - A declared
spanhas the size of the asset block after the join and the collapse. Raises aDimensionMismatch. - The value type carries an absence wherever the layer might spell one. Raises a
DomainErrornaming the type, fromabsent_value. pnldescribes the assets ofX, and a time-varyingpnlalso its observations. Raises aDimensionMismatch, fromproject_panel_clock.- Each row that a time-varying
pnltakes is a row ofX. Without a collapse, this is each emitted row. With a collapse, it is the last joined row of each period. An outer join that adds a row breaks this. Raises anArgumentError, frommatched_row_indices. - The layer padded nothing, under
strict. Raises anArgumentErrorcarrying the padding report, fromassert_join_padding. Otherwise the layer warns.
Returns
pr::PricesResult: The price data, holding the Listing Span in itsspanfield.
Examples
julia> X = TimeArray(Date(2020, 1, 1):Day(1):Date(2020, 1, 4), [100.0 missing; 102.0 103.0; 101.0 104.0; 103.0 102.0], ["A", "B"]);julia> pr = price_ingestion(PriceIngestion(), X);julia> values(pr.X)[1, :]2-element Vector{Float64}: 100.0 NaNjulia> Matrix(pr.span)4×2 Matrix{Bool}: 1 0 1 1 1 1 1 1Related