Price ingestion: private API

PortfolioOptimisers.project_panel_clock — Function
project_panel_clock(pnl::Nothing, tsj, X::TimeSeries.TimeArray, ca::Tuple) -> nothing
project_panel_clock(pnl::AssetPanel, tsj, X::TimeSeries.TimeArray, ca::Tuple) -> AssetPanel

Put a caller's AssetPanel on the clock the ingestion emits.

A caller states a Panel Field on the clock of the table they hold. The join and the collapse of price_ingestion move that clock, so price_ingestion makes the projection. After it, the PricesResult states one clock and everything it holds is on that clock. A static panel has no observation axis, so feature_row_indices returns Colon() for it and the panel passes through unchanged.

A collapse puts many rows into one period. Each period takes the Panel Field values of the last row of its group, as LastRow does. The timestamp function of ca does not change this row. So under (Dates.week, first, last) a week pairs the prices of its last day with the Panel Field values of the same day.

Algorithm

  1. pnl is nothing: return nothing.
  2. Otherwise read the incoming clock ts_old and the asset names off X, and check the panel against them with check_asset_panel.
  3. Find the timestamp of the row that each emitted observation takes, giving ts_row. Without a collapse, ts_row is tsj. With a collapse, collapse the row positions of tsj with the period and timestamp functions of ca and the value function last. This gives the last row of each group, as TimeSeries.collapse forms the groups, and ts_row holds the timestamps of those rows.
  4. Find the rows of ts_row in the incoming clock with feature_row_indices, and view the panel over them with port_opt_view. The view takes the whole asset axis and its names, so that it also cuts a square tensor Panel Field on its label axis.

Arguments

  • pnl: The caller's AssetPanel, or nothing.
  • tsj: The timestamps of the joined table, before the collapse.
  • X: The asset table the caller passed. Its timestamps are the incoming clock, and its column names are the asset names.
  • ca: The collapse_args of the PriceIngestion. An empty tuple states no collapse.

Validation

  • The panel describes the assets of X, and a time-varying panel also its observations. Raises a DimensionMismatch.
  • Every entry of ts_row is a timestamp of X, when the panel is time-varying. Raises an ArgumentError, from matched_row_indices.

Returns

  • pnl′::Option{<:AssetPanel}: The panel on the emitted clock, or nothing.

Related

source
PortfolioOptimisers.unify_gaps — Function
unify_gaps(A::TimeSeries.TimeArray, ::Type{T} = absence_type(series_value_type(A))) -> TimeSeries.TimeArray

Spell every absent price of one series the one way the ingestion layer carries, in the value type T.

A source spells an absent price one of two ways. An outer join of ragged per-asset histories pads with NaN, and a wide table built from a tidy one leaves missing. The layer unifies both as NaN, which no deletion step reads. This defines a gap for every later piece, so it runs before the Span Rule and before the join.

The caller does not name the target type. The default of T derives it from the series alone with absence_type, and the conversion uses this default for a PricesResult built by hand. price_ingestion first promotes the asset, factor and benchmark series to one type, because the join lays them in one table, and passes that type as T.

Algorithm

  1. Refuse an infinite value with assert_no_infinite_price, which throws an error that names it. The Span Rule reads a non-finite cell as unpriced and the conversion reads it as a price, so the two would disagree about the cell.
  2. The values are already of type T, so their gaps are already NaN: return A untouched.
  3. At least one value is missing: rebuild the values. Map each missing to absent_value(T) and convert every other entry to T. This is the one place where a missing becomes an absence, so the refusal of a type that cannot hold one happens here, by name.
  4. Otherwise convert every entry to T. Nothing is absent, so the function spells nothing and does not ask whether T can spell an absence. This includes an element type that admits missing when no value is missing.

Arguments

  • A: One price series.
  • T: The value type to carry it in. Defaults to absence_type(series_value_type(A)).

Validation

  • Every value is missing, NaN, or finite. An infinity raises a DomainError naming the column and the observation.
  • T carries an absence when a value is missing. Raises a DomainError naming T, from absent_value.
  • With the default T: the element type of the series is not Missing alone. Raises a DomainError, from absence_type.

Returns

  • A′::TimeSeries.TimeArray: The same series in type T, with every absent price spelled NaN.

Related

source
PortfolioOptimisers.assert_no_infinite_price — Function
assert_no_infinite_price(v::AbstractArray, names, ts) -> nothing

Refuse an infinite value in a price series by name.

The layer spells an absent price NaN, and an infinity is not that spelling. Two pieces of the layer read an infinity differently. listing_span reads a non-finite cell as unpriced, so the Span Rule keeps an interior infinity inside the listing. The conversion reads the same cell as a price, and the return that follows it is a finite -100 %. This function refuses the value where the layer fixes the spelling, so every later piece reads one definition of a gap.

Algorithm

  1. Find the first value that is not missing, not NaN and not finite, giving k.
  2. No such value: return.
  3. Read the observation and the column of k from its Cartesian index. A one-column series can hold its values in a vector, and then the column is the first.
  4. Raise a DomainError that names the value, the column and the timestamp.

Arguments

  • v: The values of one series, observations × columns.
  • names: The series' column names.
  • ts: The series' timestamps.

Validation

  • Every value is missing, NaN, or finite. Raises a DomainError naming the first offending column and observation.

Returns

  • nothing.

Related

source
PortfolioOptimisers.series_value_type — Function
series_value_type(A::Nothing) -> Type{Union{}}
series_value_type(A::TimeSeries.TimeArray) -> Type

Read the value type of one optional price series, without the Missing a wide table may carry.

The ingestion layer derives the type it carries prices in from the series themselves. Each series contributes the type of its values without Missing, because the target type spells an absence and the source does not. A series the caller omitted contributes Union{}, the identity of promote_type, so the site that promotes needs no branch for an absent table.

Algorithm

The method that Julia selects is the algorithm.

  1. A is nothing: return Union{}.
  2. A is a table: return the element type of its values with Missing removed. A table whose element type is Missing alone gives Union{} too, so it contributes nothing to the promotion.

Arguments

  • A: One price series, or nothing.

Returns

  • T::Type: The value type, Union{} for no series.

Related

source
PortfolioOptimisers.absence_type — Function
absence_type(::Type{T}) -> Type
absence_type(::Type{Union{}}) -> Union{}

Derive the type the ingestion layer carries a series of value type T in.

The layer spells an absent price NaN, so the carried type must be able to hold one. The layer reads that type off the data and does not name it: the type is float_if_integer of T. A floating-point type gives itself, so a Float32 panel stays Float32. An integer type gives its floating-point type, Float64 for Int and BigFloat for BigInt. A Rational and a number type the library does not know give themselves.

This function does not decide whether the type can hold an absence, because a series with no gap and no padding never spells one. absent_value throws an error that names the type when the layer writes an absence in it.

Algorithm

The method that Julia selects is the algorithm.

  1. T is Union{}: refuse. Every price table holds only missing, so no value type exists to derive.
  2. Otherwise return float_if_integer of T.

Arguments

Validation

  • T !== Union{}, which series_value_type gives for a table whose element type is Missing alone. Raises a DomainError.

Returns

Related

source
PortfolioOptimisers.absent_value — Function
absent_value(::Type{T}) -> T

Return the one spelling of an absent value in type T, and throw an error that names a type that cannot hold one.

The layer spells an absence one way, NaN, so a type that cannot represent NaN cannot state a gap. This function refuses such a type, for example a Rational panel that holds a gap or that a join must pad. Without it, a conversion raises an InexactError that names nothing. The refusal names the type and what the caller can do. An integer panel never reaches the refusal, because absence_type widens it to a floating-point type first.

The layer stays open to number types it does not know. A type carries an absence when convert(T, NaN) returns a value for which isnan holds.

Algorithm

  1. Convert NaN to T. A conversion that throws gives nothing.
  2. Refuse T unless the conversion gave a value and isnan holds for it.
  3. Return that value.

Arguments

  • T: The type an absence must be spelled in.

Validation

  • convert(T, NaN) succeeds and isnan of it holds. Otherwise raises a DomainError naming T.

Returns

  • x::T: The absence, convert(T, NaN).

Related

source
PortfolioOptimisers.assert_pad_spellable — Function
assert_pad_spellable(::Type{T}, method::Symbol) -> nothing

Before a join that can pad, throw an error that names a value type that cannot spell the pad.

TimeSeries.merge pads with NaN converted to the table's value type, which has the same bits as absent_value(T). For a type that cannot hold an absence, merge raises an InexactError that names nothing. This function asks first, so the refusal names the type. It asks before every join that can pad, whether or not the join then pads a row. An inner join drops an observation and does not pad it, so it asks for nothing.

Algorithm

  1. method is :inner: return.
  2. Otherwise ask absent_value(T), which throws an error that names the type.

Arguments

  • T: The value type of the tables the layer joins, the layer's unification target.
  • method: The join, as TimeSeries.merge takes it.

Validation

  • T carries an absence when method is not :inner. Raises a DomainError naming T, from absent_value.

Returns

  • nothing.

Related

source
PortfolioOptimisers.align_series — Function
align_series(A::TimeSeries.TimeArray{T}, ts::AbstractVector) -> TimeSeries.TimeArray

Put one carried series on the clock the ingestion layer emits, and pad the observations where it is silent.

The layer carries the implied volatilities beside the assets and does not join them into the asset table, because they carry the asset names and a join would rename them. The alignment works as a left join works for a factor. The emitted clock decides. The function keeps each observation the series states, and each observation the series does not state becomes an absence that the layer carries.

Algorithm

  1. Fill a table of length(ts) rows and the series' columns with absent_value of its type. This asks for an absence even when the series covers the clock, as a join that can pad does.
  2. Write each row of the series whose timestamp is in ts at that timestamp's row.

Arguments

  • A: The series to align.
  • ts: The emitted clock.

Validation

  • T carries an absence. Raises a DomainError naming T, from absent_value.

Returns

  • A′::TimeSeries.TimeArray: The series on ts, NaN where it was silent.

Related

source
PortfolioOptimisers.padded_observations — Function
padded_observations(A::Nothing, ts::AbstractVector) -> Int
padded_observations(A::TimeSeries.TimeArray, ts::AbstractVector) -> Int

Count the observations of the emitted clock at which one series is silent.

The join or the alignment padded each of those observations, and the layer carries each padded observation as an absence and names it. A series the caller omitted is silent nowhere.

Algorithm

The method that Julia selects is the algorithm.

  1. A is nothing: 0.
  2. A is a table: the number of timestamps in ts absent from A's.

Arguments

  • A: One price series, or nothing.
  • ts: The clock the layer emitted.

Returns

  • n::Int: The number of padded observations.

Related

source
PortfolioOptimisers.padding_report_line — Function
padding_report_line(name::String, A::Option{<:TimeSeries.TimeArray}, ts::AbstractVector) -> Option{String}

Write the line of the padding report that one series owes, or nothing when the layer padded it nowhere.

Algorithm

  1. Count the padded observations with padded_observations.
  2. None: return nothing.
  3. Otherwise write a line that names the table, the count out of the clock's length, and every column. A series silent at an observation is silent in all its columns.

Arguments

  • name: How the caller spells the series, "X", "F", "B" or "iv".
  • A: The series, or nothing.
  • ts: The clock the layer emitted.

Returns

  • line::Option{String}: The report line, or nothing.

Related

source
PortfolioOptimisers.assert_join_padding — Function
assert_join_padding(lines::AbstractVector, strict::Bool) -> nothing

Name what the join and the alignment padded, warning by default and refusing under strict.

A carried absence raises no error later. A fit that reads a padded factor or a padded one-column benchmark drops that observation, and a fit that reads a padded implied volatility or a padded column of a per-asset benchmark drops that asset, see shared_series_view and coverage_mask. Neither drop names its cause. So this function names it through strict_diagnostic, which a Held Gap and PriceGapFill also use to report. A caller who runs a walk-forward and sets strict knows that the layer padded no covariate.

Algorithm

  1. Drop the nothing entries. None is left, so the layer padded nothing: return.
  2. Otherwise join the lines into one report that names the three join methods, and pass it to strict_diagnostic.

Arguments

  • lines: The report lines padding_report_line wrote, nothing entries included.
  • strict: If true, throws an ArgumentError; if false, issues a warning.

Validation

  • The layer padded nothing. Raises an ArgumentError under strict.

Returns

  • nothing.

Related

source
PortfolioOptimisers.series_names — Function
series_names(A::Nothing) -> Vector{Symbol}
series_names(A::TimeSeries.TimeArray) -> Vector{Symbol}

Name the columns of one optional price table, and name none when the caller passed none.

A series the caller omitted contributes no column name. It contributes an empty list, so no site that reads the names needs an isnothing branch.

Algorithm

The method that Julia selects is the algorithm.

  1. A is nothing: return an empty name list.
  2. A is a table: return its column names.

Arguments

  • A: One price table, or nothing.

Returns

  • n::Vector{Symbol}: The table's column names, empty when there is no table.

Related

source
PortfolioOptimisers.assert_disjoint_series_names — Function
assert_disjoint_series_names(a::AbstractVector{Symbol}, b::AbstractVector{Symbol}, na::String, nb::String) -> nothing

Refuse a column name that two of the price tables both carry.

A column name says which series a column came from. The layer merges the asset, factor and benchmark tables onto one clock and then splits the blocks by name. TimeSeries.merge renames the second of two columns that share a name by appending _1. So a shared name makes one block take the other's column with no error, and the new name belongs to no block. The conversion also cannot choose between the two columns. X's AAPL and F's AAPL are different series, and to keep either one is worse than to refuse both.

Arguments

  • a: Column names of the first table.
  • b: Column names of the second table.
  • na: How the caller spells the first table.
  • nb: How the caller spells the second table.

Validation

Returns

  • nothing.

Related

source
PortfolioOptimisers.assert_unreserved_series_names — Function
assert_unreserved_series_names(n::AbstractVector{Symbol}, nn::String) -> nothing

Refuse a series named after the observation clock.

The conversion writes the clock into a column named timestamp. A series with the same name then becomes timestamp_1, and the clock keeps the name. So a lookup of the series by its name reads the dates as prices. The layer owns the name, and a caller who holds a series with that name must rename it.

Arguments

  • n: Column names of one table.
  • nn: How the caller spells that table.

Validation

Returns

  • nothing.

Related

source
PortfolioOptimisers.assert_distinct_series_names — Function
assert_distinct_series_names(X::TimeSeries.TimeArray, F::Option{<:TimeSeries.TimeArray} = nothing, B::Option{<:TimeSeries.TimeArray} = nothing, E::Option{<:TimeSeries.TimeArray} = nothing) -> nothing

Check that every price series reaching the layer can still be named after the join.

Both entry points of the layer run this check before they merge, price_ingestion and prices_to_returns. After it, every later piece can split the merged table by name. Each name belongs to exactly one of the asset, factor and benchmark blocks, and no name is the clock's.

The Exogenous Series E is converted beside the three blocks, and a consumer selects its columns by name. So its names are also disjoint from the three blocks, and no name is the clock's.

This function cannot check two series of one table. The TimeSeries.TimeArray constructor runs TimeSeries.replace_dupes! over its column names, so it renames the duplicates of a table before the table exists, and no duplicate reaches this function.

Algorithm

  1. Read the four name lists, an absent table naming none, with series_names.
  2. Refuse a name shared by two of them with assert_disjoint_series_names, over all six pairs.
  3. Refuse the clock's own name in any of the four with assert_unreserved_series_names.

Arguments

  • X: Asset prices, observations × assets.
  • F: Optional factor prices.
  • B: Optional benchmark prices.
  • E: Optional Exogenous Series.

Validation

Returns

  • nothing.

Related

source
PortfolioOptimisers.assert_span_shape — Function
assert_span_shape(span::Nothing, nobs, na) -> nothing
assert_span_shape(span::AbstractMatrix{Bool}, nobs::Integer, na::Integer) -> nothing

Check that a Listing Span has the shape of the price panel it describes.

A span states which assets are listed at each observation of the price clock, so it has the shape of the asset prices. A PricesResult that holds no span states no universe, and there is nothing to check.

Algorithm

The method that Julia selects is the algorithm.

  1. span is nothing: the PricesResult states no universe, so there is nothing to check.
  2. Otherwise check the span's shape against the price panel's.

Arguments

  • span: The Listing Span on the price clock, or nothing.
  • nobs: Observation count of the price panel.
  • na: Asset count of the price panel.

Validation

  • size(span) == (nobs, na). Raises a DimensionMismatch.

Returns

  • nothing.

Related

source
PortfolioOptimisers.returns_universe_masks — Function
returns_universe_masks(span::Nothing, R::AbstractMatrix)
returns_universe_masks(span::AbstractMatrix{Bool}, R::AbstractMatrix)

Derive the two universe masks that the AssetPanel of the ReturnsResult states.

A PricesResult from the layer arrives with a Listing Span. This function projects the span onto the returns clock and intersects it with finiteness. A PricesResult built outside the layer arrives without a span and states no universe, whether or not its prices hold a gap. The function does not derive a span for it, because a derivation from one window reads a delisting that straddles the window end as an asset that was never listed. The library still handles the gaps of such a PricesResult, because with no panel the Coverage Universe reads finiteness alone. So pnl === nothing means that the returns data states no universe beyond the finiteness of its returns: the layer did not build the PricesResult, or the prices held no gap.

Algorithm

The method that Julia selects is the algorithm.

  1. span is nothing: return nothing, nothing.
  2. span is given: project and intersect it with universe_masks, then compress the pair with compress_all_true.

Arguments

  • span: The Listing Span over the surviving price rows and assets, or nothing.
  • R: The returns panel the conversion produced, observations × assets.

Returns

  • (amsk, emsk): The two masks, or nothing, nothing when the PricesResult states no universe.

Related

source
PortfolioOptimisers.compress_all_true — Function
compress_all_true(amsk::AbstractMatrix{Bool}, emsk::AbstractMatrix{Bool})

Store a pair of universe masks that is true everywhere as the constant it is.

A gapless ingestion states a universe in which every asset is listed at every observation and every return is finite, so both masks are true everywhere. A PortfolioOptimisers.AllTrueMask stores that in two integers. So a panel that carries such masks beside its Panel Fields costs O(1) memory for them, not one bit per observation and asset. With no Panel Field, attach_universe_masks reads the type and emits no panel at all. The estimation mask is a subset of the active mask. When the estimation mask is all true, the active mask is all true too, so the function tests the estimation mask alone.

Algorithm

  1. Some entry of emsk is false: return the two masks unchanged.
  2. Otherwise return one PortfolioOptimisers.AllTrueMask of the same size as both masks.

Arguments

  • amsk: The active mask.
  • emsk: The estimation mask.

Returns

  • (amsk, emsk): The two masks, compressed when they admit it.

Related

source
PortfolioOptimisers.attach_universe_masks — Function
attach_universe_masks(pnl, amsk::Nothing, emsk::Nothing)
attach_universe_masks(pnl::Option{<:AssetPanel{Nothing}}, amsk::AllTrueMask, emsk::AllTrueMask) -> nothing
attach_universe_masks(pnl::Option{<:AssetPanel{Nothing}}, amsk::AbstractMatrix{Bool}, emsk::AbstractMatrix{Bool}) -> AssetPanel
attach_universe_masks(pnl::AssetPanel{<:AbstractVector}, amsk::AbstractMatrix{Bool}, emsk::AbstractMatrix{Bool}) -> AssetPanel

Put the two universe masks onto the AssetPanel that the ReturnsResult holds.

The layer emits one ReturnsResult, and the masks go on it. A Pipeline step has one output, and PricesToReturns cannot emit the masks by another route without putting them on the price clock, where the masks are not stated. Because the ReturnsResult holds the masks, port_opt_view slices the universe together with the returns and needs no extra code.

The function keeps a caller's Panel Fields and replaces both masks with the layer's. The active mask cannot be a declaration that a panel brings into the layer. A time-varying panel holds an amsk by construction, and asset_panel writes an all-true one when the caller states none. So an all-true amsk beside a time-varying Panel Field looks the same as a declared calendar in which every asset is listed. The only way to state a listing is span on PriceIngestion. The estimation mask states a fact about the data and not about the instruments, so the conversion derives it again in every case with universe_masks, and emsk ⊆ amsk holds by construction.

Algorithm

The method that Julia selects is the algorithm.

  1. No masks: return the panel unchanged. The PricesResult states no universe.
  2. All-true masks and no Panel Field: return nothing. Every asset is listed and every return is finite, so the masks state nothing that the finiteness of the returns does not state, and the Coverage Universe of no panel is the same.
  3. Other masks and no panel: return an AssetPanel of the two masks and pf = nothing. This is the layer's common case when the prices hold a gap: a caller holding only prices has no feature data.
  4. Masks and a panel with no Panel Field: apply steps 2 and 3. The function replaces both masks, so the panel brings nothing else.
  5. Masks and a panel with Panel Fields: keep its Panel Fields and replace both its masks. A static panel's fields carry no observation axis, so the function first lifts them onto the clock of the masks with panel_field_lift.

Arguments

  • pnl: The Asset Panel that the PricesResult held, or nothing.
  • amsk: The active mask, or nothing.
  • emsk: The estimation mask, or nothing.

Returns

  • pnl′::Option{<:AssetPanel}: The panel that the ReturnsResult holds, or nothing when there is no panel or the panel states nothing.

Related

source
PortfolioOptimisers.listing_span_view — Function
listing_span_view(span::Nothing, ts_new, ts_old, j) -> nothing
listing_span_view(span::AbstractMatrix{Bool}, ts_new, ts_old, j) -> SubArray
listing_span_view(span::ListingSpan, ts_new, ts_old, j) -> Union{ListingSpan, SubArray}

View the Listing Span of a PricesResult over the surviving timestamps and the assets j, or return nothing when the PricesResult holds none.

The span states a fact about the instruments, so the span of a window is a view of the panel-wide span, never a span derived again from the window. A span derived from a window reads a delisting that straddles the window end as an asset that was never listed. A view is exact, and over a PortfolioOptimisers.ListingSpan it stores no boolean cell.

The span is parallel to the price clock by position. So the function finds its rows from the surviving timestamps with matched_row_indices, as it does for a time-varying AssetPanel.

Algorithm

The method that Julia selects is the algorithm.

  1. span is nothing: return nothing, and match no timestamp.
  2. span is a matrix: find its rows with matched_row_indices, and view it at those rows and the assets j.
  3. span is a PortfolioOptimisers.ListingSpan and the rows are a contiguous window in clock order, for example a fold's window or the whole clock. Subtract the number of rows before the window from both bounds, keep the bounds of the assets j, and return a new ListingSpan of the window's length. The result keeps two integers per asset and does not expand them into a view of booleans. A bound outside the window states the same listings inside it as a bound at its edge. Any other row selection can split an interval in two, which one interval cannot state, so it takes step 2.

Arguments

  • span: The Listing Span on the price clock, or nothing.
  • ts_new: Timestamps that survived the selection.
  • ts_old: Timestamps of the price clock the span is parallel to.
  • j: Asset indices, a vector or Colon().

Returns

  • s::Option{<:AbstractMatrix{Bool}}: A new ListingSpan for a contiguous window of a ListingSpan, a view of span for any other selection, or nothing.

Related

source