Price ingestion: private API

project_panel_clock(pnl::Nothing, ts_new, ts_old, nx::VecStr) -> nothing
project_panel_clock(pnl::AssetPanel, ts_new, ts_old, nx::VecStr) -> AssetPanel

Put a caller's AssetPanel on the clock the ingestion emits.

A caller states a Panel Field on the clock of the table they hold, and the collapse is the one step of price_ingestion that renumbers an observation, so the projection is owed at the door: after it, the carrier states one clock and everything the carrier holds is on it. A static panel has no observation axis, so feature_row_indices answers Colon() for one and it rides through unchanged.

Algorithm

1. A carrier holding no panel projects to none.
2. Otherwise check the panel against the asset axis and the incoming clock with check_asset_panel.
3. Recover the emitted clock's rows in the incoming one with feature_row_indices, and view the panel over them with port_opt_view. The asset axis is whole, and it is still named so that a square tensor Panel Field is cut on its label axis too.

Arguments

• pnl: The caller's AssetPanel, or nothing.
• ts_new: The timestamps the ingestion emits.
• ts_old: The timestamps of the asset table the caller handed in.
• nx: The asset names.

Validation

• The panel describes length(nx) assets and length(ts_old) observations. Raises a DimensionMismatch.

Returns

• pnl′::Option{<:AssetPanel}: The panel on the emitted clock, or nothing.

Related

• price_ingestion
• AssetPanel
• feature_row_indices
• port_opt_view
• check_asset_panel
• LastObservation

unify_gaps(A::TimeSeries.TimeArray) -> TimeSeries.TimeArray
unify_gaps(A::TimeSeries.TimeArray, ::Type{T}) -> TimeSeries.TimeArray

Spell every absent price of one series the one way the ingestion layer carries, in the value type T.

A source spells an absent price either way — an outer join of ragged per-asset histories pads with NaN, a wide table built from a tidy one leaves missing — and the layer unifies them as NaN, which no deletion step reads. This is what defines a gap for every later piece, so it runs before the Span Rule and before the join.

The target type is derived, never named. The one-argument form reads it off the series alone with absence_type, which is what a carrier built by hand gets at the conversion; price_ingestion promotes the asset, factor and benchmark series to one type first, because the join lays them in one table, and hands that type to the two-argument form.

Algorithm

1. A series whose values are already of type T carries its gaps as NaN. Return it untouched.
2. A series holding missing is rebuilt, mapping missing to absent_value(T) and converting every other entry to T. This is the one place a missing becomes an absence, so a type that cannot carry one is refused here, by name.
3. Any other series is converted to T entry by entry. Nothing is absent, so nothing is spelled, and a type that could not spell one is not asked to.
4. Refuse an infinite value by name, in every case. An infinity is neither a price nor the marker of an absence: the Span Rule reads a non-finite cell as unpriced while the conversion would read it as a price and compute a finite return from it, so the two would disagree about one cell.

Arguments

• A: One price series.
• T: The value type to carry it in. Defaults to absence_type(series_value_type(A)).

Validation

• Every value is missing, NaN, or finite. An infinity raises a DomainError naming the column and the observation.

Returns

• A′::TimeSeries.TimeArray: The same series in type T, with every absent price spelled NaN.

Related

• PriceIngestion
• price_ingestion
• series_value_type
• absence_type
• absent_value
• listing_span

assert_no_infinite_price(v::AbstractArray, names, ts) -> nothing

Refuse an infinite value in a price series by name.

The layer has one spelling for an absent price, NaN, and an infinity is not it: listing_span reads a non-finite cell as unpriced, so the Span Rule would leave an interior infinity inside the listing, while the conversion would read it as a price and compute a finite return from the pair it forms — a -100 % on the observation after it. A value that two pieces of the layer read differently is refused where the spelling is fixed, so that every later piece meets one definition of a gap.

Arguments

• v: The values of one series, observations × columns.
• names: The series' column names.
• ts: The series' timestamps.

Validation

• Every value is missing, NaN, or finite. Raises a DomainError naming the first offending column and observation.

Returns

• nothing.

Related

• unify_gaps
• listing_span
• is_missing_value

series_value_type(A::Nothing) -> Type{Union{}}
series_value_type(A::TimeSeries.TimeArray) -> Type

Read the value type of one optional price series, without the Missing a wide table may carry.

The ingestion layer derives its unification target from the series it is handed rather than naming one, and this is what each series contributes to that derivation: the type of its values, with the absence convention stripped, because an absence is spelled by the target and not by the source. A series the caller omitted contributes Union{}, which is the identity of promote_type, so an absent table is not a branch at the site that promotes.

Algorithm

The method that Julia selects is the algorithm.

1. A is nothing: return Union{}.
2. A is a table: return the element type of its values with Missing removed.

Arguments

• A: One price series, or nothing.

Returns

• T::Type: The value type, Union{} for no series.

Related

• absence_type
• unify_gaps
• price_ingestion

absence_type(::Type{T}) -> Type

Derive the type the ingestion layer carries a series of value type T in.

The layer spells an absent price NaN, so the type it carries a series in must be able to hold one, and that type is read off the arithmetic rather than named: a return divides one price by another, so oneunit(T) / one(T) is the widening the conversion applies to every price anyway, and its type is the widest the layer needs. A floating-point type is its own answer, so a Float32 panel stays Float32; an integer panel takes the floating-point type that represents it; a number type the library has never seen takes whatever its own division returns. Whether that type can hold an absence is not decided here, because a series with no gap and no padding never spells one: absent_value refuses by name at the moment an absence is written.

Arguments

• T: The value type of a series, as series_value_type reads it.

Returns

• S::Type: typeof(oneunit(T) / one(T)).

Related

• series_value_type
• absent_value
• unify_gaps
• price_ingestion

absent_value(::Type{T}) -> T

The one spelling of an absent value in type T, or a refusal by name when T cannot carry one.

The layer has one spelling for absence, NaN, and a type that cannot represent it cannot state a gap. A Rational panel holding a gap, or an integer panel joined onto a longer clock, is refused here rather than by an InexactError from inside a conversion, and the refusal names the type and what the caller can do. The library stays open to number types it has never seen: any type whose convert from NaN answers a value isnan recognises carries an absence, and only a type that cannot is refused.

Arguments

• T: The type an absence must be spelled in.

Validation

• convert(T, NaN) succeeds and isnan of it holds. Otherwise raises a DomainError naming T.

Returns

• x::T: The absence, convert(T, NaN).

Related

• absence_type
• unify_gaps
• align_series
• price_ingestion

assert_pad_spellable(::Type{T}, method::Symbol) -> nothing

Refuse by name, before a join that pads, a value type that cannot spell the pad.

TimeSeries.merge pads with NaN converted to the table's value type, which is absent_value(T) to the bit, so a type that cannot carry an absence would be refused from inside it by an InexactError that names nothing. Asking first is what makes it a refusal by name. An inner join drops an observation rather than padding it, so it spells no absence and asks for none.

Algorithm

1. method is :inner: return.
2. Otherwise ask absent_value(T), which refuses by name.

Arguments

• T: The value type of the table being joined, the layer's unification target.
• method: The join, as TimeSeries.merge takes it.

Validation

• T carries an absence when method pads. Raises a DomainError naming T, from absent_value.

Returns

• nothing.

Related

• absent_value
• absence_type
• price_ingestion

align_series(A::TimeSeries.TimeArray{T}, ts::AbstractVector) -> TimeSeries.TimeArray

Put one carried series on the clock the ingestion layer emits, padding the observations it is silent at.

The implied volatilities are carried beside the assets rather than joined into their table, because they are named after the assets and a join would rename them. They are aligned the way a left join aligns a factor: the emitted clock is authoritative, an observation the series states is taken, and one it does not is an absence the layer carries.

Algorithm

1. Fill a table of length(ts) rows and the series' columns with absent_value of its type.
2. Write each row of the series whose timestamp is in ts at that timestamp's row.

Arguments

• A: The series to align, in a type T that carries an absence.
• ts: The emitted clock.

Returns

• A′::TimeSeries.TimeArray: The series on ts, NaN where it was silent.

Related

• absent_value
• padded_observations
• price_ingestion

padded_observations(A::Nothing, ts::AbstractVector) -> Int
padded_observations(A::TimeSeries.TimeArray, ts::AbstractVector) -> Int

Count the observations of the emitted clock at which one series is silent.

Each of those is an observation the join or the alignment padded, and a padded observation is an absence the layer carries and names. A series the caller omitted is silent nowhere.

Algorithm

The method that Julia selects is the algorithm.

1. A is nothing: 0.
2. A is a table: the number of timestamps in ts absent from A's.

Arguments

• A: One price series, or nothing.
• ts: The clock the layer emitted.

Returns

• n::Int: The number of padded observations.

Related

• padding_report_line
• assert_join_padding
• align_series

padding_report_line(name::String, A::Option{<:TimeSeries.TimeArray}, ts::AbstractVector) -> Option{String}

Write the line of the padding report that one series owes, or nothing when the layer padded it nowhere.

Algorithm

1. Count the padded observations with padded_observations.
2. None: return nothing.
3. Otherwise name the table, the count against the clock's length, and every column, because a series silent at an observation is silent in all of its columns.

Arguments

• name: How the caller spells the series, "X", "F", "B" or "iv".
• A: The series, or nothing.
• ts: The clock the layer emitted.

Returns

• line::Option{String}: The report line, or nothing.

Related

• padded_observations
• assert_join_padding

assert_join_padding(lines::AbstractVector, strict::Bool) -> nothing

Name what the join and the alignment padded, warning by default and refusing under strict.

A carried absence is invisible where a refused one was not, and the factor, benchmark and implied-volatility axes carry no universe that would name it later. So the layer names it here, through strict_diagnostic, which is the shape a Held Gap and a PriceGapFill report with. A caller running a walk-forward sets strict and is then certain no covariate was padded.

Algorithm

1. Drop the nothing entries. None left: nothing was padded, return.
2. Otherwise join the lines into one report naming the join in force and the two others, and hand it to strict_diagnostic.

Arguments

• lines: The report lines padding_report_line wrote, nothing entries included.
• strict: If true, throws an ArgumentError; if false, issues a warning.

Validation

• Nothing was padded. Raises an ArgumentError under strict.

Returns

• nothing.

Related

• padding_report_line
• strict_diagnostic
• PriceIngestion
• price_ingestion

series_names(A::Nothing) -> Vector{Symbol}
series_names(A::TimeSeries.TimeArray) -> Vector{Symbol}

Name the columns of one optional price table, and name none when the caller passed none.

A series the caller omitted contributes no column name, so it contributes an empty list rather than an isnothing branch at every site that reads one.

Algorithm

The method that Julia selects is the algorithm.

1. A is nothing: return an empty name list.
2. A is a table: return its column names.

Arguments

• A: One price table, or nothing.

Returns

• n::Vector{Symbol}: The table's column names, empty when there is no table.

Related

• assert_distinct_series_names
• price_ingestion
• prices_to_returns

assert_disjoint_series_names(a::AbstractVector{Symbol}, b::AbstractVector{Symbol}, na::String, nb::String) -> nothing

Refuse a column name that two of the price tables both carry.

A column name is what says which series a column came from: the asset, factor and benchmark tables are merged onto one clock, and the blocks are taken apart by name afterwards. TimeSeries.merge renames the second of two columns that share a name, appending _1, so a shared name silently makes one block take the other's column, and the minted name belongs to no block at all. A shared name is also not the conversion's to resolve — X's AAPL and F's AAPL are different series, and keeping either one is worse than refusing both.

Arguments

• a: Column names of the first table.
• b: Column names of the second table.
• na: How the caller spells the first table.
• nb: How the caller spells the second table.

Validation

• isdisjoint(a, b). Raises a ConflictingArgumentError naming the shared columns.

Returns

• nothing.

Related

• assert_distinct_series_names
• assert_unreserved_series_names
• series_names
• ConflictingArgumentError

assert_unreserved_series_names(n::AbstractVector{Symbol}, nn::String) -> nothing

Refuse a series named after the observation clock.

The conversion writes the clock into a column named timestamp, and a series of the same name takes that column's place: the clock keeps the name and the series is renamed timestamp_1, so the block the series belongs to reads the dates as prices. The name is the layer's, and a caller holding a series of that name renames it.

Arguments

• n: Column names of one table.
• nn: How the caller spells that table.

Validation

• :timestamp ∉ n. Raises a ConflictingArgumentError.

Returns

• nothing.

Related

• assert_distinct_series_names
• assert_disjoint_series_names
• ConflictingArgumentError

assert_distinct_series_names(X::TimeSeries.TimeArray, F::Option{<:TimeSeries.TimeArray} = nothing, B::Option{<:TimeSeries.TimeArray} = nothing) -> nothing

Check that every price series reaching the layer can still be named after the join.

This is the door's check, and both doors take it: price_ingestion before it merges, and prices_to_returns before it does. What it buys is that every later piece may split the merged table by name — a name belongs to exactly one of the asset, factor and benchmark blocks, and none of them is the clock's.

Two series of one table cannot be checked here. Every TimeSeries.TimeArray constructor runs TimeSeries.replace_dupes! over its column names, so a table's own duplicates are renamed before the table exists and no duplicate reaches this function.

Algorithm

1. Read the three name lists, an absent table naming none, with series_names.
2. Refuse a name shared by two of them with assert_disjoint_series_names, over all three pairs.
3. Refuse the clock's own name in any of them with assert_unreserved_series_names.

Arguments

• X: Asset prices, observations × assets.
• F: Optional factor prices.
• B: Optional benchmark prices.

Validation

• The asset, factor and benchmark names are pairwise disjoint. Raises a ConflictingArgumentError naming the shared columns.
• None of them is timestamp. Raises a ConflictingArgumentError.

Returns

• nothing.

Related

• assert_disjoint_series_names
• assert_unreserved_series_names
• series_names
• price_ingestion
• prices_to_returns
• ConflictingArgumentError

assert_span_shape(span::Nothing, nobs, na) -> nothing
assert_span_shape(span::AbstractMatrix{Bool}, nobs::Integer, na::Integer) -> nothing

Check that a Listing Span fits the price panel it rides on.

A span states which assets are listed at each observation of the price clock, so it is the shape of the asset prices. A carrier that states none states no universe, and there is nothing to check.

Algorithm

The method that Julia selects is the algorithm.

1. span is nothing: the carrier states no universe, so there is nothing to check.
2. Otherwise check the span's shape against the price panel's.

Arguments

• span: The Listing Span on the price clock, or nothing.
• nobs: Observation count of the price panel.
• na: Asset count of the price panel.

Validation

• size(span) == (nobs, na). Raises a DimensionMismatch.

Returns

• nothing.

Related

• prices_to_returns
• PricesToReturns
• price_ingestion
• returns_universe_masks

returns_universe_masks(span::Nothing, R::AbstractMatrix)
returns_universe_masks(span::AbstractMatrix{Bool}, R::AbstractMatrix)

Derive the two universe masks the returns carrier's AssetPanel states.

The layer's carrier arrives with a Listing Span, and this projects it onto the returns clock and intersects it with finiteness. A carrier built outside the layer arrives without one and states no universe, whether or not its prices hold a gap: a window-local derivation reads a delisting straddling the window end as an asset that was never listed, so it answers a question it cannot answer correctly. The gaps of such a carrier are still handled — with no panel the Coverage Universe reads finiteness alone — and pnl === nothing keeps its single meaning: the carrier was not built by the layer.

Algorithm

The method that Julia selects is the algorithm.

1. span is nothing: return nothing, nothing.
2. span is given: project and intersect it with universe_masks, then compress the pair with compress_all_true.

Arguments

• span: The Listing Span over the surviving price rows and assets, or nothing.
• R: The returns panel the conversion produced, observations × assets.

Returns

• (amsk, emsk): The two masks, or nothing, nothing when the carrier states no universe.

Related

• universe_masks
• compress_all_true
• attach_universe_masks
• prices_to_returns

compress_all_true(amsk::AbstractMatrix{Bool}, emsk::AbstractMatrix{Bool})

Store a pair of universe masks that is true everywhere as the constant it is.

A gapless ingestion states a universe in which every asset is listed at every observation and every return is finite, so both masks are true throughout. PortfolioOptimisers.AllTrueMask says exactly that in two integers, which is what makes always emit a panel cost O(1) rather than observations × assets bits. The estimation mask is a subset of the active one, so testing the estimation mask alone answers for both.

Algorithm

1. Every entry of emsk is true: return one PortfolioOptimisers.AllTrueMask as both masks.
2. Otherwise return the two masks unchanged.

Arguments

• amsk: The active mask.
• emsk: The estimation mask.

Returns

• (amsk, emsk): The two masks, compressed when they admit it.

Related

• PortfolioOptimisers.AllTrueMask
• universe_masks
• returns_universe_masks
• AssetPanel

attach_universe_masks(pnl, amsk::Nothing, emsk::Nothing)
attach_universe_masks(pnl::Nothing, amsk::AbstractMatrix{Bool}, emsk::AbstractMatrix{Bool}) -> AssetPanel
attach_universe_masks(pnl::AssetPanel, amsk::AbstractMatrix{Bool}, emsk::AbstractMatrix{Bool}) -> AssetPanel

Put the two universe masks onto the AssetPanel the returns carrier holds.

The layer emits one carrier, and the masks ride on it: a Pipeline step has one out-slot, and mask emission cannot leave PricesToReturns without landing on the price clock, where the masks are not stated. Carrying them makes port_opt_view slice the universe in step with the returns for free.

A caller's Panel Fields are kept. Both masks are the layer's to state. The active mask is not a declaration a panel can carry to the door: a time-varying panel holds an amsk by construction, and asset_panel writes an all-true one when the caller states none, so an all-true amsk beside a time-varying Panel Field cannot be told from a declared all-listed calendar. The one door for a listing statement is span on PriceIngestion. The estimation mask is a statement about the data rather than about the instruments, so it is re-derived in every case and emsk ⊆ amsk holds by construction rather than by refusal.

Algorithm

The method that Julia selects is the algorithm.

1. No masks: return the panel unchanged. The carrier states no universe, so pnl === nothing keeps its one meaning — the carrier was not built by the layer.
2. Masks and no panel: return an AssetPanel of the two masks and no Panel Field. This is the layer's common case: a caller holding only prices has no feature data.
3. Masks and a panel: keep its Panel Fields and replace both its masks. A static panel's fields carry no observation axis, so they are lifted onto the masks' clock with panel_field_lift first.

Arguments

• pnl: The Asset Panel the price carrier held, or nothing.
• amsk: The active mask, or nothing.
• emsk: The estimation mask, or nothing.

Returns

• pnl′::Option{<:AssetPanel}: The panel the returns carrier holds.

Related

• AssetPanel
• asset_panel: writes the all-true amsk a time-varying build carries when none is stated.
• PriceIngestion: its span is the door for a listing statement.
• returns_universe_masks
• panel_field_lift
• prices_to_returns

span_carrier_view(span::Nothing, ts_new, ts_old, j) -> nothing
span_carrier_view(span::AbstractMatrix{Bool}, ts_new, ts_old, j) -> SubArray
span_carrier_view(span::ListingSpan, ts_new, ts_old, j)

View a price carrier's Listing Span over the surviving timestamps and the assets j, or return nothing when the carrier holds none.

The span is a statement about the instruments, so a window's span is the panel-wide one viewed, never one re-derived from the window: re-deriving reads a delisting that straddles the window end as an asset that was never listed, which is the divergence a panel-wide derivation exists to avoid. A view is exact and, over a PortfolioOptimisers.ListingSpan, allocates no cell.

The span is held positionally parallel to the price clock, so its rows are recovered from the surviving timestamps the same way a time-varying AssetPanel's are, by matched_row_indices.

Algorithm

The method that Julia selects is the algorithm.

1. span is nothing: return nothing, without matching any timestamp.
2. span is a matrix: recover its rows with matched_row_indices, and view it at those rows and the assets j.
3. span is a PortfolioOptimisers.ListingSpan and the recovered rows are a contiguous window in clock order — a fold's window, or the whole clock: shift the two bounds by the rows dropped in front and subset them at j, so the two integers per asset survive the cut rather than being expanded into a view of booleans. Any other row selection can split an interval in half, which no interval can say, and falls back to step 2.

Arguments

• span: The Listing Span on the price clock, or nothing.
• ts_new: Timestamps that survived the selection.
• ts_old: Timestamps of the price clock the span is parallel to.
• j: Asset index.

Returns

• A view of span, or nothing.

Related

• PricesResult
• PortfolioOptimisers.ListingSpan
• listing_span
• matched_row_indices
• port_opt_view