Asset Panel builder

abstract type AbstractPanelFieldInput <: AbstractEstimator

Supertype of the raw, blank-carrying forms one Panel Field enters asset_panel in.

All concrete types holding one Panel Field's raw values, its metadata and its fill policy should subtype AbstractPanelFieldInput.

An input is not a carrier and never becomes one. It holds the blanks, and asset_panel resolves them on the way into the Panel Field it builds; nothing downstream ever sees an unresolved panel. This is why the blank-carrying form is a plain argument to the builder rather than a preprocessing estimator: an estimator fitted inside a fold would need a carrier for the unfilled panel, and the finiteness rule on every Panel Field gives it none.

Interfaces

In order to implement a new concrete type that works seamlessly with the library, subtype AbstractPanelFieldInput and implement the following methods:

panel_input_is_static

• panel_input_is_static(inp::AbstractPanelFieldInput) -> Bool: Whether the raw values carry no observation axis.

Arguments

• inp: The concrete subtype instance.

Returns

• static::Bool: true when the input is static.

panel_resolve

• panel_resolve(inp::AbstractPanelFieldInput) -> Tuple: The Panel Field's values with every blank resolved, and the boolean array recording which cells were observed.

Arguments

• inp: The concrete subtype instance.

Returns

• vals::AbstractArray: The resolved values, in the raw input's own shape.
• obs::AbstractArray{Bool}: Whether each raw cell was observed, the same shape as vals.

panel_input_field

• panel_input_field(inp::AbstractPanelFieldInput, vals::AbstractArray, obs::BitArray) -> AbstractPanelField: Build the Panel Field from the resolved values, with obs as its observed mask unless the fill policy is NoPanelFill, which admits no blank and so records no mask.

Arguments

• inp: The concrete subtype instance.
• vals: The resolved values, as panel_resolve returned them.
• obs: The observed cells, as panel_resolve returned them.

Returns

• f::AbstractPanelField: The Panel Field.

Related

• NumericPanelInput
• CategoricalPanelInput
• TensorPanelInput
• asset_panel
• panel_input_is_static
• panel_resolve
• panel_input_field
• AbstractPanelField

abstract type AbstractPanelFillAlgorithm <: AbstractAlgorithm

Supertype of the policies that resolve a blank cell of a raw Panel Field.

All concrete types stating what a Panel Field's blank cell becomes should subtype AbstractPanelFillAlgorithm.

A blank never reaches a carrier. asset_panel resolves every one of them, so every Panel Field comes out finite. The policy says what the resolved value is; the observed mask the Panel Field also carries says which cells the resolution touched.

Interfaces

In order to implement a new concrete type that works seamlessly with the library, subtype AbstractPanelFillAlgorithm and implement the following methods:

panel_fill

• panel_fill(alg::AbstractPanelFillAlgorithm, v::AbstractVector, name::AbstractString) -> Vector: One asset's column of one raw Panel Field, along the observation axis, with every blank resolved.

Arguments

• alg: The concrete subtype instance.
• v: One asset's raw values along the observation axis, blanks included.
• name: The Panel Field's name, displayed in an error message.

Returns

• filled::Vector: The same length as v, and free of blanks.

Related

• NoPanelFill
• ConstantPanelFill
• ForwardPanelFill
• BackwardPanelFill
• asset_panel

struct NumericPanelInput{__T_name, __T_vals, __T_alg} <: AbstractPanelFieldInput

Raw form of a Panel Field holding one numeric quantity per observation and asset.

Fields

• name: The Panel Field's name, which names its column of a derived Feature Matrix.

• vals: Raw values, blanks included: assets when static, observations × assets when time-varying. A blank is a missing, a nothing or a NaN.

• alg: The policy that resolves the blanks.

Constructors

NumericPanelInput(;    name::AbstractString,    vals::AbstractArray,    alg::AbstractPanelFillAlgorithm = NoPanelFill()) -> NumericPanelInput

Keywords correspond to the struct's fields.

Validation

• !isempty(name).
• !isempty(vals).

Examples

julia> NumericPanelInput(; name = "mcap", vals = [1.0 2.0; 3.0 4.0])NumericPanelInput  name ┼ String: "mcap"  vals ┼ 2×2 Matrix{Float64}   alg ┴ NoPanelFill()

Related

• AbstractPanelFieldInput
• NumericPanelField
• CategoricalPanelInput
• TensorPanelInput
• asset_panel

struct CategoricalPanelInput{__T_name, __T_vals, __T_levels, __T_alg} <: AbstractPanelFieldInput

Raw form of a Panel Field holding one category label per observation and asset.

Fields

• name: The Panel Field's name, which prefixes each of its columns of a derived Feature Matrix.

• vals: Raw labels, blanks included: assets when static, observations × assets when time-varying. A blank is a missing or a nothing.

• levels: The category levels, in column order, or nothing to read them off the resolved labels in sorted order.

• alg: The policy that resolves the blanks. A val it carries must itself be a level.

Constructors

CategoricalPanelInput(;    name::AbstractString,    vals::AbstractArray,    levels::Option{<:VecStr} = nothing,    alg::AbstractPanelFillAlgorithm = NoPanelFill()) -> CategoricalPanelInput

Keywords correspond to the struct's fields.

Validation

• !isempty(name).
• !isempty(vals).
• If levels is not nothing, it passes assert_panel_labels.
• A fill policy that carries a val carries a string. See assert_categorical_fill.

Examples

julia> CategoricalPanelInput(; name = "sector", vals = ["T" "E"; "T" "E"])CategoricalPanelInput    name ┼ String: "sector"    vals ┼ 2×2 Matrix{String}  levels ┼ nothing     alg ┴ NoPanelFill()

Related

• AbstractPanelFieldInput
• CategoricalPanelField
• NumericPanelInput
• TensorPanelInput
• asset_panel
• assert_categorical_fill
• Option
• VecStr

struct TensorPanelInput{__T_name, __T_vals, __T_axis, __T_labels, __T_groups, __T_alg} <: AbstractPanelFieldInput

Raw form of a Panel Field whose third axis carries its own labels, and optionally its own groups.

Fields

• name: The Panel Field's name, which prefixes each of its columns of a derived Feature Matrix.

• vals: Raw values, blanks included: assets × labels when static, observations × assets × labels when time-varying. A blank is a missing, a nothing or a NaN.

• axis: Name of what the third axis represents, such as "factor".

• labels: Labels of the third-axis entries, one per third-axis entry of vals.

• groups: Optional group of each third-axis entry, such as a Factor Family, one per label.

• alg: The policy that resolves the blanks.

Constructors

TensorPanelInput(;    name::AbstractString,    vals::AbstractArray,    axis::AbstractString,    labels::VecStr,    groups::Option{<:VecStr} = nothing,    alg::AbstractPanelFillAlgorithm = NoPanelFill()) -> TensorPanelInput

Keywords correspond to the struct's fields.

Validation

• !isempty(name).
• !isempty(vals).
• size(vals, 3) == length(labels).
• axis, labels and groups are checked by TensorPanelField, which this input builds.

Examples

julia> TensorPanelInput(; name = "beta", vals = ones(2, 2, 1), axis = "factor", labels = ["size"])TensorPanelInput    name ┼ String: "beta"    vals ┼ Array{Float64, 3}: [1.0 1.0; 1.0 1.0;;;]    axis ┼ String: "factor"  labels ┼ Vector{String}: ["size"]  groups ┼ nothing     alg ┴ NoPanelFill()

Related

• AbstractPanelFieldInput
• TensorPanelField
• NumericPanelInput
• CategoricalPanelInput
• asset_panel
• Option
• VecStr

struct NoPanelFill <: AbstractPanelFillAlgorithm

Refuses a blank cell instead of resolving one.

This is the default, and it is the right policy for a Panel Field that is complete by construction. A Panel Field carrying this policy contributes no observed-mask column, because it has nothing to record: every cell is observed or the build throws.

Constructors

NoPanelFill() -> NoPanelFill

Examples

julia> NoPanelFill()NoPanelFill()

Related

• AbstractPanelFillAlgorithm
• ConstantPanelFill
• ForwardPanelFill
• BackwardPanelFill
• asset_panel

struct ConstantPanelFill{__T_val} <: AbstractPanelFillAlgorithm

Resolves every blank cell to one constant.

This is the policy for a quantity whose absence means a value: a zero dividend before the first payment, or a residual category before a classification exists.

Fields

• val: The value every blank cell becomes: a number for a numeric or tensor Panel Field, a level label for a categorical one.

Constructors

ConstantPanelFill(;    val::Union{<:Number, <:AbstractString} = 0.0) -> ConstantPanelFill

Keywords correspond to the struct's fields.

Validation

• If val is a Number, isfinite(val). A non-finite fill would put an infinity into a Panel Field, which assert_panel_finite refuses.

Examples

julia> ConstantPanelFill()ConstantPanelFill  val ┴ Float64: 0.0

Related

• AbstractPanelFillAlgorithm
• NoPanelFill
• ForwardPanelFill
• asset_panel

struct ForwardPanelFill{__T_val, __T_lim} <: AbstractPanelFillAlgorithm

Resolves a blank cell to the nearest earlier observed value of the same asset.

This is the safe fill over a cross-validation fold. It looks backward along the observation axis, so a fold that starts later reads only rows that fold already holds, and the value a fold computes does not depend on rows outside it.

A leading blank has no earlier value to take, so it falls through to val.

Fields

• val: The value a cell becomes when the fill reaches it and no earlier observed value is available, or the run of blanks is longer than lim.

• lim: Longest run of consecutive blanks the fill carries a value across, or nothing for no limit.

Constructors

ForwardPanelFill(;    val::Union{<:Number, <:AbstractString} = 0.0,    lim::Option{<:Integer} = nothing) -> ForwardPanelFill

Keywords correspond to the struct's fields.

Validation

• If val is a Number, isfinite(val).
• If lim is not nothing, lim > 0.

Examples

julia> ForwardPanelFill()ForwardPanelFill  val ┼ Float64: 0.0  lim ┴ nothing

Related

• AbstractPanelFillAlgorithm
• BackwardPanelFill
• ConstantPanelFill
• asset_panel

struct BackwardPanelFill{__T_val, __T_lim} <: AbstractPanelFillAlgorithm

Resolves a blank cell to the nearest later observed value of the same asset.

Fields

• val: The value a cell becomes when the fill reaches it and no later observed value is available, or the run of blanks is longer than lim.

• lim: Longest run of consecutive blanks the fill carries a value across, or nothing for no limit.

Constructors

BackwardPanelFill(;    val::Union{<:Number, <:AbstractString} = 0.0,    lim::Option{<:Integer} = nothing) -> BackwardPanelFill

Keywords correspond to the struct's fields.

Validation

• If val is a Number, isfinite(val).
• If lim is not nothing, lim > 0.

Examples

julia> BackwardPanelFill()BackwardPanelFill  val ┼ Float64: 0.0  lim ┴ nothing

Related

• AbstractPanelFillAlgorithm
• ForwardPanelFill
• ConstantPanelFill
• asset_panel

asset_panel(
    inputs::AbstractVector{<:AbstractPanelFieldInput};
    amsk::Option{<:AbstractMatrix{Bool}} = nothing,
    emsk::Option{<:AbstractMatrix{Bool}} = nothing
) -> AssetPanel

Build the AssetPanel a carrier holds, from the raw, blank-carrying form of each Panel Field.

This is the build seam. It takes each Panel Field's raw values with its fill policy, and it returns the panel alone: the panel owns the values, so there is nothing else for a carrier to be handed. The blanks stop here, and every Panel Field comes out finite.

The result goes straight into the keyword the carriers have, ReturnsResult(; nx = nx, X = X, pnl = asset_panel(inputs)), and the same keyword reaches prices_to_returns.

The static entry is the rank of the raw values. An input whose values carry no observation axis is a static input: a fundamentals table or a sector classification with no history is that shape. There ForwardPanelFill and BackwardPanelFill are refused, because there is no observation axis to carry a value along.

An input set that is static throughout, with no mask, builds a static panel. A static input that meets a time-varying input, or that meets a mask, is lifted: panel_build_observations reads the observation count the build takes, and panel_field_lift wraps the static values in a RepeatedLeading, which stores them once and indexes a leading observation axis. A lifted Panel Field carries no observed mask, because every cell of a static input was observed.

Algorithm

1. Check that inputs is not empty and that the Panel Field names are unique.
2. Check each input's fill policy against its shape, with assert_panel_input_fill.
3. Read the observation count the build takes, with panel_build_observations.
4. Resolve every input with panel_resolve, which fills its blanks and records the observed cells, and build its Panel Field with panel_input_field. Lift a static Panel Field of a time-varying build with panel_field_lift.
5. Return the panel with no mask when the build is static.
6. Otherwise fill in the masks that were not given, and return the panel. A missing active mask is all-true. A missing estimation mask is the active mask, because the estimation mask is a subset of the active mask and the only subset that needs no further information is the whole of it; an all-true default would break the subset rule at the first inactive cell. The AssetPanel constructor checks that every Panel Field shares one shape.

Arguments

• inputs: The raw Panel Fields, in the order their columns are derived in.
• amsk: The active mask (observations × assets), or nothing for all-true.
• emsk: The estimation mask (observations × assets), or nothing for the active mask.

Validation

• !isempty(inputs). Raises an IsEmptyError.
• The Panel Field names are non-empty and unique. See assert_panel_labels.
• A static input carries no directional fill policy. See assert_panel_input_fill.

Returns

• pnl::AssetPanel: The Asset Panel.

Examples

julia> pnl = asset_panel([NumericPanelInput(; name = "mcap", vals = [1.0, 2.0, 3.0]),                          CategoricalPanelInput(; name = "sector", vals = ["Fin", "Tech", "Fin"])]);julia> panel_feature_matrix(pnl)[1]3-element Vector{String}: "mcap" "sector=Fin" "sector=Tech"

Related

• AssetPanel
• AbstractPanelFieldInput
• AbstractPanelFillAlgorithm
• panel_input_field
• panel_build_observations
• panel_field_lift
• RepeatedLeading
• panel_feature_matrix
• ReturnsResult
• prices_to_returns
• Option

asset_panel(ape::Nothing, pr, rd::ReturnsResult, X) -> AssetPanel
asset_panel(ape::Nothing, pr::ReturnsResult, rd::Nothing, X) -> AssetPanel
asset_panel(ape::Nothing, pr, rd::Nothing, X) -> Union{}

Resolve the AssetPanel a FeatureDistance with no producer measures.

nothing in the ape slot says read the panel the data carrier already holds. The carriers reach the kernel as the two keywords pr and rd, and this verb resolves the source by dispatch: a ReturnsResult in either slot answers its pnl, and rd wins when both hold one, because the data carrier is where a panel is data rather than a by-product. Pr_RR admits a ReturnsResult in the pr slot, which is what clusterise(cle, rd) and every Pipeline step pass, so the second method is not a fallback but the shortest public call.

A prior result alone carries no panel, so it raises an IsNothingError naming the two ways forward.

Algorithm

The method that Julia selects is the algorithm.

1. rd is a ReturnsResult: answer rd.pnl.
2. pr is a ReturnsResult and there is no rd: answer pr.pnl.
3. Neither slot holds a data carrier: raise.

Each of the first two checks that the carrier it read holds a panel, with assert_asset_panel_supplied.

Arguments

• ape: nothing, which reads the carrier's panel.
• pr: Prior result or returns result. Both carry the asset returns matrix X and the feature matrix Z, so either can supply them.
• rd: The returns result to use.
• X: Returns matrix of the subproblem. Unread here; a producer reads it.

Validation

• A data carrier is present, and it holds an AssetPanel. Raises an IsNothingError.

Returns

• pnl::AssetPanel: The Asset Panel the data carrier holds.

Related

• AbstractAssetPanelEstimator
• assert_asset_panel_supplied
• FeatureDistance
• AssetPanel
• ReturnsResult
• RegressionPanel
• PhylogenyPanel

asset_panel(ape::RegressionPanel, pr, rd, X) -> AssetPanel
asset_panel(ape::PhylogenyPanel, pr, rd, X) -> AssetPanel

Build the static AssetPanel a producer returns, at the point of use.

Each method returns a panel holding one TensorPanelField, because a loadings matrix and a proximity matrix are each one quantity with a labelled third axis. The trailing axis is labelled off the data carrier or the regression block where a name exists there, and positionally otherwise; panel_axis_labels and regression_factor_names state the rule.

A producer runs on the subproblem's own prior and returns, so nothing views what it built and a fold refits it. Standalone on a prior fitted on a point-in-time Asset Panel, a RegressionPanel reads the loadings on the prior's Investable Mask and answers the full universe, with a zero row and a false observed mask outside it, so the panel it builds can be handed back to an optimiser; expand_investable_loadings states the rule.

Algorithm

A RegressionPanel takes four steps:

1. Check that a prior result reached the call, with assert_producer_prior.
2. Check that the prior carries a regression, with assert_prior_regression.
3. Reduce the prior to its Investable Mask through investable_mask and port_opt_view, and read the loadings there. A prior fitted on a point-in-time Asset Panel writes NaN on the loadings of every asset outside its mask, and a Panel Field admits no NaN. Inside an optimiser the prior arrives reduced and the view is the whole universe.
4. Check that every loading on the mask is finite, and refuse otherwise: an asset the check counts has a finite moment and a loadings row that is not, which is a defect of the regression.
5. Label the loadings axis with panel_axis_labels, from regression_factor_names.
6. Expand the loadings back onto the full universe with expand_investable_loadings: a zero row and a false observed mask outside the mask, the same rule every uncertainty set fitted standalone on such a prior follows.
7. Return the panel holding them as the field "loadings" on the axis "factor".

A PhylogenyPanel takes three steps:

1. Grade the structure into an assets × assets matrix with phylogeny_features.
2. Label the trailing axis with panel_axis_labels, from carrier_asset_names.
3. Return the panel holding that matrix as the field "proximity" on the axis "asset".

Arguments

• ape: The producer.
• pr: Prior result or returns result. Both carry the asset returns matrix X and the feature matrix Z, so either can supply them.
• rd: The returns result to use. Read for the axis names alone.
• X: Returns matrix of the subproblem, observations × assets.

Validation

• A RegressionPanel needs a prior result carrying a regression. Raises an IsNothingError.
• A RegressionPanel needs every loading inside the prior's Investable Mask to be finite. Raises an IsNonFiniteError counting the assets whose loadings are not.

Returns

• pnl::AssetPanel: A static Asset Panel holding one tensor Panel Field.

Related

• AbstractAssetPanelEstimator
• RegressionPanel
• PhylogenyPanel
• TensorPanelField
• panel_axis_labels
• FeatureDistance

panel_fill(alg::NoPanelFill, v::AbstractVector, name::AbstractString) -> Vector
panel_fill(alg::ConstantPanelFill, v::AbstractVector, name::AbstractString) -> Vector
panel_fill(alg::ForwardPanelFill, v::AbstractVector, name::AbstractString) -> Vector
panel_fill(alg::BackwardPanelFill, v::AbstractVector, name::AbstractString) -> Vector

Resolve the blanks of one asset's column of one raw Panel Field, along the observation axis.

Algorithm

The method that Julia selects is the algorithm, and the four differ in where the replacement comes from.

1. NoPanelFill: throw when any cell is blank, naming the Panel Field and the first offending observation. Otherwise return the column unchanged.
2. ConstantPanelFill: replace every blank by alg.val.
3. ForwardPanelFill: walk the observations in order, carrying the last observed value. Fill a blank with the carried value while the run of blanks is no longer than alg.lim, and with alg.val otherwise.
4. BackwardPanelFill: the same walk, in reverse order. This looks forward in time; the type's docstring states what that costs a fold.

Arguments

• alg: The fill policy.
• v: One asset's raw values along the observation axis, blanks included.
• name: The Panel Field's name, displayed in the NoPanelFill error message.

Validation

• Under NoPanelFill, v carries no blank. Raises an ArgumentError.

Returns

• filled::Vector: The same length as v, and free of blanks.

Related

• AbstractPanelFillAlgorithm
• is_panel_blank
• asset_panel

panel_resolve(inp::NumericPanelInput) -> Tuple{Array, BitArray}
panel_resolve(inp::CategoricalPanelInput) -> Tuple{Array{String}, BitArray}
panel_resolve(inp::TensorPanelInput) -> Tuple{Array, BitArray}

Resolve one raw Panel Field's blanks, and record which of its cells were observed.

Algorithm

The method that Julia selects is the algorithm, and the three differ only in the element type they resolve into. A categorical Panel Field resolves into String. A numeric and a tensor Panel Field resolve into the type their own filled cells carry, which panel_value_eltype derives: the raw array's element type is Union{Missing, Float64}, Union{Nothing, Float64} or Any, and none of those is the type the cells carry. Narrowing by the values is what drops the blank from the resolved field, and it is also what keeps a Float32 field in Float32.

1. Fill the blanks with panel_fill_array.
2. Derive the resolved element type from the filled cells, with panel_value_eltype.
3. Walk the raw cells, recording which were observed and copying the filled value into the output.
4. Check that a numeric or a tensor Panel Field carries no non-finite value, with assert_panel_finite.

Arguments

• inp: The raw Panel Field.

Returns

• vals::AbstractArray: The resolved values, the same size as the raw ones.
• obs::BitArray: The observed mask, the same size as the raw values.

Related

• AbstractPanelFieldInput
• panel_fill_array
• panel_input_field
• asset_panel

panel_input_field(inp::NumericPanelInput, vals, obs) -> NumericPanelField
panel_input_field(inp::CategoricalPanelInput, vals, obs) -> CategoricalPanelField
panel_input_field(inp::TensorPanelInput, vals, obs) -> TensorPanelField

Return the Panel Field a resolved raw Panel Field builds, deriving what the input left to be derived.

Algorithm

The method that Julia selects is the algorithm.

1. NumericPanelInput: a NumericPanelField over the resolved values.
2. CategoricalPanelInput: a CategoricalPanelField over inp.levels, or, when that is nothing, over the distinct resolved labels in sorted order. The resolved labels are read rather than the raw ones, so a level that only a fill policy introduces still gets a code. Each label is then encoded to its level's position.
3. TensorPanelInput: a TensorPanelField over the input's own axis, labels and groups.

The observed mask rides only when the fill policy is not NoPanelFill: a Panel Field that refuses a blank observed every cell, so a mask of it carries no information.

Arguments

• inp: The raw Panel Field.
• vals: The resolved values, as panel_resolve returned them.
• obs: The observed mask, as panel_resolve returned it.

Validation

• Every resolved label of a categorical Panel Field is one of its levels. Raises an ArgumentError.

Returns

• f::AbstractPanelField: The Panel Field.

Related

• AbstractPanelFieldInput
• AbstractPanelField
• panel_resolve
• asset_panel

panel_input_is_static(inp::NumericPanelInput) -> Bool
panel_input_is_static(inp::CategoricalPanelInput) -> Bool
panel_input_is_static(inp::TensorPanelInput) -> Bool

Return whether a raw Panel Field carries no observation axis.

The rank of the raw values is what declares the shape: a numeric or a categorical input is assets when static and observations × assets when time-varying, and a tensor input is assets × labels when static and observations × assets × labels when time-varying.

Algorithm

The method that Julia selects is the algorithm, and the three differ only in the rank the static shape takes.

Arguments

• inp: The raw Panel Field.

Returns

• static::Bool: true when the raw values carry no observation axis.

Related

• AbstractPanelFieldInput
• asset_panel
• panel_is_static