Asset Panel Estimators

An asset panel estimator builds a static AssetPanel each time a distance needs one. FeatureDistance holds one in its ape field. The estimator builds the panel from the prior result and the returns of the optimisation that runs it, which inside a nested optimisation is the inner one. RegressionPanel takes the factor loadings that a factor prior fitted. PhylogenyPanel takes the closeness of each pair of assets in a network or a clustering of the assets. It reads no prior, so it also works before a prior exists, as in asset selection. A new phylogeny feature rule subtypes AbstractPhylogenyFeatureAlgorithm and adds a method of phylogeny_features.

PortfolioOptimisers.AbstractPhylogenyFeatureAlgorithm — Type
abstract type AbstractPhylogenyFeatureAlgorithm <: AbstractAlgorithm

Abstract supertype for all phylogeny feature algorithms.

A phylogeny feature algorithm is the rule that turns the structure of a PhylogenyPanel source into an assets × assets proximity matrix. The one member, Proximity, scores each pair of assets by a decay of their separation.

Two other choices need no new member. A different fall-off is an AbstractSeparationDecayAlgorithm, which Proximity holds in its decay field. A different measure of separation is an AbstractSeparationAlgorithm, which the source NetworkEstimator holds in its sep field. A new member is for a rule that is not a decay of a separation, for example a matrix of role similarity.

Every member puts each asset at the top of its own scale, so the diagonal of the matrix holds the largest entries. PhylogenyPanel states why the diagonal matters.

Interfaces

To add a rule, subtype AbstractPhylogenyFeatureAlgorithm and implement the method below. A clustering source needs no new method, because the co-membership method of phylogeny_features accepts every member.

phylogeny_features

  • phylogeny_features(alg::MyPhylogenyFeatureAlgorithm, pl::AbstractNetworkEstimator, X::MatNum; kwargs...) -> Matrix: Build the feature matrix of the graph that pl fits on X.

Arguments

  • alg: The new phylogeny feature algorithm.
  • pl: Network estimator that fits the graph.
  • X: Asset returns matrix, observations × assets.
  • kwargs...: Keyword arguments for the phylogeny routines.

Returns

  • Z::Matrix: Square assets × assets feature matrix, with the largest entry of each row on the diagonal.

Related

source
PortfolioOptimisers.Proximity — Type
struct Proximity{__T_decay} <: AbstractPhylogenyFeatureAlgorithm

Phylogeny feature algorithm that scores each pair of assets by a decay of their separation.

The sep field of the source NetworkEstimator sets the separation and its budget. HopCount counts the edges of a shortest path, and PathLength sums their weights. The decay field sets the fall-off. A pair inside the budget scores the decay of its separation, and every other pair scores zero.

The score reads the separation, not the walk count sum(A^i) of the adjacency matrix A. A hub gathers walks faster than other assets, so a walk count mixes the number of neighbours of two assets into their closeness. phylogeny_matrix sums that walk count and then clamps it to [0, 1], which removes the number of steps. Proximity keeps it.

Mathematical definition

\[\begin{align} Z_{i,\,k} &= \begin{cases} f\left(D_{i,\,k}\right) & D_{i,\,k} \leq d_{\mathrm{max}}\,, \\ 0 & \text{otherwise}\,. \end{cases} \end{align}\]

Where:

  • $\mathbf{Z}$: Proximity matrix, of size $N \times N$. Entry $Z_{i,\,k}$ scores how close asset $k$ is to asset $i$, and a larger entry means a closer pair.
  • $\mathbf{D}$: Separation matrix, of size $N \times N$. Entry $D_{i,\,k}$ is the separation of assets $i$ and $k$ in the structure, $D_{i,\,i} = 0$, and $D_{i,\,k} = +\infty$ when no path joins the two assets.
  • $d_{\mathrm{max}}$: Separation budget in scope.
  • $f$: Separation decay, the score that separation_decay gives a separation under the decay algorithm in scope.

The diagonal is $f(0)$, the largest entry of $\mathbf{Z}$, because a decay is largest at a separation of zero. An unreachable pair has $D_{i,\,k} = +\infty$, which exceeds every budget, so it scores zero. A decay is non-negative inside the budget, so a zero entry means that the pair is unreachable, outside the budget, or scored zero by the decay. No shipped decay scores zero inside the budget.

The fall-off and the budget

decay sets the fall-off, and sep on the source sets the budget. The budget is the only truncation. The two are separate because an exponential never reaches zero, so a fall-off cannot state a budget. Under the default LinearDecay and a HopCount budget of n, the asset itself scores n + 1, a direct neighbour scores n, and a pair n hops apart scores 1. Under ExponentialDecay or ReciprocalDecay the diagonal is 1, and the parameter of the decay sets the fall-off whatever the budget.

decay = nothing scores 1 at every separation, so Proximity(; decay = nothing) gives 1 inside the budget and 0 outside it. The result is the indicator of the neighbourhood that the budget selects, not a matrix of ones.

Hop counts and path lengths

A Z graded over hop counts and a Z graded over path lengths are both valid inputs to every consumer, but their values do not compare. The budgets have different units, the supports differ, and under LinearDecay the scales differ too. On real data the two separations order the pairs almost alike, and PathLength states the measurement. That agreement does not make the numbers of one run comparable with the numbers of another.

Under the default dmax = nothing of PathLength, the budget is the observed diameter of the graph. Under LinearDecay the diagonal $d_{\mathrm{max}} + 1$ then depends on the data, and a diameter that changes between cross-validation folds shifts every entry of Z. A fixed dmax keeps the scale fixed across folds. A decay with f(0) = 1 does not have this problem.

Unreachable pairs

separation_matrix writes a sentinel for an unreachable pair. The sentinel is typemax(Int) for a hop count, and Inf for a path length over Float64 weights. The budget test fails at the sentinel, so the kernel never evaluates the decay there. ReciprocalDecay needs that guard, because it overflows 1 + d at typemax(Int), and a fractional power then raises a DomainError.

Fields

  • decay: Separation decay algorithm, the rule by which the score falls off as two assets get further apart, or nothing for a flat score of one. Distinct from the exponentially weighted moment estimators' decay, which is a smoothing constant over observations.

Constructors

Proximity(;    decay::Option{<:AbstractSeparationDecayAlgorithm} = LinearDecay()) -> Proximity

Keywords correspond to the struct's fields.

Examples

julia> Proximity(; decay = ExponentialDecay(; rate = 0.5))Proximity  decay ┼ ExponentialDecay        │   rate ┴ Float64: 0.5

Related

source
PortfolioOptimisers.RegressionPanel — Type
struct RegressionPanel <: AbstractAssetPanelEstimator

Builds an Asset Panel that holds the factor loadings of the wrapped prior.

RegressionPanel takes pr.rr.L, the coordinates that a factor model gives each asset, as the values of one tensor Panel Field, "loadings" on the axis "factor". It needs no estimation of its own. A factor prior computes the loadings already, and refits them in each fold, so the features follow the fold.

The matrix is L, not M

L is assets × reduced_dimensions, and DimensionReductionRegression sets it. It holds the coordinates of each asset in the reduced space. M holds the loadings on every factor, rebuilt from L. pr.rr.L always has a value, because Regression returns M when L is unset. So this producer needs no branch, and it works behind the time-series and the cross-sectional regression estimators alike.

Both matrices have one row per asset, so the asset axis of the panel is the asset axis of the returns data, with no transpose. The label axis holds factors or reduced dimensions, not assets, so a loadings panel is never square in the sense of features_are_assets.

Validation

  • The call supplies a prior result. Raises an IsNothingError that names the call site.
  • The wrapped prior holds a regression, see assert_prior_regression. The order of nesting has no effect, because every wrapping estimator forwards rr and the factor block fpr. A prior that computes no regression, such as EmpiricalPrior, raises the error.

Examples

julia> RegressionPanel()RegressionPanel()

Related

source
PortfolioOptimisers.PhylogenyPanel — Type
struct PhylogenyPanel{__T_pl, __T_alg} <: AbstractAssetPanelEstimator

Builds an Asset Panel that holds a square proximity matrix, graded from a graph or a partition.

In an assets × assets proximity matrix, feature k of a row reads "is close to asset k". So a metric over the rows measures the overlap of two neighbourhoods, which is a standard measure of topological similarity, and it needs almost no new estimation code. PhylogenyPanel returns the matrix as one tensor Panel Field, "proximity" on the axis "asset".

It is the one producer whose label axis is the asset axis, and it needs no flag for that. It refits, so a subproblem gets its own square matrix over its own assets, not a slice of a larger matrix. Where a panel supplied by hand meets a view, features_are_assets compares the labels of the field with the asset names of the returns data.

Fields

  • pl: Structure source, always an estimator so that it refits per fold: a network estimator (a graph, whose sep measures the separations alg grades) or a clustering estimator (a partition, for which alg is inert). A precomputed result is not accepted – an Estimator does not hold a Result.
  • alg: Phylogeny feature algorithm: the rule turning the source's separations into feature values. Inert for a partition source, which has no separation to grade.

Constructors

PhylogenyPanel(;    pl::NwE_ClE = NetworkEstimator(),    alg::AbstractPhylogenyFeatureAlgorithm = Proximity()) -> PhylogenyPanel

Keywords correspond to the struct's fields.

Propagated parameters

factory passes its arguments on to these @fprop fields:

  • pl: Recursively updated through factory.

View parameters

port_opt_view takes the view of these @vprop fields at the selected indices:

Why the diagonal includes self

The diagonal selects between two different algorithms. The table measures a three-node path 1 - 2 - 3 under the default AngularDist and the default one-hop budget. The decay is nothing, so the diagonal is the only difference between the two columns.

pairzero diagonalself included
1-3, 2 hops0.00.333
1-2, 1 hop0.50.196

The default LinearDecay grades the path over the same budget, so its distances are 0.436 for 1-3 and 0.239 for 1-2. The order of the two pairs is the order of the right-hand column.

With a zero diagonal, the two endpoints, which are not adjacent, have a distance of zero, and each adjacent pair has the largest distance. The metric then compares two rows by their shared neighbours, and ignores whether the two assets are neighbours of each other. That is structural equivalence, a similarity of role, and it is the opposite of proximity.

The diagonal also keeps a subproblem well defined. A view of a spanning tree on a subset of the assets can isolate a vertex. With a zero diagonal, the row of an isolated asset is a zero row, and the zero-vector convention of AngularDist puts every isolated asset at distance 0 from every other isolated asset. With self included, each isolated asset is at distance 0.5 from every other asset, the largest distance between two non-negative rows.

A clustering source

NwE_ClE bounds pl, so it is a graph (NetworkEstimator) or a partition (ClustersEstimator). Both are estimators, so both refit.

A partition holds less information than a graph. Its matrix is the co-membership indicator P * transpose(P), where P is the assets × clusters membership matrix. Row i marks the assets in the cluster of asset i, so two rows are equal or disjoint. On seven assets in the clusters [1, 1, 1, 2, 2, 3, 3], the distance matrix under the default AngularDist takes two values, 0.0 inside a cluster and 0.5 across clusters, whatever the sizes of the clusters. The raw output of phylogeny_matrix is P * transpose(P) - I, and it takes three values, 0.0, 0.333 and 0.5. In a cluster of two assets, the - I leaves each row with a single 1 at the other member, so the two rows are orthogonal, and the distance inside that cluster equals the distance across clusters. This producer adds I back, which repairs that case, but the matrix stays coarse.

A clustering of a feature matrix built from a clustering gives back much the same clustering. Use a graph source unless the partition is the thing to measure.

Provenance

The source is a NetworkEstimator and never a precomputed PhylogenyResult, because an Estimator does not hold a Result. The producer is therefore endogenous. It filters the graph from the correlation of the returns, so it measures the topology that the correlation implies and no structure outside it. It refits on every fold and on every subproblem.

It reads no prior result, so it is the one producer that runs where no prior exists yet, for example in preselection.

pl reads the returns matrix that the reader of the panel picked, so it states no source of its own, and the constructor refuses an x_src on it that is not nothing, see assert_inner_reader.

Examples

julia> PhylogenyPanel()PhylogenyPanel   pl ┼ NetworkEstimator      │      ce ┼ PortfolioOptimisersCovariance      │         │   ce ┼ Covariance      │         │      │    me ┼ SimpleExpectedReturns      │         │      │       │   w ┴ nothing      │         │      │    ce ┼ GeneralCovariance      │         │      │       │   ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true)      │         │      │       │    w ┴ nothing      │         │      │   alg ┼ FullMoment()      │         │      │     w ┴ nothing      │         │   mp ┼ MatrixProcessing      │         │      │     pdm ┼ Posdef      │         │      │         │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton      │         │      │         │   kwargs ┴ @NamedTuple{}: NamedTuple()      │         │      │      dn ┼ nothing      │         │      │      dt ┼ nothing      │         │      │     alg ┼ nothing      │         │      │   order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg)      │      de ┼ Distance      │         │   power ┼ nothing      │         │     alg ┴ CanonicalDistance()      │     alg ┼ KruskalTree      │         │     args ┼ Tuple{}: ()      │         │   kwargs ┴ @NamedTuple{}: NamedTuple()      │     sep ┼ HopCount      │         │   n ┼ CountOf      │         │     │   val ┴ Int64: 1      │   x_src ┴ nothing  alg ┼ Proximity      │   decay ┴ LinearDecay()

Related

source
PortfolioOptimisers.phylogeny_features — Function
phylogeny_features(alg::Proximity, pl::AbstractNetworkEstimator,
                   X::MatNum; kwargs...)
phylogeny_features(alg::AbstractPhylogenyFeatureAlgorithm,
                   pl::AbstractClustersEstimator, X::MatNum; kwargs...)

Turn a graph source or a partition source into a square assets × assets feature matrix.

This is the kernel of PhylogenyPanel. Every method returns a matrix of element type eltype(X), which is the element type of the returns. It is never the Int or BitMatrix that the phylogeny routines give, so AngularDist keeps its BLAS gemm path and a Float32 history gives a Float32 matrix.

pl is an estimator, a NetworkEstimator or a ClustersEstimator. It is never a precomputed PhylogenyResult or Clusters, because an Estimator does not hold a Result. So each call builds the structure from X, and the structure follows a cross-validation fold or the subproblem of a meta-optimiser.

A graph has separations, and alg decays them. A partition has none, because two assets are in one cluster or they are not. So every algorithm gives the same co-membership matrix of a partition, and alg has no effect there. FeatureDistance treats its collapse alg the same way on a static feature matrix. sep is a field of NetworkEstimator alone, so no other field loses its effect on a partition.

Mathematical definition

\[\begin{align} Z_{i,\,k} &= \begin{cases} f\left(D_{i,\,k}\right) & D_{i,\,k} \leq d_{\mathrm{max}}\,, \\ 0 & \text{otherwise}\,, \end{cases} \quad \text{for a graph}\,, \\ Z_{i,\,k} &= \begin{cases} 1 & c_{i} = c_{k}\,, \\ 0 & \text{otherwise}\,, \end{cases} \quad \text{for a partition}\,. \end{align}\]

Where:

  • $\mathbf{Z}$: Proximity matrix, of size $N \times N$. Entry $Z_{i,\,k}$ scores how close asset $k$ is to asset $i$, and a larger entry means a closer pair.
  • $\mathbf{D}$: Separation matrix, of size $N \times N$. Entry $D_{i,\,k}$ is the separation of assets $i$ and $k$ in the structure, $D_{i,\,i} = 0$, and $D_{i,\,k} = +\infty$ when no path joins the two assets.
  • $d_{\mathrm{max}}$: Separation budget in scope.
  • $f$: Separation decay, the score that separation_decay gives a separation under the decay algorithm in scope.
  • $c_{i}$: Cluster of asset $i$ in the partition that pl fits.

The diagonal holds the largest entry of each row. It is $f(0)$ for a graph, and $1$ for a partition. Proximity states $f(0)$ for each decay. assert_separation_decay checks that $f(0)$ is the largest score of a decay before the kernel scores a pair.

Algorithm

Over a graph source, under Proximity:

  1. Build the structure from X through separation_graph, giving g.
  2. Resolve the separation algorithm against the structure through resolve_separation, giving sep.
  3. Measure the separations through separation_matrix, giving d.
  4. Read the budget through separation_budget, and score d in the element type eltype(X) through _proximity_features, giving Z.

Over a partition source, under any algorithm:

  1. Build the co-membership matrix from X through phylogeny_matrix, and convert it to the element type eltype(X).
  2. Add the identity, which restores the diagonal that phylogeny_matrix subtracts, giving Z.

Arguments

  • alg: Phylogeny feature algorithm.
  • pl: Structure source, a network estimator for a graph or a clustering estimator for a partition.
  • X: Asset returns matrix, observations × assets.
  • kwargs...: Keyword arguments for the phylogeny routines.

Returns

  • Z::Matrix: Square assets × assets feature matrix of element type eltype(X).

Related

source
PortfolioOptimisers.asset_panel — Method
asset_panel(ape::RegressionPanel, pr, rd, X) -> AssetPanel
asset_panel(ape::PhylogenyPanel, pr, rd, X) -> AssetPanel

Build the static AssetPanel of a producer, at the point of use.

Each method returns a panel that holds one TensorPanelField, because a loadings matrix and a proximity matrix are each one quantity with a labelled second axis. The labels of the label axis come from the returns data or from the regression block where a name exists, and are positions otherwise. panel_axis_labels and regression_factor_names state the rule.

A producer runs on the prior and the returns of its own subproblem, so no view reads the panel that it builds, and a fold refits it. Called on its own, on a prior fitted on a point-in-time Asset Panel, a RegressionPanel reads the loadings on the Investable Mask of the prior and answers for the full universe. Outside the mask each asset gets a zero row and a false observed mask, so an optimiser can take the panel back. expand_investable_loadings states the rule.

Algorithm

A RegressionPanel takes seven steps:

  1. Check that a prior result reached the call, with assert_producer_prior.
  2. Check that the prior holds a regression, with assert_prior_regression.
  3. Reduce the prior to its Investable Mask through investable_mask and port_opt_view, giving prr, and read its loadings L. A prior fitted on a point-in-time Asset Panel writes NaN into the loadings of every asset outside its mask, and a Panel Field admits no NaN. Inside an optimiser the prior arrives reduced, and the view covers the whole universe.
  4. Count the assets on the mask whose loadings are not all finite, giving nnf, and refuse a count above zero. Such an asset has a finite moment and a loadings row that is not finite, which is a defect of the regression.
  5. Label the loadings axis with panel_axis_labels, from regression_factor_names.
  6. Expand the loadings back onto the full universe with expand_investable_loadings, giving vals and omsk.
  7. Return the panel that holds them as the field "loadings" on the axis "factor".

A PhylogenyPanel takes three steps:

  1. Grade the structure into an assets × assets matrix with phylogeny_features, giving Zp.
  2. Label the label axis with panel_axis_labels, from returns_asset_names.
  3. Return the panel that holds Zp as the field "proximity" on the axis "asset".

Arguments

  • ape: The producer.
  • pr: Prior result or returns result. Both carry the asset returns matrix X, so either can supply it.
  • rd: The returns result to use. Read for the axis names alone.
  • X: Returns matrix of the subproblem, observations × assets.

Validation

Returns

  • pnl::AssetPanel: A static Asset Panel that holds one tensor Panel Field.

Related

source