Asset Panel Estimators
An asset panel estimator builds a static AssetPanel each time a distance needs one. FeatureDistance holds one in its ape field. The estimator builds the panel from the prior result and the returns of the optimisation that runs it, which inside a nested optimisation is the inner one. RegressionPanel takes the factor loadings that a factor prior fitted. PhylogenyPanel takes the closeness of each pair of assets in a network or a clustering of the assets. It reads no prior, so it also works before a prior exists, as in asset selection. A new phylogeny feature rule subtypes AbstractPhylogenyFeatureAlgorithm and adds a method of phylogeny_features.
PortfolioOptimisers.AbstractPhylogenyFeatureAlgorithm — Type
abstract type AbstractPhylogenyFeatureAlgorithm <: AbstractAlgorithmAbstract supertype for all phylogeny feature algorithms.
A phylogeny feature algorithm is the rule that turns the structure of a PhylogenyPanel source into an assets × assets proximity matrix. The one member, Proximity, scores each pair of assets by a decay of their separation.
Two other choices need no new member. A different fall-off is an AbstractSeparationDecayAlgorithm, which Proximity holds in its decay field. A different measure of separation is an AbstractSeparationAlgorithm, which the source NetworkEstimator holds in its sep field. A new member is for a rule that is not a decay of a separation, for example a matrix of role similarity.
Every member puts each asset at the top of its own scale, so the diagonal of the matrix holds the largest entries. PhylogenyPanel states why the diagonal matters.
Interfaces
To add a rule, subtype AbstractPhylogenyFeatureAlgorithm and implement the method below. A clustering source needs no new method, because the co-membership method of phylogeny_features accepts every member.
phylogeny_features
phylogeny_features(alg::MyPhylogenyFeatureAlgorithm, pl::AbstractNetworkEstimator, X::MatNum; kwargs...) -> Matrix: Build the feature matrix of the graph thatplfits onX.
Arguments
alg: The new phylogeny feature algorithm.pl: Network estimator that fits the graph.X: Asset returns matrix,observations × assets.kwargs...: Keyword arguments for the phylogeny routines.
Returns
Z::Matrix: Squareassets × assetsfeature matrix, with the largest entry of each row on the diagonal.
Related
PortfolioOptimisers.Proximity — Type
struct Proximity{__T_decay} <: AbstractPhylogenyFeatureAlgorithmPhylogeny feature algorithm that scores each pair of assets by a decay of their separation.
The sep field of the source NetworkEstimator sets the separation and its budget. HopCount counts the edges of a shortest path, and PathLength sums their weights. The decay field sets the fall-off. A pair inside the budget scores the decay of its separation, and every other pair scores zero.
The score reads the separation, not the walk count sum(A^i) of the adjacency matrix A. A hub gathers walks faster than other assets, so a walk count mixes the number of neighbours of two assets into their closeness. phylogeny_matrix sums that walk count and then clamps it to [0, 1], which removes the number of steps. Proximity keeps it.
Mathematical definition
\[\begin{align} Z_{i,\,k} &= \begin{cases} f\left(D_{i,\,k}\right) & D_{i,\,k} \leq d_{\mathrm{max}}\,, \\ 0 & \text{otherwise}\,. \end{cases} \end{align}\]
Where:
- $\mathbf{Z}$: Proximity matrix, of size $N \times N$. Entry $Z_{i,\,k}$ scores how close asset $k$ is to asset $i$, and a larger entry means a closer pair.
- $\mathbf{D}$: Separation matrix, of size $N \times N$. Entry $D_{i,\,k}$ is the separation of assets $i$ and $k$ in the structure, $D_{i,\,i} = 0$, and $D_{i,\,k} = +\infty$ when no path joins the two assets.
- $d_{\mathrm{max}}$: Separation budget in scope.
- $f$: Separation decay, the score that
separation_decaygives a separation under the decay algorithm in scope.
The diagonal is $f(0)$, the largest entry of $\mathbf{Z}$, because a decay is largest at a separation of zero. An unreachable pair has $D_{i,\,k} = +\infty$, which exceeds every budget, so it scores zero. A decay is non-negative inside the budget, so a zero entry means that the pair is unreachable, outside the budget, or scored zero by the decay. No shipped decay scores zero inside the budget.
The fall-off and the budget
decay sets the fall-off, and sep on the source sets the budget. The budget is the only truncation. The two are separate because an exponential never reaches zero, so a fall-off cannot state a budget. Under the default LinearDecay and a HopCount budget of n, the asset itself scores n + 1, a direct neighbour scores n, and a pair n hops apart scores 1. Under ExponentialDecay or ReciprocalDecay the diagonal is 1, and the parameter of the decay sets the fall-off whatever the budget.
decay = nothing scores 1 at every separation, so Proximity(; decay = nothing) gives 1 inside the budget and 0 outside it. The result is the indicator of the neighbourhood that the budget selects, not a matrix of ones.
Hop counts and path lengths
A Z graded over hop counts and a Z graded over path lengths are both valid inputs to every consumer, but their values do not compare. The budgets have different units, the supports differ, and under LinearDecay the scales differ too. On real data the two separations order the pairs almost alike, and PathLength states the measurement. That agreement does not make the numbers of one run comparable with the numbers of another.
Under the default dmax = nothing of PathLength, the budget is the observed diameter of the graph. Under LinearDecay the diagonal $d_{\mathrm{max}} + 1$ then depends on the data, and a diameter that changes between cross-validation folds shifts every entry of Z. A fixed dmax keeps the scale fixed across folds. A decay with f(0) = 1 does not have this problem.
Unreachable pairs
separation_matrix writes a sentinel for an unreachable pair. The sentinel is typemax(Int) for a hop count, and Inf for a path length over Float64 weights. The budget test fails at the sentinel, so the kernel never evaluates the decay there. ReciprocalDecay needs that guard, because it overflows 1 + d at typemax(Int), and a fractional power then raises a DomainError.
Fields
decay: Separation decay algorithm, the rule by which the score falls off as two assets get further apart, ornothingfor a flat score of one. Distinct from the exponentially weighted moment estimators'decay, which is a smoothing constant over observations.
Constructors
Proximity(; decay::Option{<:AbstractSeparationDecayAlgorithm} = LinearDecay()) -> ProximityKeywords correspond to the struct's fields.
Examples
julia> Proximity(; decay = ExponentialDecay(; rate = 0.5))Proximity decay ┼ ExponentialDecay │ rate ┴ Float64: 0.5Related
PortfolioOptimisers.RegressionPanel — Type
struct RegressionPanel <: AbstractAssetPanelEstimatorBuilds an Asset Panel that holds the factor loadings of the wrapped prior.
RegressionPanel takes pr.rr.L, the coordinates that a factor model gives each asset, as the values of one tensor Panel Field, "loadings" on the axis "factor". It needs no estimation of its own. A factor prior computes the loadings already, and refits them in each fold, so the features follow the fold.
The matrix is L, not M
L is assets × reduced_dimensions, and DimensionReductionRegression sets it. It holds the coordinates of each asset in the reduced space. M holds the loadings on every factor, rebuilt from L. pr.rr.L always has a value, because Regression returns M when L is unset. So this producer needs no branch, and it works behind the time-series and the cross-sectional regression estimators alike.
Both matrices have one row per asset, so the asset axis of the panel is the asset axis of the returns data, with no transpose. The label axis holds factors or reduced dimensions, not assets, so a loadings panel is never square in the sense of features_are_assets.
Validation
- The call supplies a prior result. Raises an
IsNothingErrorthat names the call site. - The wrapped prior holds a regression, see
assert_prior_regression. The order of nesting has no effect, because every wrapping estimator forwardsrrand the factor blockfpr. A prior that computes no regression, such asEmpiricalPrior, raises the error.
Examples
julia> RegressionPanel()RegressionPanel()Related
PortfolioOptimisers.PhylogenyPanel — Type
struct PhylogenyPanel{__T_pl, __T_alg} <: AbstractAssetPanelEstimatorBuilds an Asset Panel that holds a square proximity matrix, graded from a graph or a partition.
In an assets × assets proximity matrix, feature k of a row reads "is close to asset k". So a metric over the rows measures the overlap of two neighbourhoods, which is a standard measure of topological similarity, and it needs almost no new estimation code. PhylogenyPanel returns the matrix as one tensor Panel Field, "proximity" on the axis "asset".
It is the one producer whose label axis is the asset axis, and it needs no flag for that. It refits, so a subproblem gets its own square matrix over its own assets, not a slice of a larger matrix. Where a panel supplied by hand meets a view, features_are_assets compares the labels of the field with the asset names of the returns data.
Fields
pl: Structure source, always an estimator so that it refits per fold: a network estimator (a graph, whosesepmeasures the separationsalggrades) or a clustering estimator (a partition, for whichalgis inert). A precomputed result is not accepted – an Estimator does not hold a Result.
alg: Phylogeny feature algorithm: the rule turning the source's separations into feature values. Inert for a partition source, which has no separation to grade.
Constructors
PhylogenyPanel(; pl::NwE_ClE = NetworkEstimator(), alg::AbstractPhylogenyFeatureAlgorithm = Proximity()) -> PhylogenyPanelKeywords correspond to the struct's fields.
Propagated parameters
factory passes its arguments on to these @fprop fields:
pl: Recursively updated throughfactory.
View parameters
port_opt_view takes the view of these @vprop fields at the selected indices:
pl: Recursively viewed throughport_opt_view.
Why the diagonal includes self
The diagonal selects between two different algorithms. The table measures a three-node path 1 - 2 - 3 under the default AngularDist and the default one-hop budget. The decay is nothing, so the diagonal is the only difference between the two columns.
| pair | zero diagonal | self included |
|---|---|---|
1-3, 2 hops | 0.0 | 0.333 |
1-2, 1 hop | 0.5 | 0.196 |
The default LinearDecay grades the path over the same budget, so its distances are 0.436 for 1-3 and 0.239 for 1-2. The order of the two pairs is the order of the right-hand column.
With a zero diagonal, the two endpoints, which are not adjacent, have a distance of zero, and each adjacent pair has the largest distance. The metric then compares two rows by their shared neighbours, and ignores whether the two assets are neighbours of each other. That is structural equivalence, a similarity of role, and it is the opposite of proximity.
The diagonal also keeps a subproblem well defined. A view of a spanning tree on a subset of the assets can isolate a vertex. With a zero diagonal, the row of an isolated asset is a zero row, and the zero-vector convention of AngularDist puts every isolated asset at distance 0 from every other isolated asset. With self included, each isolated asset is at distance 0.5 from every other asset, the largest distance between two non-negative rows.
A clustering source
NwE_ClE bounds pl, so it is a graph (NetworkEstimator) or a partition (ClustersEstimator). Both are estimators, so both refit.
A partition holds less information than a graph. Its matrix is the co-membership indicator P * transpose(P), where P is the assets × clusters membership matrix. Row i marks the assets in the cluster of asset i, so two rows are equal or disjoint. On seven assets in the clusters [1, 1, 1, 2, 2, 3, 3], the distance matrix under the default AngularDist takes two values, 0.0 inside a cluster and 0.5 across clusters, whatever the sizes of the clusters. The raw output of phylogeny_matrix is P * transpose(P) - I, and it takes three values, 0.0, 0.333 and 0.5. In a cluster of two assets, the - I leaves each row with a single 1 at the other member, so the two rows are orthogonal, and the distance inside that cluster equals the distance across clusters. This producer adds I back, which repairs that case, but the matrix stays coarse.
A clustering of a feature matrix built from a clustering gives back much the same clustering. Use a graph source unless the partition is the thing to measure.
Provenance
The source is a NetworkEstimator and never a precomputed PhylogenyResult, because an Estimator does not hold a Result. The producer is therefore endogenous. It filters the graph from the correlation of the returns, so it measures the topology that the correlation implies and no structure outside it. It refits on every fold and on every subproblem.
It reads no prior result, so it is the one producer that runs where no prior exists yet, for example in preselection.
pl reads the returns matrix that the reader of the panel picked, so it states no source of its own, and the constructor refuses an x_src on it that is not nothing, see assert_inner_reader.
Examples
julia> PhylogenyPanel()PhylogenyPanel pl ┼ NetworkEstimator │ ce ┼ PortfolioOptimisersCovariance │ │ ce ┼ Covariance │ │ │ me ┼ SimpleExpectedReturns │ │ │ │ w ┴ nothing │ │ │ ce ┼ GeneralCovariance │ │ │ │ ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true) │ │ │ │ w ┴ nothing │ │ │ alg ┼ FullMoment() │ │ │ w ┴ nothing │ │ mp ┼ MatrixProcessing │ │ │ pdm ┼ Posdef │ │ │ │ alg ┼ UnionAll: NearestCorrelationMatrix.Newton │ │ │ │ kwargs ┴ @NamedTuple{}: NamedTuple() │ │ │ dn ┼ nothing │ │ │ dt ┼ nothing │ │ │ alg ┼ nothing │ │ │ order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg) │ de ┼ Distance │ │ power ┼ nothing │ │ alg ┴ CanonicalDistance() │ alg ┼ KruskalTree │ │ args ┼ Tuple{}: () │ │ kwargs ┴ @NamedTuple{}: NamedTuple() │ sep ┼ HopCount │ │ n ┼ CountOf │ │ │ val ┴ Int64: 1 │ x_src ┴ nothing alg ┼ Proximity │ decay ┴ LinearDecay()Related
PortfolioOptimisers.phylogeny_features — Function
phylogeny_features(alg::Proximity, pl::AbstractNetworkEstimator,
X::MatNum; kwargs...)
phylogeny_features(alg::AbstractPhylogenyFeatureAlgorithm,
pl::AbstractClustersEstimator, X::MatNum; kwargs...)Turn a graph source or a partition source into a square assets × assets feature matrix.
This is the kernel of PhylogenyPanel. Every method returns a matrix of element type eltype(X), which is the element type of the returns. It is never the Int or BitMatrix that the phylogeny routines give, so AngularDist keeps its BLAS gemm path and a Float32 history gives a Float32 matrix.
pl is an estimator, a NetworkEstimator or a ClustersEstimator. It is never a precomputed PhylogenyResult or Clusters, because an Estimator does not hold a Result. So each call builds the structure from X, and the structure follows a cross-validation fold or the subproblem of a meta-optimiser.
A graph has separations, and alg decays them. A partition has none, because two assets are in one cluster or they are not. So every algorithm gives the same co-membership matrix of a partition, and alg has no effect there. FeatureDistance treats its collapse alg the same way on a static feature matrix. sep is a field of NetworkEstimator alone, so no other field loses its effect on a partition.
Mathematical definition
\[\begin{align} Z_{i,\,k} &= \begin{cases} f\left(D_{i,\,k}\right) & D_{i,\,k} \leq d_{\mathrm{max}}\,, \\ 0 & \text{otherwise}\,, \end{cases} \quad \text{for a graph}\,, \\ Z_{i,\,k} &= \begin{cases} 1 & c_{i} = c_{k}\,, \\ 0 & \text{otherwise}\,, \end{cases} \quad \text{for a partition}\,. \end{align}\]
Where:
- $\mathbf{Z}$: Proximity matrix, of size $N \times N$. Entry $Z_{i,\,k}$ scores how close asset $k$ is to asset $i$, and a larger entry means a closer pair.
- $\mathbf{D}$: Separation matrix, of size $N \times N$. Entry $D_{i,\,k}$ is the separation of assets $i$ and $k$ in the structure, $D_{i,\,i} = 0$, and $D_{i,\,k} = +\infty$ when no path joins the two assets.
- $d_{\mathrm{max}}$: Separation budget in scope.
- $f$: Separation decay, the score that
separation_decaygives a separation under the decay algorithm in scope. - $c_{i}$: Cluster of asset $i$ in the partition that
plfits.
The diagonal holds the largest entry of each row. It is $f(0)$ for a graph, and $1$ for a partition. Proximity states $f(0)$ for each decay. assert_separation_decay checks that $f(0)$ is the largest score of a decay before the kernel scores a pair.
Algorithm
Over a graph source, under Proximity:
- Build the structure from
Xthroughseparation_graph, givingg. - Resolve the separation algorithm against the structure through
resolve_separation, givingsep. - Measure the separations through
separation_matrix, givingd. - Read the budget through
separation_budget, and scoredin the element typeeltype(X)through_proximity_features, givingZ.
Over a partition source, under any algorithm:
- Build the co-membership matrix from
Xthroughphylogeny_matrix, and convert it to the element typeeltype(X). - Add the identity, which restores the diagonal that
phylogeny_matrixsubtracts, givingZ.
Arguments
alg: Phylogeny feature algorithm.pl: Structure source, a network estimator for a graph or a clustering estimator for a partition.X: Asset returns matrix,observations × assets.kwargs...: Keyword arguments for the phylogeny routines.
Returns
Z::Matrix: Squareassets × assetsfeature matrix of element typeeltype(X).
Related
PortfolioOptimisers.asset_panel — Method
asset_panel(ape::RegressionPanel, pr, rd, X) -> AssetPanel
asset_panel(ape::PhylogenyPanel, pr, rd, X) -> AssetPanelBuild the static AssetPanel of a producer, at the point of use.
Each method returns a panel that holds one TensorPanelField, because a loadings matrix and a proximity matrix are each one quantity with a labelled second axis. The labels of the label axis come from the returns data or from the regression block where a name exists, and are positions otherwise. panel_axis_labels and regression_factor_names state the rule.
A producer runs on the prior and the returns of its own subproblem, so no view reads the panel that it builds, and a fold refits it. Called on its own, on a prior fitted on a point-in-time Asset Panel, a RegressionPanel reads the loadings on the Investable Mask of the prior and answers for the full universe. Outside the mask each asset gets a zero row and a false observed mask, so an optimiser can take the panel back. expand_investable_loadings states the rule.
Algorithm
A RegressionPanel takes seven steps:
- Check that a prior result reached the call, with
assert_producer_prior. - Check that the prior holds a regression, with
assert_prior_regression. - Reduce the prior to its Investable Mask through
investable_maskandport_opt_view, givingprr, and read its loadingsL. A prior fitted on a point-in-time Asset Panel writesNaNinto the loadings of every asset outside its mask, and a Panel Field admits noNaN. Inside an optimiser the prior arrives reduced, and the view covers the whole universe. - Count the assets on the mask whose loadings are not all finite, giving
nnf, and refuse a count above zero. Such an asset has a finite moment and a loadings row that is not finite, which is a defect of the regression. - Label the loadings axis with
panel_axis_labels, fromregression_factor_names. - Expand the loadings back onto the full universe with
expand_investable_loadings, givingvalsandomsk. - Return the panel that holds them as the field
"loadings"on the axis"factor".
A PhylogenyPanel takes three steps:
- Grade the structure into an
assets × assetsmatrix withphylogeny_features, givingZp. - Label the label axis with
panel_axis_labels, fromreturns_asset_names. - Return the panel that holds
Zpas the field"proximity"on the axis"asset".
Arguments
ape: The producer.pr: Prior result or returns result. Both carry the asset returns matrixX, so either can supply it.rd: The returns result to use. Read for the axis names alone.X: Returns matrix of the subproblem, observations × assets.
Validation
- A
RegressionPanelneeds a prior result that holds a regression. Raises anIsNothingError. - A
RegressionPanelneeds every loading inside the Investable Mask of the prior to be finite. Raises anIsNonFiniteErrorthat counts the assets whose loadings are not finite.
Returns
pnl::AssetPanel: A static Asset Panel that holds one tensor Panel Field.
Related