Cross-Sectional Regression (a)

Types

PortfolioOptimisers.AbstractCrossSectionalSolveAlgorithm — Type
abstract type AbstractCrossSectionalSolveAlgorithm <: AbstractRegressionAlgorithm

Abstract supertype for all cross-sectional solve algorithm types.

A member decides what CrossSectionalLinearRegression and CrossSectionalTargetRegression do when the weighted design of an observation is rank deficient. The members differ in whether a rank test runs at all, and in what happens when it fails. The choice changes the answer, because Julia's \ answers a deficient design in three ways. A square design goes to an LU factorisation that throws LinearAlgebra.SingularException on an exactly zero pivot. A non-square one goes to a column-pivoted QR whose solve completes the orthogonal factorisation, so it returns the minimum-norm solution and agrees with a pseudo-inverse. A design that is only nearly dependent passes the rank test of every member and returns a badly conditioned answer that a pseudo-inverse would truncate.

A member means the same thing under both estimators, so the two give the same factor returns for a least-squares target. CrossSectionalTargetRegression refuses MinimumNormSolve, whose answer on a target is the answer of PseudoInverseFallback.

All concrete and/or abstract types representing cross-sectional solve algorithms should be subtypes of AbstractCrossSectionalSolveAlgorithm.

Interfaces

In order to implement a new concrete type that works seamlessly with the library, subtype AbstractCrossSectionalSolveAlgorithm and implement the following methods. A member that only CrossSectionalLinearRegression reads needs only cross_sectional_solve, and a member that only CrossSectionalTargetRegression reads needs only cross_sectional_target_solve.

A new member gets no mask of dropped columns, because the regression keeps dc at nothing for every member except DependentColumnDrop. So a new member that drops a column gives it no mark, and the regression diagnostics and the factor attribution read the pseudo-inverse of the full design of each observation for it.

Each verb returns the factor returns and the residual count of the fit that it makes, because only the fit knows the parameters that it spent. A least-squares fit is a projection onto the span of the design, so it spends the rank of the design, and not its number of columns. Read that rank with cross_sectional_rank, the rank test of every member of the library, so that the count of a new member agrees with theirs and with the t-statistics of cs_regression_t_stats. A penalised fit spends fewer, and residual_count reads its count. The regression writes the count of each observation into edof of CrossSectionalRegression.

cross_sectional_solve

Arguments

  • alg: The concrete subtype instance.
  • A: Weighted design of one observation, eligible assets × factors.
  • y: Weighted target of one observation, of length eligible assets.
  • t: Index of the observation, for the message of a refusal.
  • F: Column-pivoted QR of A that the regression took, or nothing.
  • ws: Workspace of the task, as cross_sectional_solve states it.

Returns

  • res::CrossSectionalSolveResult: The factor returns of the observation, of length size(A, 2), and the residual count of the fit, size(A, 1) less the rank of the columns that the fit solves. The method builds it with the constructor of CrossSectionalSolveResult, which checks the count.

cross_sectional_target_solve

Arguments

  • alg: The concrete subtype instance.
  • tgt: Regression target of the observation, which carries its weights. Fit it with StatsAPI.fit(tgt, A, y).
  • obs: The observation, (; A, y, D, t, F): the exposures A and the returns y of the eligible assets, the weighted design D for a rank test, the index t of the observation, and the column-pivoted QR F of D that the regression already took, or nothing.

Returns

  • res::CrossSectionalSolveResult: The factor returns of the observation, of length size(obs.A, 2), and the residual count of the fit, residual_count of the fitted target and the design that it fitted.

assert_cross_sectional_target_solve

Arguments

  • alg: The concrete subtype instance.
  • tgt: Regression target.

Returns

  • nothing. A refusal throws an ArgumentError.

Examples

julia> struct MySolve <: PortfolioOptimisers.AbstractCrossSectionalSolveAlgorithm endjulia> CrossSectionalTargetRegression(; alg = MySolve()).algMySolve()

Related

source
PortfolioOptimisers.PseudoInverseFallback — Type
struct PseudoInverseFallback <: AbstractCrossSectionalSolveAlgorithm

Solves the full-rank design directly and pseudo-inverts a rank-deficient one.

This is the default of CrossSectionalLinearRegression and of CrossSectionalTargetRegression. It never throws on a dependent factor set. A square design that \ would refuse reaches the pseudo-inverse instead, and a non-square one reaches the same minimum-norm answer through either route. The rank test costs a second factorisation of every design that passes it.

A target has no pseudo-inverse, so on a rank-deficient design it fits the columns that DependentColumnDrop keeps, and the answer is projected onto the row space of the weighted design. Every coefficient vector that differs from the fit by a vector of the null space gives the same linear predictor, and the projection is the one of least norm among them. For a least-squares target it is the answer of CrossSectionalLinearRegression. The projection reads the fit through the linear predictor alone, which is what is_basis_invariant states of a target, so the constructor refuses a target that answers false.

Examples

julia> PseudoInverseFallback()PseudoInverseFallback()

Related

source
PortfolioOptimisers.RankDeficiencyRefusal — Type
struct RankDeficiencyRefusal <: AbstractCrossSectionalSolveAlgorithm

Solves the full-rank design directly and refuses a rank-deficient one.

The refusal names the observation, the rank it measured, the factor count and the count of eligible assets, so a caller can tell a dependent factor set apart from a cross-section that is too small. Under CrossSectionalTargetRegression the refusal runs before the target fits.

Examples

julia> RankDeficiencyRefusal()RankDeficiencyRefusal()

Related

source
PortfolioOptimisers.UncheckedSolve — Type
struct UncheckedSolve <: AbstractCrossSectionalSolveAlgorithm

Runs no rank test and takes whatever \ returns.

It is the cheapest member, because it factorises the design once instead of twice. A rank-deficient non-square design reaches a column-pivoted QR whose solve returns the minimum-norm answer, so it agrees with MinimumNormSolve there. An exactly singular square design reaches an LU factorisation instead and throws LinearAlgebra.SingularException. That is the one design on which the member fails.

Under CrossSectionalTargetRegression the target sees the full design, and its own factorisation decides. GLM drops a collinear column by its own pivot and gives it a return of zero, and its pivot need not drop the column that the rank test of DependentColumnDrop drops. A factor set that is dependent to rounding alone can pass the pivot of GLM and fail its Cholesky factorisation, which throws LinearAlgebra.PosDefException. On the carry fold of a CrossSectionalFactorPrior, a move solves an observation again when cross_sectional_rank finds it rank-deficient, so an observation that the pivot of GLM drops and the rank test keeps can keep the answer of the old basis.

Examples

julia> UncheckedSolve()UncheckedSolve()

Related

source
PortfolioOptimisers.MinimumNormSolve — Type
struct MinimumNormSolve <: AbstractCrossSectionalSolveAlgorithm

Always pseudo-inverts, so it runs no rank test and takes no threshold.

Every observation takes the minimum-norm least-squares solution, whatever its rank, so one rule gives the factor returns of an observation that lost a factor and of one that did not. It is the most expensive member, because it takes a singular value decomposition of every design. It parts from UncheckedSolve on a square design, which \ sends to an LU factorisation, and on a design that is only nearly dependent, where the pseudo-inverse truncates a singular value that \ keeps.

CrossSectionalTargetRegression refuses it. A target has no pseudo-inverse, and the projection of PseudoInverseFallback changes nothing on a design of full rank, so the member would be a second name for that one.

Examples

julia> MinimumNormSolve()MinimumNormSolve()

Related

source
PortfolioOptimisers.DependentColumnDrop — Type
struct DependentColumnDrop <: AbstractCrossSectionalSolveAlgorithm

Solves the full-rank design directly, and drops the dependent columns of a rank-deficient one.

The member reads the rank r of the weighted design with cross_sectional_rank. When r falls short of the factor count, it keeps the columns that the first r pivots of the column-pivoted QR of the design name, solves on them, and gives every other factor a return of zero. A design of rank zero gives zero to every factor. The answer is a basic solution, not the answer of least norm. Its fitted values and its residuals are those of PseudoInverseFallback, and its factor returns are not. The pivot prefers the column of larger weighted norm, so which factor takes zero depends on the scale of the exposures and on the basis of a Factor Family.

The fit marks each dropped factor in the dc field of CrossSectionalRegression. So the t-statistics of cs_regression_t_stats and the standard errors of a factor attribution give a dropped factor no estimation error, and read the kept factors on the columns that the fit solved, rather than the pseudo-inverse of the whole design.

Examples

julia> DependentColumnDrop()DependentColumnDrop()

Related

source
PortfolioOptimisers.NormalEquationsSolve — Type
struct NormalEquationsSolve <: AbstractCrossSectionalSolveAlgorithm

Solves the normal equations of a well-conditioned design, and gives every other design to PseudoInverseFallback.

The member forms the Gram matrix $\mathbf{D}^{\intercal} \mathbf{D}$ of the weighted design and factorises it by a Cholesky factorisation with pivots. That costs about half of the column-pivoted QR that the other members take. The Gram matrix squares the condition number of the design, so the answer loses about twice the digits that the QR loses. The triangular factor of the factorisation has the condition number of the design, and cross_sectional_gram_conditioned estimates it in the 1-norm. When the factorisation has full rank and that estimate is at most $\epsilon^{-1/4}$, about $8 \times 10^{3}$ in Float64, the member solves the normal equations, and it counts every column, as the rank test of cross_sectional_rank does on such a design. Otherwise it takes the answer and the count of PseudoInverseFallback, so it never answers a nearly dependent design with the normal equations. The ratio of the pivots is not the test, because it can be far below the condition number of the design.

Warning

The normal equations lose precision with the square of the condition number. Against the QR solve of PseudoInverseFallback, the largest relative error of the factor returns over 20 random 1600 × 28 designs was 5e-15 at a condition number of 10, 2e-13 at 1e2 and 2e-11 at 1e3. At 3e3 the member solved 4 of the 20 designs by the normal equations, and at 1e4 none. The member is the default of no estimator. The leverage check of the regression takes its own QR of the design under this member, so the full gain of the member needs leverage = false on the estimator as well.

Under CrossSectionalTargetRegression, the member runs the same test on the weighted design. A design that passes goes to the target as it stands, so GLM factorises its own Gram matrix, and a design that fails takes the answer of PseudoInverseFallback. So the constructor refuses a target that is_basis_invariant answers false for, as it does under PseudoInverseFallback.

Mathematical definition

\[\begin{align} \mathbf{\Pi}^{\intercal} \mathbf{D}^{\intercal} \mathbf{D} \mathbf{\Pi} &= \mathbf{U}^{\intercal} \mathbf{U}\,, & \hat{\kappa} &\approx \lVert \mathbf{U} \rVert_{1} \lVert \mathbf{U}^{-1} \rVert_{1} \le \epsilon^{-1/4}\,. \end{align}\]

Where:

  • $\mathbf{D}$: Weighted design of one observation, eligible assets × factors.
  • $\mathbf{\Pi}$, $\mathbf{U}$: Permutation and triangular factor of the pivoted Cholesky factorisation of the Gram matrix.
  • $\hat{\kappa}$: Estimate of the condition number of $\mathbf{U}$ in the 1-norm, within a factor of $K$ of the condition number of $\mathbf{D}$. The member solves the normal equations when it holds and the factorisation has full rank.
  • $K$: Number of factors.
  • $\epsilon$: Machine epsilon of the element type of $\mathbf{D}$.

Examples

julia> NormalEquationsSolve()NormalEquationsSolve()

Related

source
PortfolioOptimisers.CrossSectionalSolveResult — Type
struct CrossSectionalSolveResult{__T_f, __T_edof} <: AbstractResult

The answer of the solve of one observation of a Cross-Sectional Regression: the factor returns of the observation and the residual count of its fit.

cross_sectional_solve and cross_sectional_target_solve return it, and a new solve algorithm builds it in its own methods, as the # Interfaces section of AbstractCrossSectionalSolveAlgorithm states. So the type is the contract of the interface, and its constructor checks the count that a solve gives. The regression writes f into its row of factor returns, and edof into its entry of edof of CrossSectionalRegression. A caller reads the fields by name, as res.f or (; f, edof) = res.

Fields

  • f: Factor returns of the observation, one entry per column of its design.
  • edof: Residual count of the fit, the number of rows of the design less the parameters that the fit spent.

Constructors

CrossSectionalSolveResult(f::VecNum, edof::Real)CrossSectionalSolveResult(; f::VecNum, edof::Real)

The positional constructor checks the values. The keyword constructor takes each field by name, with no default, and calls it.

Validation

  • edof is finite. Otherwise, the constructor raises a DomainError. A count below zero passes, because a fit of more columns than rows spends more parameters than it has rows.

Examples

julia> res = PortfolioOptimisers.CrossSectionalSolveResult(; f = [0.5, -0.25], edof = 3);julia> (res.f, res.edof)([0.5, -0.25], 3)julia> PortfolioOptimisers.CrossSectionalSolveResult([0.5, -0.25], NaN)ERROR: DomainError with NaN:isfinite(edof) must hold. Gotedof => NaN[...]

Related

source
PortfolioOptimisers.AbstractUnseenMemberRule — Type
abstract type AbstractUnseenMemberRule <: AbstractAlgorithm

Abstract supertype for the Unseen Member rule of a CrossSectionalFactorPrior, the rule that says what return an Unseen Member gets at an observation.

An Unseen Member is a member of a constrained Factor Family that no asset of positive regression weight loads on at one observation. The exposures lag the returns, so the zero-sum condition of an observation reads the benchmark weights of an earlier one, which can still weight the member after its last asset delists. The data state nothing about its return there. The rule states the return, and with it the zero-sum condition of the observation.

The fit stores the rule on its CrossSectionalFactorModel block. The regression diagnostics and the standard errors of a factor_attribution call the verb of the rule on the stored exposures and weights, so they read the design that the fit regressed on.

Interfaces

In order to implement a new concrete type that works seamlessly with the library, subtype AbstractUnseenMemberRule and implement the following method:

unseen_member_design

  • unseen_member_design(rule::MyUnseenMemberRule, fcb::FactorFamilyBasis, B::Arr3Num, Zl::Arr3Num, W::MatNum) -> UnseenMemberDesignResult: Returns the design that the regression of the prior solves, and the change of each observation that maps its coefficients back to the reduced factor returns.

Arguments

  • rule: The member of the family.
  • fcb: The Factor Family Basis of the lagged exposures, one row per observation of Zl.
  • B: Lagged exposures on the raw axis, assets × factors × observations. Earlier releases held the observations on the first axis. A method written for them reads B[t, :, :], and it must now read B[:, :, t].
  • Zl: Lagged exposures on the reduced axis, which fcb gives from B, in the same layout.
  • W: Cross-sectional weights matrix observations × assets. The pairs of positive weight are the sample of each observation.

Returns

  • res::UnseenMemberDesignResult: Z, the design to regress on, assets × factors × observations, and P, one t => P_t pair per observation that the rule changes, where P_t is a reduced factors × reduced factors matrix, and the factor returns of t are P_t times the coefficients of t. P is empty when the rule changes no observation. The method builds it with the constructor of UnseenMemberDesignResult, which checks each change against the design.

Related

source
PortfolioOptimisers.ZeroUnseenMember — Type
struct ZeroUnseenMember <: AbstractUnseenMemberRule

The Unseen Member rule that gives an Unseen Member a return of zero at its observation, and holds the zero-sum condition of the observation over the other members of its family.

The data state nothing about the return of the member, so the rule states the value that an Empty Factor gets, and that the member gets at an observation that gives it no benchmark weight. The condition over the whole family still holds, because the member adds a zero term to it. The observation is identified again, so its factor returns do not depend on the member that the family drops. unseen_member_design states the change of the design.

Constructors

ZeroUnseenMember() -> ZeroUnseenMember

Examples

julia> ZeroUnseenMember()ZeroUnseenMember()

Related

source
PortfolioOptimisers.SolvedUnseenMember — Type
struct SolvedUnseenMember <: AbstractUnseenMemberRule

The Unseen Member rule that keeps an Unseen Member in the zero-sum condition of its observation, and lets the solve algorithm of the regression state its return.

The member keeps the benchmark weight of the lagged exposures in the condition. Its own column of the raw design is zero, so the member fixes the condition by itself, the condition no longer identifies the other members of the family, and the reduced design of the observation is rank-deficient. The solve algorithm of the Cross-Sectional Regression Estimator then gives the answer. Under PseudoInverseFallback it is the minimum-norm answer on the reduced axis, which depends on the member that the family drops. The residuals of the pairs of positive weight are the residuals of ZeroUnseenMember when the members that the sample sees span the same fitted values, which they do for a one-hot family.

A CrossSectionalFactorModel that a caller builds without a rule takes this one, because its design is then the reduced exposures themselves.

Constructors

SolvedUnseenMember() -> SolvedUnseenMember

Examples

julia> SolvedUnseenMember()SolvedUnseenMember()

Related

source
PortfolioOptimisers.CrossSectionalLinearRegression — Type
struct CrossSectionalLinearRegression{__T_alg, __T_intercept, __T_leverage, __T_ex} <: AbstractCrossSectionalRegressionEstimator

Fits one weighted least squares per observation across the assets, in closed form.

The solve runs on the weighted design $\sqrt{w_{t,i}} \, \boldsymbol{z}_{t,i}$ rather than on the normal matrix $\mathbf{Z}_{t}^{\intercal} \mathbf{W}_{t} \mathbf{Z}_{t}$. The condition number of the normal matrix is the square of the condition number of the weighted design. alg decides what happens when the weighted design is rank deficient.

The square root sets the arithmetic of the solve. A Rational weight has a floating-point square root, so the solve of a Rational panel runs in floating point. f keeps the Rational type of the panel, and holds the floating-point answer written as a Rational, not the exact least-squares answer. A caller who needs the exact answer solves the weighted normal equations in Rational arithmetic.

The leverage check and the solve read one column-pivoted QR of the weighted design, so the check costs a triangular solve on top of the factorisation of the solve. leverage = false skips the check, and with NormalEquationsSolve the fit then takes no QR of a well-conditioned design.

Warning

leverage = false states that no eligible pair has a leverage of one. On a design with such a pair, for example the only member of a level of a one-hot family, the residual of the pair keeps the round-off of the subtraction rather than an exact zero, and no reader of the fit leaves it out. Its idiosyncratic variance is then a round-off too, and a standardised residual that divides by it is noise, under both members of AbstractLeverageOneMask.

Algorithm

  1. Check Z, X and W, and take the eligibility mask, per # Validation of cross_sectional_regression.
  2. For each observation t, gather the eligible assets, their weights w, their exposures A and their returns y.
  3. When intercept is true, subtract the weighted means ybar and xbar from y and from A, so the fit runs through the weighted centroid of the cross-section.
  4. Scale A and y by sqrt.(w), giving the weighted design and the weighted target.
  5. When leverage is true, take the column-pivoted QR of the weighted design, and mark the pairs of leverage one through cross_sectional_leverage_one.
  6. Solve the weighted design through the branch alg selects, on the same factorisation when the member reads one, giving the row t of f.
  7. When intercept is true, set the entry t of b to ybar - dot(f[t, :], xbar).
  8. Subtract the systematic part from X, giving eps.

Fields

  • intercept: intercept: Whether a per-observation intercept is fitted. When false, the regression runs through the origin of the cross-section.
  • leverage: leverage: Whether the fit marks the Leverage-One Pairs of each observation in the h1 field of CrossSectionalRegression, through cross_sectional_leverage_one. When false, the fit runs no leverage check and h1 is nothing, so every reader of the mask reads every residual as data.
  • ex: Parallel execution strategy. It runs the fits of the observations, which are independent problems. Each fit writes its own row, so every executor gives the same result, and a refusal is the one of the first observation that fails.

Constructors

CrossSectionalLinearRegression(;    alg::AbstractCrossSectionalSolveAlgorithm = PseudoInverseFallback(),    intercept::Bool = false,    leverage::Bool = true,    ex::FLoops.Transducers.Executor = ThreadedEx()) -> CrossSectionalLinearRegression

Keywords correspond to the struct's fields.

Examples

julia> CrossSectionalLinearRegression()CrossSectionalLinearRegression        alg ┼ PseudoInverseFallback()  intercept ┼ Bool: false   leverage ┼ Bool: true         ex ┴ Transducers.ThreadedEx{@NamedTuple{}}: Transducers.ThreadedEx()

Related

source
PortfolioOptimisers.CrossSectionalTargetRegression — Type
struct CrossSectionalTargetRegression{__T_tgt, __T_alg, __T_intercept, __T_leverage, __T_ex} <: AbstractCrossSectionalRegressionEstimator

Fits one external regression model per observation across the assets.

The cross-sectional weights reach the model as observation weights, through factory and the target's own kwargs, so both targets of the library, LinearModel and GeneralisedLinearModel, run here unchanged. A caller's own target runs here too, when it states the methods of the # Interfaces section of AbstractRegressionTarget. A target with no weight method is refused, because its fit would ignore the cross-sectional weights. Unlike CrossSectionalLinearRegression, the fit refuses an observation with no eligible asset, because an external model has no cross-section to read.

alg decides what the fit does with a rank-deficient weighted design, as it does for CrossSectionalLinearRegression, and cross_sectional_target_solve states each member. The default PseudoInverseFallback fits the columns that the rank test keeps and projects the answer onto the row space of the weighted design, so a least-squares target gives the factor returns of CrossSectionalLinearRegression. A rank test is needed because a factor set can be dependent to rounding alone, such as a beta that shrinks fully to the mean of its industry, and the factorisation of GLM refuses such a design with a PosDefException. A rank-deficient design also comes from ordinary use. Under SolvedUnseenMember, the design of an observation with an Unseen Member is rank-deficient by construction.

Algorithm

  1. Check Z, X and W, and take the eligibility mask, per # Validation of cross_sectional_regression.
  2. For each observation t, gather the eligible assets, their weights w, their exposures A and their returns y. Refuse when no asset is eligible.
  3. When intercept is true, subtract the weighted means ybar and xbar from y and from A. The target fits no intercept column of its own, so the intercept is recovered from the centroid rather than fitted.
  4. When leverage is true, take the column-pivoted QR of the weighted design sqrt.(w) .* A, and mark the pairs of leverage one through cross_sectional_leverage_one. The rank test of alg reads the same factorisation.
  5. Build the per-observation target with factory(tgt, StatsBase.aweights(w)), fit it through the branch alg selects, and read its coefficients into the row t of f.
  6. When intercept is true, set the entry t of b to ybar - dot(f[t, :], xbar).
  7. Subtract the systematic part from X, giving eps.

The warning of CrossSectionalLinearRegression about leverage = false holds here too.

Fields

  • tgt: Regression model target.
  • intercept: intercept: Whether a per-observation intercept is fitted. When false, the regression runs through the origin of the cross-section.
  • leverage: leverage: Whether the fit marks the Leverage-One Pairs of each observation in the h1 field of CrossSectionalRegression, through cross_sectional_leverage_one. When false, the fit runs no leverage check and h1 is nothing, so every reader of the mask reads every residual as data.
  • ex: Parallel execution strategy. It runs the fits of the observations, which are independent problems. Each fit writes its own row, so every executor gives the same result, and a refusal is the one of the first observation that fails.

Constructors

CrossSectionalTargetRegression(;    tgt::AbstractRegressionTarget = LinearModel(),    alg::AbstractCrossSectionalSolveAlgorithm = PseudoInverseFallback(),    intercept::Bool = false,    leverage::Bool = true,    ex::FLoops.Transducers.Executor = ThreadedEx()) -> CrossSectionalTargetRegression

Keywords correspond to the struct's fields.

Validation

Examples

julia> CrossSectionalTargetRegression()CrossSectionalTargetRegression        tgt ┼ LinearModel            │   kwargs ┴ @NamedTuple{}: NamedTuple()        alg ┼ PseudoInverseFallback()  intercept ┼ Bool: false   leverage ┼ Bool: true         ex ┴ Transducers.ThreadedEx{@NamedTuple{}}: Transducers.ThreadedEx()

Related

source
PortfolioOptimisers.CrossSectionalRegression — Type
struct CrossSectionalRegression{__T_f, __T_eps, __T_n, __T_edof, __T_b, __T_h1, __T_dc, __T_lv} <: AbstractCrossSectionalRegressionResult

Holds the factor returns, the residuals, the eligible asset counts and the optional intercepts of a fitted cross-sectional regression.

The result is a sibling of Regression rather than a widening of it, because the two disagree on what an asset index means. A Regression holds one row per asset, and port_opt_view slices its rows. This result holds one row per observation and one column per asset, so the same index slices its columns. It carries no loadings matrix, because the exposures are the regression's input and an Exposure Estimator produces them.

Mathematical definition

\[\begin{align} \left(b_{t}, \boldsymbol{f}_{t}\right) &= \underset{b, \boldsymbol{f}}{\arg\min} \sum_{i = 1}^{N} w_{t,i} \left(x_{t,i} - b - \boldsymbol{z}_{t,i}^{\intercal} \boldsymbol{f}\right)^{2} \\ \boldsymbol{\varepsilon}_{t} &= \boldsymbol{x}_{t} - b_{t} \boldsymbol{1} - \mathbf{Z}_{t} \boldsymbol{f}_{t}\,, \\ \nu_{t} &= n_{t} - \operatorname{tr}\left(2 \mathbf{S}_{t} - \mathbf{S}_{t}^{\intercal} \mathbf{S}_{t}\right)\,. \end{align}\]

Where:

  • $\boldsymbol{f}_{t}$: Factor returns of observation $t$, the $t$-th row of f.
  • $\boldsymbol{x}_t$: Asset returns for observation $t$, the $t$-th row of the returns matrix.
  • $\boldsymbol{\varepsilon}_{t}$: Residuals of observation $t$, the $t$-th row of eps.
  • $\mathbf{Z}_{t}$: Exposure slice of observation $t$, $N \times K$, one row per asset.
  • $\boldsymbol{z}_{t,i}$: Exposures of asset $i$ at observation $t$, the $i$-th row of $\mathbf{Z}_{t}$.
  • $w_{t,i} \geq 0$: Cross-sectional weight of asset $i$ at observation $t$. A weight of zero excludes the pair from the fit.
  • $b_{t}$: Intercept of observation $t$, the $t$-th entry of b. When b is unset, the minimisation holds $b$ at zero.
  • $\nu_{t}$: Residual count of observation $t$, the $t$-th entry of edof.
  • $n_{t}$: Number of eligible assets at observation $t$, the $t$-th entry of n.
  • $\mathbf{S}_{t}$: Smoother of the fit of observation $t$, the $n_{t} \times n_{t}$ matrix that maps the returns of the eligible assets to their fitted values.
  • $N$: Number of assets.
  • $K$: Number of factors.

Each observation is one independent problem, so a factor return is a cross-sectional quantity and never a time-series one. When the weighted design of an observation is rank deficient, the minimiser is not unique, and the solve algorithm of the estimator picks one of them.

The minimisation is the fit of CrossSectionalLinearRegression and of a least-squares target. A CrossSectionalTargetRegression whose target is not least squares, such as a penalised target or a GeneralisedLinearModel whose link is not the identity, minimises the loss of its target with the same weights. Under an intercept, every fit reads the returns and the exposures centred on their weighted means, and $b_{t}$ is the weighted mean of the returns less the weighted mean of the exposures times $\boldsymbol{f}_{t}$. For a least-squares fit, this is the joint minimiser above.

The fit of an observation spends parameters on its eligible assets, and $\nu_{t}$ is the count that it leaves. It follows from $\mathbb{E}[\mathrm{RSS}_{t}] = \sigma^{2} \operatorname{tr}((\mathbf{I} - \mathbf{S}_{t})^{\intercal} (\mathbf{I} - \mathbf{S}_{t}))$ under a correct mean, so it is the count of the chi-squared law of the residuals. A least-squares fit is a projection onto the span of the weighted design, so $\mathbf{S}_{t}^{\intercal} \mathbf{S}_{t} = \mathbf{S}_{t}$, and it spends the rank of the design, not its number of columns. A rank-deficient observation thus spends fewer parameters than $K$, as the t-statistics of cs_regression_t_stats count it. A penalised fit spends fewer still, and residual_count reads its count. The intercept spends one more parameter at an observation with an eligible asset, because the design is centred on the weighted mean, and the fit of the centred design leaves the weighted mean alone. A Cross-Sectional Factor Prior reads edof for the degrees of freedom of each idiosyncratic variance, through cross_sectional_variance_counts.

A pair whose leverage is one has a direction of the design of its own, for example the only member of a level of a one-hot family. The fit reproduces its return whatever the return is, so its residual is zero by construction and tells nothing about its idiosyncratic risk. h1 marks these pairs, and a least-squares fit writes an exact zero there in place of the round-off of the subtraction.

A factor whose column DependentColumnDrop dropped at an observation takes a return of zero by the rule, not from the data. dc marks these pairs of observation and factor, so a diagnostic of the fit gives such a factor a variance of zero, and it reads the other factors on the columns that the fit kept.

An Empty Factor has no nonzero exposure at a pair of positive weight, so the fit cannot estimate it at any observation and gives it a return of zero. lv records which factors the fit estimated. A refit of a CrossSectionalFactorPrior under new scenario weights reads it, so it fits the factor prior on the same factors as the fit.

Fields

  • f: Factor returns matrix observations × factors. Row t holds the coefficients of the cross-sectional fit of observation t.
  • eps: Residual matrix observations × assets. It is the part of the returns the exposures do not explain, and an entry of an excluded pair is whatever the arithmetic of that pair produced, so a missing return leaves a missing residual.
  • n: Count of assets that entered each fit, of length observations. An asset enters when its cross-sectional weight is positive.
  • edof: Residual count of each fit, of length observations, or nothing. An entry is the count of assets of the fit less the parameters that the fit spent: the rank of the weighted design for a least-squares fit, the count that residual_count reads for every other fit, and one more for the intercept. nothing states no count, as a result that a caller builds by hand does, and a Cross-Sectional Factor Prior then states no degrees of freedom of its idiosyncratic variances.
  • b: b: Regression intercept vector.
  • h1: Leverage-one mask observations × assets, or nothing. An entry is true where the pair entered the fit with a leverage of one, so the fit reproduced its return and its residual is zero by construction. nothing states no mark, as a result that a caller builds by hand does.
  • dc: Dropped-column mask observations × factors, or nothing. An entry is true where DependentColumnDrop dropped the column of the factor at the observation, so the fit gave the factor a return of zero by the rule. nothing states no drop, as a fit under every other solve algorithm, and a result that a caller builds by hand, does.
  • lv: Live-factor mask, one entry per column of f, or nothing. An entry is true where the factor is not an Empty Factor, so the fit estimated its return, and false where the fit gave it a return of zero because no pair of positive weight loads on it. nothing states no record, as a result that a caller builds by hand, or that a carry fold joins from its steps, does.

Constructors

CrossSectionalRegression(;    f::MatNum,    eps::MatNum,    n::AbstractVector{<:Integer},    edof::Option{<:VecNum} = nothing,    b::Option{<:VecNum} = nothing,    h1::Option{<:AbstractMatrix{Bool}} = nothing,    dc::Option{<:AbstractMatrix{Bool}} = nothing,    lv::Option{<:AbstractVector{Bool}} = nothing) -> CrossSectionalRegression

Keywords correspond to the struct's fields.

Validation

  • !isempty(f), !isempty(eps) and !isempty(n).
  • size(f, 1) == size(eps, 1) == length(n).
  • all(x -> x >= 0, n).
  • If provided, length(edof) == length(n).
  • If provided, !isempty(b), and length(b) == size(f, 1).
  • If provided, size(h1) == size(eps).
  • If provided, size(dc) == size(f).
  • If provided, length(lv) == size(f, 2).

View parameters

CrossSectionalRegression defines its own port_opt_view method rather than deriving one from field tags.

  • eps and h1 are sliced on their second axis, which is the asset axis of a cross-sectional result.
  • f, n, edof, b, dc and lv pass through unchanged. Each is indexed by observation, by factor or by both, and neither axis follows an asset selection.

Examples

julia> CrossSectionalRegression(; f = [1.0 2.0; 3.0 4.0], eps = [0.1 0.2 0.3; 0.4 0.5 0.6],                                n = [3, 3], edof = [1, 1])CrossSectionalRegression     f ┼ 2×2 Matrix{Float64}   eps ┼ 2×3 Matrix{Float64}     n ┼ Vector{Int64}: [3, 3]  edof ┼ Vector{Int64}: [1, 1]     b ┼ nothing    h1 ┼ nothing    dc ┼ nothing    lv ┴ nothing

Related

source

Functions

PortfolioOptimisers.cross_sectional_regression — Function
cross_sectional_regression(cre::Union{CrossSectionalLinearRegression,
                                      CrossSectionalTargetRegression}, Z::Arr3Num,
                           X::MatNum, W::MatNum) -> CrossSectionalRegression
cross_sectional_regression(csr::CrossSectionalRegression, args...) -> CrossSectionalRegression

Fit one regression per observation across the assets, or return a fitted result unchanged.

The verb is its own rather than a fourth argument of regression, because regression(re::Regression, args...) is a greedy passthrough that returns its first argument for any trailing arguments, so a time-series result handed to a cross-sectional call would return silently instead of raising.

The weight matrix W is an argument rather than a field, because a two-pass weighting scheme calls the estimator twice on one design with two different weight matrices, and a policy stored on the estimator would force a second estimator object or a mutation.

A refusal names an observation by its row of X. The verb fits cross_sectional_block_regression with no observation before X.

Algorithm

  1. Take the eligibility mask through cross_sectional_design_mask.
  2. For each observation t, gather the eligible assets, their weights w, their exposures A and their returns y into buffers of the task, and record their count in n. The observations are independent problems, and cre.ex runs them through cross_sectional_foreach, which gives each task its buffers once. Each writes its own row, so every executor gives the same result.
  3. When cre.intercept is true, take the weighted means ybar and xbar of y and of A, and subtract them. An observation with no eligible asset takes zero for both.
  4. Scale A by sqrt.(w), giving the weighted design. When cre.leverage is true or the solve algorithm is DependentColumnDrop, take its column-pivoted QR once. When cre.leverage is true, mark the eligible assets of leverage one in the row t of h1, through cross_sectional_leverage_one on that factorisation. Otherwise h1 is nothing.
  5. Take the factor returns and the residual count of the observation through cross_sectional_coefficients, on the same factorisation, and write the factor returns into the row t of f. An observation with no eligible asset takes zero factor returns under CrossSectionalLinearRegression, except under RankDeficiencyRefusal, whose rank test reads an empty design as rank zero and refuses it by name.
  6. When cre.intercept is true, write ybar - dot(f[t, :], xbar) into the entry t of b. Under DependentColumnDrop, mark the dropped columns of the observation in the row t of dc, through cross_sectional_mark_dropped!.
  7. Write the residual count into the entry t of edof, less one for the intercept when cre.intercept is true and the observation has an eligible asset. edof takes the working type of f, because a penalised fit counts a real number of parameters, and the integer count of a least-squares fit is exact in that type.
  8. Subtract the systematic part, through cross_sectional_systematic, from X, giving eps.
  9. When h1 is set and cross_sectional_least_squares answers true, write an exact zero into eps at every pair that h1 marks. The fit reproduced the return of that pair, so zero is its residual in exact arithmetic.

Arguments

  • cre: Cross-sectional regression estimator.
  • csr: A cross-sectional regression result.
  • Z::Arr3Num: Exposure tensor assets × factors × observations.
  • X::MatNum: Asset returns matrix observations × assets.
  • W::MatNum: Cross-sectional weights matrix observations × assets.
  • args...: Additional positional arguments (ignored by the passthrough).

Validation

Returns

  • csr::CrossSectionalRegression: The fitted result, or the input result unchanged.

Examples

julia> Z = reshape([1.0, 0.0, 0.5, 0.0, 1.0, 0.5], 3, 2, 1);julia> cross_sectional_regression(CrossSectionalLinearRegression(), Z, [1.0 2.0 1.5], ones(1, 3))CrossSectionalRegression     f ┼ 1×2 Matrix{Float64}   eps ┼ 1×3 Matrix{Float64}     n ┼ Vector{Int64}: [3]  edof ┼ Vector{Float64}: [1.0]     b ┼ nothing    h1 ┼ 1×3 BitMatrix    dc ┼ nothing    lv ┴ nothing

Related

source
PortfolioOptimisers.port_opt_view — Method
port_opt_view(csr::CrossSectionalRegression, i, args...)

Return a view of a CrossSectionalRegression result, selecting only the assets indexed by i.

Algorithm

  1. Take a column view of eps over i, giving the residuals of the selected assets. The asset axis of a cross-sectional result is the second one, because a row of eps is one observation.
  2. Take the same view of h1 when it is set, through nothing_scalar_array_view_odd_order.
  3. Build a new CrossSectionalRegression from the two views and the six untouched fields, which re-runs every guard of the constructor.

Arguments

  • csr: A cross-sectional regression result.
  • i: Indices of the assets to select.
  • args...: Additional positional arguments (ignored).

Returns

  • csr::CrossSectionalRegression: A new result whose residuals are restricted to the selected assets.

Examples

julia> csr = CrossSectionalRegression(; f = [1.0 2.0], eps = [0.1 0.2 0.3], n = [3], edof = [1])CrossSectionalRegression     f ┼ 1×2 Matrix{Float64}   eps ┼ 1×3 Matrix{Float64}     n ┼ Vector{Int64}: [3]  edof ┼ Vector{Int64}: [1]     b ┼ nothing    h1 ┼ nothing    dc ┼ nothing    lv ┴ nothingjulia> PortfolioOptimisers.port_opt_view(csr, [1, 3])CrossSectionalRegression     f ┼ 1×2 Matrix{Float64}   eps ┼ 1×2 SubArray{Float64, 2, Matrix{Float64}, Tuple{Base.Slice{Base.OneTo{Int64}}, Vector{Int64}}, false}     n ┼ Vector{Int64}: [3]  edof ┼ Vector{Int64}: [1]     b ┼ nothing    h1 ┼ nothing    dc ┼ nothing    lv ┴ nothing

Related

source
PortfolioOptimisers.cross_sectional_solve — Function
cross_sectional_solve(alg::AbstractCrossSectionalSolveAlgorithm, A::MatNum, y::VecNum,
                      t::Integer, F = nothing, ws::NamedTuple = (;))
    -> CrossSectionalSolveResult

Solve the weighted design A against the weighted target y through the branch alg selects, and count the residual degrees of freedom of the fit.

Every answer is a least-squares fit, so its fitted values are the projection of y onto the span of the columns that it solves. The projection spends the rank r of those columns, read by cross_sectional_rank, and the count is size(A, 1) - r. The t-statistics of cs_regression_t_stats read the same rank.

The fifth argument is the column-pivoted QR of A that cross_sectional_regression took, or nothing. A member of the library that reads a QR reads F, so the fit factorises the design once. The sixth argument is the workspace of the task, a NamedTuple whose optional fields a member reads with get: the work vector b of a solve through F, which cross_sectional_full_solve solves in, and the Gram matrix gram and the right side v of the normal equations, which cross_sectional_gram_factor and cross_sectional_normal_solve fill. Each gives the bits of the allocating form. Every member takes the six arguments in one method, with F = nothing and ws = (;) as defaults, and a member that reads neither ignores them.

Algorithm

  1. UncheckedSolve returns A \ y, with no rank test. A square A that \ solves has full rank. On every other A, \ solves through the column-pivoted QR of A, so the method takes that factorisation, solves through it, and reads the rank off it, with one factorisation of the design.
  2. MinimumNormSolve returns LinearAlgebra.pinv(A) * y, and counts the rank that cross_sectional_rank reads.
  3. PseudoInverseFallback takes the column-pivoted QR of A and reads cross_sectional_rank off it. It returns cross_sectional_full_solve, the answer of A \ y through the same factorisation, when the rank equals the factor count, and LinearAlgebra.pinv(A) * y otherwise.
  4. RankDeficiencyRefusal takes the same factorisation, refuses a rank-deficient A with assert_cross_sectional_rank, and returns cross_sectional_full_solve otherwise.
  5. DependentColumnDrop takes the same factorisation and reads the rank off it. It returns cross_sectional_full_solve when the rank equals the factor count, and the answer of cross_sectional_column_drop on the pivot of the factorisation otherwise.
  6. NormalEquationsSolve takes the pivoted Cholesky factorisation of the Gram matrix of A through cross_sectional_gram_factor. It solves the normal equations through it with cross_sectional_normal_solve when the factorisation passes the test of the member, with a count of size(A, 1) - size(A, 2). Otherwise it returns the answer of PseudoInverseFallback, which reads F and ws. Its own solve reads no QR.

Arguments

  • alg: Cross-sectional solve algorithm.
  • A::MatNum: Weighted design of one observation, eligible assets × factors.
  • y::VecNum: Weighted target of one observation, of length eligible assets.
  • t::Integer: Index of the observation, named by the refusal of RankDeficiencyRefusal.
  • F: Column-pivoted QR of A, as LinearAlgebra.qr(A, LinearAlgebra.ColumnNorm()) returns it, or nothing to take it here.
  • ws::NamedTuple: Workspace, (; b, gram, v) or any part of it, overwritten. (;) allocates every buffer.

Validation

Returns

  • res::CrossSectionalSolveResult: f, the factor returns of the observation, of length size(A, 2), and edof, the residual count of the fit, size(A, 1) less the rank of the columns that it solves.

Related

source
PortfolioOptimisers.cross_sectional_target_solve — Function
cross_sectional_target_solve(alg::UncheckedSolve, tgt::AbstractRegressionTarget,
                             obs::NamedTuple) -> CrossSectionalSolveResult
cross_sectional_target_solve(alg::RankDeficiencyRefusal, tgt::AbstractRegressionTarget,
                             obs::NamedTuple) -> CrossSectionalSolveResult
cross_sectional_target_solve(alg::Union{DependentColumnDrop, PseudoInverseFallback},
                             tgt::AbstractRegressionTarget, obs::NamedTuple)
    -> CrossSectionalSolveResult
cross_sectional_target_solve(alg::NormalEquationsSolve, tgt::AbstractRegressionTarget,
                             obs::NamedTuple) -> CrossSectionalSolveResult

Fit the target of one observation of a CrossSectionalTargetRegression through the branch alg selects, and read the residual count of the fit.

The count is residual_count of the fitted target and the design that it fitted. An unpenalised fit on the columns that the rank test keeps spends one parameter for each of them, so a rank-deficient observation counts its rank, as cross_sectional_solve counts it. A penalised fit states its own count. Under UncheckedSolve the target fits the full design, and its own count decides: a fit of GLM counts the rank that its pivot keeps, and the fallback of residual_count counts every column.

Algorithm

  1. UncheckedSolve fits tgt to A and y, with no rank test.
  2. RankDeficiencyRefusal takes the column-pivoted QR of D, refuses a rank-deficient D with assert_cross_sectional_rank, and fits tgt to A and y through cross_sectional_target_coef on the same factorisation otherwise.
  3. DependentColumnDrop and PseudoInverseFallback take the column-pivoted QR of D, and read its rank r off it with cross_sectional_rank. Every later step reads the same factorisation. When r equals the factor count, they fit tgt to A and y through cross_sectional_target_coef. Otherwise they fit tgt the same way to the columns of A that cross_sectional_column_drop keeps, and PseudoInverseFallback projects the answer onto the row space of D with cross_sectional_row_space.
  4. NormalEquationsSolve tests D through cross_sectional_gram_factor. It fits tgt to A and y when D passes, so the target factorises its own Gram matrix, and it takes the answer of PseudoInverseFallback otherwise.

A member that takes a QR of D reads F of obs when the regression already took it. So every member but UncheckedSolve and NormalEquationsSolve fits a basis-invariant target at the rank and at the precision of D, which the fit of CrossSectionalLinearRegression reads too.

The constructor of CrossSectionalTargetRegression refuses MinimumNormSolve, so no method takes it.

Arguments

  • alg: Cross-sectional solve algorithm.

  • tgt: Regression target of the observation, which carries its weights.

  • obs::NamedTuple: The observation, (; A, y, D, t, F):

    • A::MatNum: Exposures of the eligible assets, eligible assets × factors, already demeaned when an intercept is fitted.
    • y::VecNum: Returns of the eligible assets, already demeaned when an intercept is fitted.
    • D::MatNum: Weighted design sqrt.(w) .* A, which the rank test reads.
    • t::Integer: Index of the observation, named by the refusal of RankDeficiencyRefusal.
    • F: Column-pivoted QR of D that the regression took for its leverage check, or nothing. A tuple with no F reads as nothing.

Validation

Returns

  • res::CrossSectionalSolveResult: f, the factor returns of the observation, of length size(A, 2), and edof, the residual count of the fit.

Related

source
PortfolioOptimisers.assert_cross_sectional_target_solve — Function
assert_cross_sectional_target_solve(alg::MinimumNormSolve, tgt::AbstractRegressionTarget)
assert_cross_sectional_target_solve(alg::Union{PseudoInverseFallback, NormalEquationsSolve},
                                    tgt::AbstractRegressionTarget)
assert_cross_sectional_target_solve(alg::AbstractCrossSectionalSolveAlgorithm,
                                    tgt::AbstractRegressionTarget)

Check that a CrossSectionalTargetRegression can solve a rank-deficient design of tgt under alg.

Algorithm

The method that Julia selects is the algorithm.

  1. MinimumNormSolve is refused. On a target its answer is the answer of PseudoInverseFallback.
  2. PseudoInverseFallback is refused when is_basis_invariant answers false for tgt. Its projection keeps the linear predictor of the fit, so it is the answer of least norm only for a target that reads the design through the linear predictor alone. NormalEquationsSolve gives a design that fails its test to PseudoInverseFallback, so the same rule refuses it.
  3. Every other member passes.

Arguments

  • alg: Cross-sectional solve algorithm.
  • tgt: Regression target.

Validation

Returns

  • nothing.

Related

source
PortfolioOptimisers.cross_sectional_rank — Function
cross_sectional_rank(A::MatNum; R::MatNum = LinearAlgebra.qr(A, LinearAlgebra.ColumnNorm()).R) -> Int

Return the numerical rank of A, read off the diagonal of a column-pivoted QR.

The test is written out rather than delegated to LinearAlgebra.rank(::QRPivoted), which needs Julia 1.12 while this package supports 1.11. A caller that already factorised A passes the triangular factor R of its factorisation, as UncheckedSolve does, so it factorises the design once.

Algorithm

  1. Take the triangular factor R of the column-pivoted LinearAlgebra.qr of A, unless the caller passes it.
  2. Return 0 when R has no row or no column, because the factorisation has no pivot to read. The magnitudes of the diagonal of its R are non-increasing, so they rank the columns by how much each adds to the span.
  3. Count the leading diagonal entries above min(size(A)...) * eps(real(eltype(R))) * abs(R[1, 1]), giving the numerical rank. The tolerance is the one LinearAlgebra.rank applies to a pivoted QR.

Arguments

  • A::MatNum: Weighted design of one observation, eligible assets × factors.
  • R::MatNum: Triangular factor of the column-pivoted QR of A.

Returns

  • r::Int: Numerical rank of A.

Related

source