Elastic net: private API

These are the fitted model of ElasticNet, its counts and scores, and the steps of its fit. elastic_net_sweeps runs the coordinate descent, elastic_net_polish solves the optimum on a sign pattern exactly, and elastic_net_counts reads the effective degrees of freedom off the equations of the active set.

PortfolioOptimisers.ElasticNetFit — Type
struct ElasticNetFit{__T_b, __T_gamma, __T_df, __T_rss, __T_nobs, __T_rdof} <: AbstractResult

Holds the coefficients of a fitted ElasticNet, the strength at which the fit ran, and the counts that its scores read.

StatsAPI.fit(::ElasticNet, ::MatNum, ::VecNum) returns it. StatsAPI.coef returns b, and StatsAPI.predict multiplies a design by b. StatsAPI.dof, StatsAPI.nobs, StatsAPI.deviance, StatsAPI.loglikelihood, StatsAPI.aic, StatsAPI.aicc and StatsAPI.bic read the counts, in the forms that GLM uses for a fitted LinearModel. residual_count returns rdof.

Mathematical definition

\[\begin{align} \mathbf{H} &= \mathbf{P}_{\mathcal{A}} \left(\mathbf{P}_{\mathcal{A}}^\intercal \mathbf{V} \mathbf{P}_{\mathcal{A}} + \gamma (1 - \vartheta) \mathbf{D}_{\mathcal{A}}\right)^{-1} \mathbf{P}_{\mathcal{A}}^\intercal \mathbf{V}\,,\\ \widehat{\mathrm{df}} &= \operatorname{tr}(\mathbf{H}) + \delta\,,\\ \nu &= T - \operatorname{tr}\left(2 \mathbf{S} - \mathbf{S}^\intercal \mathbf{S}\right)\,,\quad \mathbf{S} = \mathbf{V}^{1/2} \mathbf{H} \mathbf{V}^{-1/2}\,. \end{align}\]

Where:

  • $\mathbf{H}$: Smoother of the fit on its active set. The fitted response is $\mathbf{H} \boldsymbol{y}$ while the active set and the strength do not move.
  • $\mathbf{P}_{\mathcal{A}}$: Columns of the active set $\mathcal{A}$, the columns whose coefficient is nonzero and the intercept column.
  • $\mathbf{D}_{\mathcal{A}}$: Diagonal matrix with one for each penalised column of the active set and zero for the intercept column.
  • $\mathbf{V}$: Diagonal matrix of the weights of the observations of the elastic net, the identity when the fit has no observation weights.
  • $\gamma$: Regularisation strength of the elastic net.
  • $\vartheta$: Share of the $L_1$ penalty in the elastic net, the theta field of ElasticNet.
  • $\widehat{\mathrm{df}}$: Effective degrees of freedom of the fit, df.
  • $\delta$: Term of a strength that reads the response, which the sdiv field of the target adds through strength_divergence. It is zero under sdiv = nothing.
  • $\nu$: Residual count, rdof.
  • $\mathbf{S}$: Smoother of the weighted rows. It is $\mathbf{H}$ itself without weights, and it is symmetric.
  • $T$: Number of observations.

Remark (The elastic net) after Equation (28) of [109] states the trace of $\mathbf{H}$ as an unbiased estimate of the degrees of freedom at a fixed strength. Remark (The lasso with intercept) after it extends the result to an unpenalised intercept.

The residual count follows from the mean of the weighted residual sum of squares. With $\boldsymbol{y} = \boldsymbol{\mu} + \boldsymbol{\varepsilon}$ and $\mathbf{V}^{1/2} \boldsymbol{\varepsilon}$ of covariance $\sigma^{2} \mathbf{I}$,

\[\begin{align} \mathbb{E}\left[\mathrm{RSS}\right] &= \sigma^{2} \operatorname{tr}\left((\mathbf{I} - \mathbf{S})^\intercal (\mathbf{I} - \mathbf{S})\right) + \lVert \mathbf{V}^{1/2} (\mathbf{I} - \mathbf{H}) \boldsymbol{\mu} \rVert^{2}\,. \end{align}\]

So when the bias term is small, $\mathrm{RSS} / \sigma^{2}$ has the mean $\nu$, and $\nu$ is the count of the chi-squared law that ResidualInflation reads. For a projection, $\mathbf{S}^\intercal \mathbf{S} = \mathbf{S}$, and $\nu = T - p$, the count of a fitted LinearModel with $p$ columns. A ridge part shrinks $\mathbf{S}$, so $\nu$ is below $T - \operatorname{tr}(\mathbf{H})$, and the count $T - \operatorname{tr}(\mathbf{H})$ overstates the residual count. The term $\delta$ does not enter $\nu$, because no source states the residual law of a fit whose strength reads the response.

The scores take the residual variance at its maximum-likelihood value $\mathrm{RSS} / n$, with $n$ the sum of the frequency weights of glm_weights. StatsAPI.dof adds one to $\widehat{\mathrm{df}}$ for that variance, as GLM adds one to the rank of a fitted LinearModel. Equation (2.5) of [69] states AICc of a linear smoother with $\operatorname{tr}(\mathbf{H})$ in place of the number of columns, and StatsAPI.aicc with this dof is $n$ times that form plus a constant. Equation (2.16) of [70] proposes BIC with the effective degrees of freedom for the lasso, and does not derive it.

Fields

  • b: The coefficients on the columns of the design of the caller, one per column.
  • gamma: The regularisation strength $\gamma$ that the strength rule gave, on the standardised columns when the target has a ve.
  • df: The effective degrees of freedom of the fit, the trace of the smoother on the active set plus the term of sdiv.
  • rss: The weighted residual sum of squares of the fit, under the frequency weights of glm_weights.
  • nobs: The number of observations, the sum of the frequency weights, or the number of rows without weights.
  • rdof: The residual count, the number of rows less $\operatorname{tr}(2 \mathbf{S} - \mathbf{S}^\intercal \mathbf{S})$.

Related

References

  • [109] R. J. Tibshirani and J. Taylor. Degrees of freedom in lasso problems. The Annals of Statistics 40, 1198–1232 (2012). Remark (The elastic net) after Equation (28), and Remark (The lasso with intercept).
  • [69] C. M. Hurvich, J. S. Simonoff and C.-L. Tsai. Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 60, 271–293 (1998). Equation (2.5). Page 273 cites Section 3.5 of [298] for the effective degrees of freedom of a linear smoother, and the residual count above follows the derivation of this docstring. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
  • [70] H. Zou, T. Hastie and R. Tibshirani. On the "degrees of freedom" of the lasso. The Annals of Statistics 35, 2173–2192 (2007). Equation (2.16).
  • [299] T. Hastie, R. Tibshirani and J. Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd edition (Springer, 2009). Section 7.6, Equation (7.32).
source
StatsAPI.dof — Method
StatsAPI.dof(m::ElasticNetFit) -> Real
StatsAPI.nobs(m::ElasticNetFit) -> Real
StatsAPI.deviance(m::ElasticNetFit) -> Real
StatsAPI.loglikelihood(m::ElasticNetFit) -> Real
StatsAPI.aic(m::ElasticNetFit) -> Real
StatsAPI.aicc(m::ElasticNetFit) -> Real
StatsAPI.bic(m::ElasticNetFit) -> Real

Return the counts and the scores of a fitted ElasticNet.

ElasticNetFit is not a StatsAPI.StatisticalModel, so each score has a method of its own, in the form of StatsAPI. The forms match GLM for a fitted LinearModel under the same weights, with the effective degrees of freedom in place of the rank.

Mathematical definition

\[\begin{align} k &= \widehat{\mathrm{df}} + 1\,,\\ \ln \hat{L} &= -\frac{n}{2} \left(\ln \frac{2 \pi\, \mathrm{RSS}}{n} + 1\right)\,,\\ \mathrm{AIC} &= 2k - 2\ln\hat{L}\,,\\ \mathrm{AICC} &= \mathrm{AIC} + \frac{2k(k+1)}{n - k - 1}\,,\\ \mathrm{BIC} &= k\ln n - 2\ln\hat{L}\,. \end{align}\]

Where:

  • $k$: StatsAPI.dof, the effective degrees of freedom plus one for the residual variance.
  • $\widehat{\mathrm{df}}$: m.df.
  • $\hat{L}$: Maximum of the Gaussian likelihood, at the residual variance $\mathrm{RSS} / n$.
  • $\mathrm{RSS}$: StatsAPI.deviance, the weighted residual sum of squares m.rss.
  • $n$: StatsAPI.nobs, m.nobs.

Equation (2.5) of [69] states AICc with the residual variance estimated, and StatsAPI.aicc is $n$ times that form plus $n \ln 2\pi$. Equation (2.16) of [70] writes BIC with a known error variance $\sigma^{2}$, as $\mathrm{RSS} / (n \sigma^{2}) + \ln(n)\, \widehat{\mathrm{df}} / n$. The form here puts the maximum-likelihood estimate $\mathrm{RSS} / n$ in place of $\sigma^{2}$, as GLM does for a fitted LinearModel.

Arguments

  • m: The fitted model.

Returns

  • x::Real: The count or the score.

Related

References

  • [69] C. M. Hurvich, J. S. Simonoff and C.-L. Tsai. Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 60, 271–293 (1998). Equation (2.5).
  • [70] H. Zou, T. Hastie and R. Tibshirani. On the "degrees of freedom" of the lasso. The Annals of Statistics 35, 2173–2192 (2007). Equation (2.16).
source
PortfolioOptimisers.elastic_net_design — Function
elastic_net_design(tgt::ElasticNet, X::MatNum) -> Tuple

Return the standardised design of an ElasticNet, the design with each row weighted, the scale of the columns, and the weights of the rows.

Algorithm

  1. When tgt.ve is nothing, set the scale s to nothing. Otherwise take the standard deviation of each column of X under tgt.ve, and put one in place of a zero and in place of the intercept column tgt.ic. That is s.
  2. Divide each column of X by its entry of s, which gives P. When s is nothing, P is X.
  3. Resolve tgt.w against X with get_observation_weights, and scale it with glm_weights, which gives v. When v is nothing, Q is P. Otherwise multiply each row of P by its entry of v, which gives Q.

Arguments

  • tgt: The regression target.
  • X: The design matrix observations × regressors.

Returns

  • (P, Q, s, v)::Tuple: The standardised design $\mathbf{P}$, the weighted design $\mathbf{V} \mathbf{P}$, the scale of the columns or nothing, and the frequency weights of the rows or nothing.

Related

source
PortfolioOptimisers.elastic_net_sweeps — Function
elastic_net_sweeps(tgt::ElasticNet, P::MatNum, Q::MatNum, y::VecNum, z::VecNum,
                   r::VecNum, nrm2::VecNum, a::VecNum, ridge::VecNum,
                   pen::AbstractVector{Bool}) -> VecNum

Return the coefficients of an ElasticNet by cyclic coordinate descent from z, with an exact solve at the end of each sweep.

Mathematical definition

The minimiser of the objective of ElasticNet over one coefficient, with the others fixed, is

\[\begin{align} z_k &= \frac{S\left(\mathbf{P}_k^\intercal \mathbf{V} \boldsymbol{r}_k,\ a_k\right)}{\mathbf{P}_k^\intercal \mathbf{V} \mathbf{P}_k + d_k}\,,\quad S(v, a) = \operatorname{sign}(v) \max(\lvert v \rvert - a, 0)\,,\\ a_k &= \gamma \vartheta\, \mathbb{1}[k \in \mathcal{K}]\,,\quad d_k = \gamma (1 - \vartheta)\, \mathbb{1}[k \in \mathcal{K}]\,. \end{align}\]

Where:

  • $\boldsymbol{z}$: Coefficients of the elastic net, one per column of $\mathbf{P}$. Coefficient $k$ is $z_k$.
  • $\mathbf{P}$: Regressors of the elastic net, one row per observation and one column per regressor. Column $k$ is $\mathbf{P}_k$.
  • $\mathbf{V}$: Diagonal matrix of the weights of the observations of the elastic net, the identity when the fit has no observation weights.
  • $\boldsymbol{r}_k$: Residual of the response $\boldsymbol{y}$ without the term of column $k$.
  • $S$: Soft threshold.
  • $a_k$, $d_k$: Threshold and ridge term of column $k$, both zero at the intercept column.
  • $\gamma$: Regularisation strength of the elastic net.
  • $\vartheta$: Share of the $L_1$ penalty in the elastic net, the theta field of ElasticNet.
  • $\mathcal{K}$: Penalised columns of the elastic net, every column of $\mathbf{P}$ but the intercept column that an estimator names.

At the intercept column the update is the least-squares coefficient of that column on the residual.

Algorithm

  1. Take den, the denominator of each coordinate update.
  2. Sweep the columns in order. For each column k, compute rho, the product of the column of Q with the residual without its own term, and apply the coordinate update, which gives znew. Update r, and record delta, the largest change of a coefficient in the sweep.
  3. Try the exact solve of elastic_net_polish on the sign pattern of z. When it gives the optimum, return it.
  4. Repeat steps 2 and 3 until delta is below tgt.tol or tgt.iters sweeps have run, and then return z.

Arguments

  • tgt: The regression target, which gives iters and tol.
  • P: The standardised design, observations × columns.
  • Q: The weighted design, $\mathbf{V} \mathbf{P}$.
  • y: The response.
  • z: The coefficients of the start point. The function changes them.
  • r: The residual of the start point. The function changes it.
  • nrm2: The weighted squared norm of each column.
  • a: The threshold of each column.
  • ridge: The ridge term of each column.
  • pen: Whether each column is penalised.

Returns

  • z::VecNum: The coefficients on the columns of P.

Related

source
PortfolioOptimisers.elastic_net_polish — Function
elastic_net_polish(P::MatNum, Q::MatNum, y::VecNum, z::VecNum, a::VecNum,
                   ridge::VecNum, pen::AbstractVector{Bool}) -> Option{VecNum}

Returns the exact elastic-net optimum on the sign pattern of z when that pattern is the pattern of the optimum, and nothing otherwise.

A point that meets the optimality conditions below minimises the objective of ElasticNet, because the objective is convex. For theta < 1 the objective is strictly convex in the penalised coefficients, and the point is its only minimiser. At theta = 1 the objective is the lasso, and it can have more than one minimiser. Then the equations of the active columns are singular when the pattern holds more columns than there are observations, and the function returns nothing for such a pattern. The intercept column is always active, and its sign is free.

Mathematical definition

With $\mathcal{A}$ the active columns, the optimum on the sign pattern $\boldsymbol{s}$ solves

\[\begin{align} \left(\mathbf{P}_{\mathcal{A}}^\intercal \mathbf{V} \mathbf{P}_{\mathcal{A}} + \gamma (1 - \vartheta) \mathbf{D}_{\mathcal{A}}\right) \boldsymbol{z}_{\mathcal{A}} &= \mathbf{P}_{\mathcal{A}}^\intercal \mathbf{V} \boldsymbol{y} - \gamma \vartheta\, \boldsymbol{s}\,, \end{align}\]

and it is the optimum of the objective when

\[\begin{align} \operatorname{sign}(z_k) &= s_k\,,\quad k \in \mathcal{A} \cap \mathcal{K}\,,\\ \lvert \mathbf{P}_k^\intercal \mathbf{V} (\boldsymbol{y} - \mathbf{P}_{\mathcal{A}} \boldsymbol{z}_{\mathcal{A}}) \rvert &\leq \gamma \vartheta\,,\quad k \notin \mathcal{A}\,. \end{align}\]

Where:

  • $\mathcal{A}$: Active set, the columns whose coefficient in $\boldsymbol{z}$ is nonzero, and the intercept column.
  • $\mathbf{P}_{\mathcal{A}}$, $\boldsymbol{z}_{\mathcal{A}}$: Columns of the regressors and coefficients of the active set.
  • $\mathbf{D}_{\mathcal{A}}$: Diagonal matrix with one for each penalised column of the active set and zero for the intercept column.
  • $\boldsymbol{s}$: Signs of the coefficients of the active set in $\boldsymbol{z}$, zero for the intercept column.
  • $\boldsymbol{z}$: Coefficients of the elastic net, one per column of $\mathbf{P}$. Coefficient $k$ is $z_k$.
  • $\mathbf{P}$: Regressors of the elastic net, one row per observation and one column per regressor. Column $k$ is $\mathbf{P}_k$.
  • $\mathbf{V}$: Diagonal matrix of the weights of the observations of the elastic net, the identity when the fit has no observation weights.
  • $\boldsymbol{y}$: Response of the elastic net, one entry per observation.
  • $\gamma$: Regularisation strength of the elastic net.
  • $\vartheta$: Share of the $L_1$ penalty in the elastic net, the theta field of ElasticNet.
  • $\mathcal{K}$: Penalised columns of the elastic net, every column of $\mathbf{P}$ but the intercept column that an estimator names.

Algorithm

  1. Find active, the nonzero coefficients of z and the column that pen does not penalise, and their indices A. When A is empty, return nothing.
  2. Take s, the signs of the coefficients of A, zero at the intercept column. Factorise the matrix of the equations of the active columns, which gives F. When the matrix is singular, return nothing.
  3. Solve the equations with F, which gives zA. When a penalised coefficient of zA does not keep its sign in s, return nothing.
  4. Write zA into zp, zero elsewhere, and compute corr, the product of each column of Q with the residual.
  5. When an inactive column has abs(corr[k]) > a[k], return nothing. Otherwise return zp.

Arguments

  • P: The standardised design, observations × columns.
  • Q: The weighted design, $\mathbf{V} \mathbf{P}$.
  • y: The response.
  • z: The coefficients whose sign pattern the function tries.
  • a: The threshold of each column, $\gamma \vartheta$ or zero at the intercept column.
  • ridge: The ridge term of each column, $\gamma (1 - \vartheta)$ or zero at the intercept column.
  • pen: Whether each column is penalised.

Returns

  • z'::Option{VecNum}: The optimum, or nothing.

Related

source
PortfolioOptimisers.ElasticNetCounts — Type
struct ElasticNetCounts{__T_df, __T_rss, __T_nobs, __T_rdof}

Holds the effective degrees of freedom, the weighted residual sum of squares, the number of observations and the residual count of a fitted ElasticNet.

This type is an implementation detail and is not intended for direct use. elastic_net_counts makes it, and StatsAPI.fit(::ElasticNet, ::MatNum, ::VecNum) copies its fields into the ElasticNetFit that it returns. ElasticNetFit states each count and its derivation.

Fields

  • df: df: The effective degrees of freedom of the fit, the trace of the smoother on the active set plus the term of sdiv.
  • rss: rss: The weighted residual sum of squares of the fit, under the frequency weights of glm_weights.
  • nobs: nobs: The number of observations, the sum of the frequency weights, or the number of rows without weights.
  • rdof: rdof: The residual count, the number of rows less $\operatorname{tr}(2 \mathbf{S} - \mathbf{S}^\intercal \mathbf{S})$.

Related

source
PortfolioOptimisers.elastic_net_counts — Function
elastic_net_counts(tgt::ElasticNet, sys::NamedTuple) -> ElasticNetCounts

Return the effective degrees of freedom, the weighted residual sum of squares, the number of observations and the residual count of a fitted ElasticNet.

ElasticNetFit states the counts and their derivation. The method reads them off the equations of the active set that elastic_net_polish solves, so it needs the pseudo-inverse of one matrix with one row per active column.

Mathematical definition

\[\begin{align} \mathbf{B} &= \mathbf{M}^{-1} \mathbf{P}_{\mathcal{A}}^\intercal \mathbf{V} \mathbf{P}_{\mathcal{A}}\,,\quad \mathbf{M} = \mathbf{P}_{\mathcal{A}}^\intercal \mathbf{V} \mathbf{P}_{\mathcal{A}} + \gamma (1 - \vartheta) \mathbf{D}_{\mathcal{A}}\,,\\ \operatorname{tr}(\mathbf{H}) &= \operatorname{tr}(\mathbf{B})\,,\quad \operatorname{tr}(\mathbf{S}^\intercal \mathbf{S}) = \operatorname{tr}(\mathbf{B}^{2})\,. \end{align}\]

Where:

  • $\mathbf{B}$: Matrix with one row and one column per active column. It has the nonzero eigenvalues of $\mathbf{H}$.
  • $\mathbf{M}$: Matrix of the equations of the active set.
  • $\mathbf{H}$, $\mathbf{S}$: Smoother of the fit and smoother of the weighted rows, as ElasticNetFit states them.
  • $\mathbf{P}_{\mathcal{A}}$: Columns of the active set.
  • $\mathbf{D}_{\mathcal{A}}$: Diagonal matrix with one for each penalised column of the active set and zero for the intercept column.
  • $\mathbf{V}$: Diagonal matrix of the weights of the observations of the elastic net, the identity when the fit has no observation weights.
  • $\gamma$: Regularisation strength of the elastic net.
  • $\vartheta$: Share of the $L_1$ penalty in the elastic net, the theta field of ElasticNet.

The trace is invariant under a cyclic permutation, so $\operatorname{tr}(\mathbf{H}) = \operatorname{tr}(\mathbf{B})$. $\mathbf{S}$ is symmetric, and $\mathbf{S}^{2}$ has the trace of $\mathbf{B}^{2}$ for the same reason.

Algorithm

  1. Find A, the indices of the nonzero coefficients of sys.z and of the column that sys.pen does not penalise.
  2. Build the matrix of the equations of the active set, and take its pseudo-inverse Mi. For theta < 1 the matrix is positive definite, and Mi is its inverse. At theta = 1 the matrix is singular when the active set holds more columns than there are observations, and B is then the projection onto the row space of the active columns, whose trace is their rank. An LU solve of such a matrix does not always fail, and it returns rounding noise in place of the rank.
  3. Multiply Mi by the weighted Gram matrix of the active set, which gives B. Take tH, the trace of B.
  4. Add the term of tgt.sdiv to tH with strength_divergence, which gives df.
  5. Compute the weighted residual sum of squares rss and the number of observations nobs under sys.w.
  6. Take the residual count rdof, the number of rows less 2 tH - tr(B^2).

Arguments

Returns

Related

source

References

[69]
C. M. Hurvich, J. S. Simonoff and C.-L. Tsai. Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 60, 271–293 (1998).
[70]
H. Zou, T. Hastie and R. Tibshirani. On the ``degrees of freedom'' of the lasso. The Annals of Statistics 35, 2173–2192 (2007), arXiv:0712.0881.
[109]
R. J. Tibshirani and J. Taylor. Degrees of freedom in lasso problems. The Annals of Statistics 40, 1198–1232 (2012), arXiv:1111.0653.
[298]
T. J. Hastie and R. J. Tibshirani. Generalized Additive Models. No. 43 of Monographs on Statistics and Applied Probability (Chapman and Hall, London, 1990).
[299]
T. Hastie, R. Tibshirani and J. Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2 Edition (Springer, New York, 2009).