Elastic net
ElasticNet is a penalised regression target. It fits a least-squares regression with a penalty on the $L_1$ norm and on the squared $L_2$ norm of the coefficients, and every estimator that takes a regression target can fit it. The strength of the penalty comes from a rule: MiddleOfPath reads it off the data at the middle of the regularisation path, and FixedStrength takes it from you. A new rule subtypes AbstractPenaltyStrength and adds a method of penalty_strength.
The fit states its effective degrees of freedom, so StepwiseRegression can score it under :aic, :aicc and :bic. StrengthDivergence adds the term of a strength that reads the data to that count.
Strength rules
PortfolioOptimisers.AbstractPenaltyStrength — Type
abstract type AbstractPenaltyStrength <: AbstractAlgorithmAbstract supertype for the rules that set the regularisation strength of an ElasticNet.
The strength scales the penalty against the weighted squared residuals. MiddleOfPath reads it off the data, as Lai et al. [101] do. FixedStrength takes it from the caller.
Interfaces
To add a strength rule, subtype AbstractPenaltyStrength, keep its parameters in the fields of the struct, and add the methods below. A target with sdiv = nothing does not read penalty_strength_gradient, so a rule that runs under that target alone needs only penalty_strength.
penalty_strength
penalty_strength(s::MyPenaltyStrength, g::VecNum, theta::Real) -> Real: Return the regularisation strength $\gamma$ of the fit.
StatsAPI.fit(::ElasticNet, ::MatNum, ::VecNum) calls it once, before the coordinate sweeps. When every entry of g is zero, the fit returns its start point at every strength, and it does not read the strength. It records the strength in the fit.
Arguments
s: The new strength rule.g: The product of each penalised column with the residual of the start point, $\mathbf{P}_k^\intercal \mathbf{V} \boldsymbol{r}_0$ for $k \in \mathcal{K}$. It is empty when the design has no penalised column.theta: The share of the $L_1$ penalty, thethetafield of the target.
Returns
gamma::Real: The regularisation strength, non-negative.
penalty_strength_gradient
penalty_strength_gradient(s::MyPenaltyStrength, g::VecNum, theta::Real) -> VecNum: Return the gradient of the strength thatpenalty_strengthreturns, with respect tog.
StrengthDivergence reads it, because a strength that reads the response adds a term to the divergence of the fit. A rule that does not read g returns zeros.
Arguments
s: The new strength rule.g: The vector thatpenalty_strengthreads.theta: The share of the $L_1$ penalty.
Returns
dg::VecNum: The partial derivative of $\gamma$ with respect to each entry ofg, one entry per entry ofg.
Related
PortfolioOptimisers.penalty_strength — Function
penalty_strength(s::MiddleOfPath, g::AbstractVector, theta::Real) -> Real
penalty_strength(s::FixedStrength, g::AbstractVector, theta::Real) -> RealReturn the regularisation strength of an ElasticNet under its strength rule.
MiddleOfPath states the middle of the path, and FixedStrength returns its own gamma. The # Interfaces section of AbstractPenaltyStrength states the method that a rule of the caller adds.
Algorithm
The method that Julia selects is the algorithm.
MiddleOfPath: divide the largest absolute entry ofgbytheta, which giveslmax, and returnlmax * sqrt(s.ratio). An emptyggives zero.FixedStrength: returns.gamma, and read neithergnortheta.
Arguments
s: The strength rule.g: The product of each penalised column with the residual of the start point.theta: The share of the $L_1$ penalty.
Returns
gamma::Real: The regularisation strength.
Related
PortfolioOptimisers.penalty_strength_gradient — Function
penalty_strength_gradient(s::MiddleOfPath, g::AbstractVector, theta::Real) -> VecNum
penalty_strength_gradient(s::FixedStrength, g::AbstractVector, theta::Real) -> VecNumReturn the gradient of the regularisation strength of an ElasticNet with respect to the vector that penalty_strength reads.
StrengthDivergence chains it to the gradient with respect to the response. The # Interfaces section of AbstractPenaltyStrength states the method that a rule of the caller adds.
Mathematical definition
\[\begin{align} \frac{\partial \gamma}{\partial g_{j}} &= \frac{\sqrt{\rho}}{\vartheta} \operatorname{sign}(g_{k^{*}})\, \mathbb{1}[j = k^{*}]\,,\quad k^{*} = \underset{k}{\arg\max} \lvert g_{k} \rvert\,. \end{align}\]
Where:
- $\gamma$: Regularisation strength of the elastic net.
- $g_{k}$: Entry $k$ of
g, the product of penalised column $k$ with the residual of the start point. - $\rho$: Floor of the path as a fraction of $\gamma_{\max}$, the
ratiofield ofMiddleOfPath. - $\vartheta$: Share of the $L_1$ penalty in the elastic net, the
thetafield ofElasticNet.
The form is the derivative of $\gamma = \sqrt{\rho} \max_{k} \lvert g_{k} \rvert / \vartheta$, at a point where one entry holds the largest absolute value.
Algorithm
The method that Julia selects is the algorithm.
MiddleOfPath: return the emptygwhengis empty. Otherwise find the indexkof the largest absolute entry ofg, and return a vector of zeros withsign(g[k]) * sqrt(s.ratio) / thetaatk.FixedStrength: return zeros, because the strength does not readg.
Arguments
s: The strength rule.g: The product of each penalised column with the residual of the start point.theta: The share of the $L_1$ penalty.
Returns
dg::VecNum: The gradient, one entry per entry ofg.
Related
PortfolioOptimisers.MiddleOfPath — Type
struct MiddleOfPath{__T_ratio} <: AbstractPenaltyStrengthSets the regularisation strength of an ElasticNet at the geometric middle of its regularisation path.
Lai et al. [101] take "the middle one in the regularization path" and state no grid. Section 2.5 of [108] runs the path over 100 strengths on a log scale, from the smallest strength at which every penalised coefficient is zero down to ratio times that strength. Such a grid has no single middle point. The rule takes the geometric middle of the path, which lies between the 50th and the 51st point of that grid at the default ratio = 1e-3.
Mathematical definition
\[\begin{align} \gamma_{\max} &= \max_{k \in \mathcal{K}} \frac{\lvert \mathbf{P}_k^\intercal \mathbf{V} \boldsymbol{r}_0 \rvert}{\vartheta}\,,\\ \gamma &= \gamma_{\max} \sqrt{\rho}\,. \end{align}\]
Where:
- $\gamma_{\max}$: Smallest strength at which every penalised coefficient is zero.
- $\mathcal{K}$: Penalised columns of the elastic net, every column of $\mathbf{P}$ but the intercept column that an estimator names.
- $\mathbf{P}$: Regressors of the elastic net, one row per observation and one column per regressor. Column $k$ is $\mathbf{P}_k$.
- $\mathbf{V}$: Diagonal matrix of the weights of the observations of the elastic net, the identity when the fit has no observation weights.
- $\boldsymbol{r}_0$: Residual of the start point, the response less the fit of the intercept column alone, or the response when the design has no intercept column.
- $\vartheta$: Share of the $L_1$ penalty in the elastic net, the
thetafield ofElasticNet. - $\gamma$: Regularisation strength of the elastic net.
- $\rho$: Floor of the path as a fraction of $\gamma_{\max}$, the
ratiofield.
Every penalised coefficient is zero at and above $\gamma_{\max}$. The path runs from $\gamma_{\max}$ down to $\rho\, \gamma_{\max}$ on a log scale, so its geometric middle is $\gamma_{\max} \sqrt{\rho}$. A ridge alone, $\vartheta = 0$, has no $\gamma_{\max}$, so the constructor of ElasticNet refuses theta = 0 with this rule.
Fields
ratio: The floor of the path as a fraction of the strength above which every penalised coefficient is zero.
Constructors
MiddleOfPath(; ratio::Real = 1e-3) -> MiddleOfPathKeywords correspond to the struct's fields.
Validation
0 < ratio < 1. ADomainErroris thrown otherwise.
Examples
julia> MiddleOfPath()MiddleOfPath ratio ┴ Float64: 0.001Related
References
- [108] J. Friedman, T. Hastie and R. Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software 33, 1–22 (2010). Section 2.5.
- [101] Z.-R. Lai, P.-Y. Yang, X. Wu and L. Fang. A kernel-based trend pattern tracking system for portfolio optimization. Data Mining and Knowledge Discovery 32, 1708–1734 (2018). Section 3.1.2.
PortfolioOptimisers.FixedStrength — Type
struct FixedStrength{__T_gamma} <: AbstractPenaltyStrengthSets the regularisation strength of an ElasticNet to a value that the caller states.
gamma is the $\gamma$ of the objective of ElasticNet. It weighs the penalty against the sum of the weighted squared residuals, and it does not divide that sum by the number of observations. So the same gamma penalises a fit over more observations less, in relation to its residuals. Friedman et al. [108] divide the objective by twice the sum of the weights, so their strength $\lambda$ of Equation (1) is $\gamma / T_{e}$, with $T_{e}$ the sum of the scaled weights, which is the number of observations of an unweighted fit. With a variance estimator in the ve field of the target, gamma acts on the standardised columns. theta = 0 with this rule is a ridge regression.
Fields
gamma: The regularisation strength $\gamma$ of the objective.
Constructors
FixedStrength(; gamma::Real) -> FixedStrengthKeywords correspond to the struct's fields.
Validation
gamma > 0. ADomainErroris thrown otherwise.
Examples
julia> FixedStrength(; gamma = 0.5)FixedStrength gamma ┴ Float64: 0.5Related
References
- [108] J. Friedman, T. Hastie and R. Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software 33, 1–22 (2010). Equation (1).
Strength divergence
PortfolioOptimisers.AbstractStrengthDivergence — Type
abstract type AbstractStrengthDivergence <: AbstractAlgorithmAbstract supertype for the rules that add the term of a strength that reads the response to the effective degrees of freedom of an ElasticNet.
The trace of the smoother on the active set is the effective degrees of freedom of a fit at a fixed strength, Remark (The elastic net) after Equation (28) of [109]. MiddleOfPath reads the strength off the response, so the divergence of its fit has one more term. StrengthDivergence adds it. The sdiv field of ElasticNet holds the rule, and nothing, the default, leaves the term out.
Interfaces
To add a rule, subtype AbstractStrengthDivergence, and add the method below.
strength_divergence
strength_divergence(sdiv::MyStrengthDivergence, tgt::ElasticNet, sys::NamedTuple) -> Real: Return the term that the rule adds to the trace of the smoother.
StatsAPI.fit(::ElasticNet, ::MatNum, ::VecNum) calls it once, after the fit, through elastic_net_counts. The term changes df of ElasticNetFit alone, not the residual count rdof.
Arguments
sdiv: The new rule.tgt: The regression target.sys::NamedTuple: The fit,(; P, Q, y, z, pen, ridge, g, lam, nrm2, w, A, Mi):P,Q: The standardised design and the weighted design, aselastic_net_designreturns them.y: The response.z: The coefficients of the fit on the columns ofP.pen: Whether each column is penalised.ridge: The ridge term of each column.g,lam: The vector thatpenalty_strengthread, and the strength that it returned.nrm2: The weighted squared norm of each column.w: The frequency weights of the rows, ornothing.A: The indices of the active set.Mi: The pseudo-inverse of the matrix of the equations of the active set.
Returns
t::Real: The term.
Related
References
- [109] R. J. Tibshirani and J. Taylor. Degrees of freedom in lasso problems. The Annals of Statistics 40, 1198–1232 (2012). Remark (The elastic net) after Equation (28).
PortfolioOptimisers.StrengthDivergence — Type
struct StrengthDivergence <: AbstractStrengthDivergenceAdds the term of a strength that reads the response to the effective degrees of freedom of an ElasticNet.
The degrees of freedom of a fit $\hat{\boldsymbol{\mu}}$ is the expected divergence $\sum_{i} \partial \hat{\mu}_{i} / \partial y_{i}$, Equations (10) and (11) of [109]. On a fixed active set, the fit at a fixed strength is linear in the response, and its divergence is the trace of the smoother. A strength that reads the response adds the chain term below. Under FixedStrength the gradient of the strength is zero, so the rule adds zero.
Mathematical definition
\[\begin{align} \operatorname{div} \hat{\boldsymbol{\mu}} &= \operatorname{tr}(\mathbf{H}) + \nabla_{\boldsymbol{y}} \gamma^{\intercal} \frac{\partial \hat{\boldsymbol{\mu}}}{\partial \gamma}\,,\\ \frac{\partial \hat{\boldsymbol{\mu}}}{\partial \gamma} &= -\mathbf{P}_{\mathcal{A}} \mathbf{M}^{-1} \left((1 - \vartheta) \mathbf{D}_{\mathcal{A}} \boldsymbol{z}_{\mathcal{A}} + \vartheta \boldsymbol{s}\right)\,,\\ \nabla_{\boldsymbol{y}} \gamma &= (\mathbf{I} - \mathbf{H}_{0})^{\intercal} \mathbf{V} \mathbf{P}_{\mathcal{K}} \frac{\partial \gamma}{\partial \boldsymbol{g}}\,. \end{align}\]
Where:
- $\hat{\boldsymbol{\mu}}$: Fitted response, $\mathbf{P} \hat{\boldsymbol{z}}$.
- $\mathbf{H}$: Smoother of the fit on its active set, which
ElasticNetFitstates. - $\mathbf{M}$: Matrix of the equations of the active set, $\mathbf{P}_{\mathcal{A}}^\intercal \mathbf{V} \mathbf{P}_{\mathcal{A}} + \gamma (1 - \vartheta) \mathbf{D}_{\mathcal{A}}$.
- $\mathbf{P}_{\mathcal{A}}$, $\boldsymbol{z}_{\mathcal{A}}$: Columns of the regressors and coefficients of the active set.
- $\mathbf{D}_{\mathcal{A}}$: Diagonal matrix with one for each penalised column of the active set and zero for the intercept column.
- $\boldsymbol{s}$: Signs of the coefficients of the active set, zero for the intercept column.
- $\mathbf{H}_{0}$: Weighted projection onto the intercept column, the zero matrix when the design has none. The start point of
penalty_strengthis the residual $(\mathbf{I} - \mathbf{H}_{0}) \boldsymbol{y}$. - $\mathbf{P}_{\mathcal{K}}$: Penalised columns of the design.
- $\partial \gamma / \partial \boldsymbol{g}$: Gradient of the strength rule,
penalty_strength_gradient. - $\mathbf{P}$: Regressors of the elastic net, one row per observation and one column per regressor. Column $k$ is $\mathbf{P}_k$.
- $\mathbf{V}$: Diagonal matrix of the weights of the observations of the elastic net, the identity when the fit has no observation weights.
- $\gamma$: Regularisation strength of the elastic net.
- $\vartheta$: Share of the $L_1$ penalty in the elastic net, the
thetafield ofElasticNet. - $\mathcal{K}$: Penalised columns of the elastic net, every column of $\mathbf{P}$ but the intercept column that an estimator names.
The second line differentiates the equations of elastic_net_polish in $\gamma$ on the fixed active set. On the design of ElasticNet that states the gap, the count with the term matches the mean finite-difference divergence of the fit.
Examples
julia> StrengthDivergence()StrengthDivergence()Related
References
- [109] R. J. Tibshirani and J. Taylor. Degrees of freedom in lasso problems. The Annals of Statistics 40, 1198–1232 (2012). Equations (10) and (11), and Remark (The elastic net) after Equation (28).
PortfolioOptimisers.strength_divergence — Function
strength_divergence(sdiv::Nothing, tgt::ElasticNet, sys::NamedTuple) -> Real
strength_divergence(sdiv::StrengthDivergence, tgt::ElasticNet, sys::NamedTuple) -> RealReturn the term that a strength rule which reads the response adds to the effective degrees of freedom of an ElasticNet.
StrengthDivergence states the term. The # Interfaces section of AbstractStrengthDivergence states sys and the method that a rule of the caller adds.
Algorithm
The method that Julia selects is the algorithm.
nothing: return zero, so the count is the trace of the smoother.StrengthDivergence: takedg, the gradient of the strength withpenalty_strength_gradient. Multiply the pseudo-inversesys.Miby(1 - theta) z + theta sign(z)on the penalised columns of the active set, zero at the intercept column, and negate it, which gives the derivative of the active coefficients in the strength. Multiply the active columns ofPby it, which givesv, the derivative of the fitted response. With an intercept column, remove fromvits weighted projection onto that column. Return the product ofdgwith the penalised columns ofQandv.
Arguments
sdiv: The rule, ornothing.tgt: The regression target.sys: The fit.
Returns
t::Real: The term, $\nabla_{\boldsymbol{y}} \gamma^{\intercal} \partial \hat{\boldsymbol{\mu}} / \partial \gamma$.
Related
Target
PortfolioOptimisers.ElasticNet — Type
struct ElasticNet{__T_theta, __T_strength, __T_sdiv, __T_ve, __T_iters, __T_tol, __T_w, __T_ic} <: AbstractPenalisedRegressionTargetFits each response by the elastic net, a least-squares fit with a penalty on the $L_1$ norm and on the squared $L_2$ norm of the coefficients.
The objective is Equation (15) of [101], the elastic net of Zou and Hastie [110], under observation weights. It is Equation (3) of [110] with $\lambda_1 = 2 \vartheta \gamma$ and $\lambda_2 = (1 - \vartheta) \gamma$, so $\vartheta$ is the mixing weight of Equation (2) of [108], not the ratio $\lambda_2 / (\lambda_1 + \lambda_2)$ of [110]. Equation (10) of [108] is the coordinate update. strength sets $\gamma$: MiddleOfPath, the default, reads it off the data as Lai et al. [101] do, and FixedStrength takes it from the caller. theta = 0.99 is the value of [101]. It is almost the lasso, and it keeps the objective strictly convex.
The target adds no intercept and does not standardise the columns by default, as in the model of [101]. The glmnet package does both by default. An estimator that adds a column of ones names it with InterceptColumn through factory, and the fit then leaves that coefficient out of the penalty. A variance estimator in ve divides each penalised column by its standard deviation before the fit, and the fit scales the coefficients back. A column whose standard deviation is zero has no scale, so it keeps its units. ve reads the observation weights of the target, so the scale and the fit weigh the rows alike.
The observation weights enter as the frequency weights of glm_weights, the weights that LinearModel passes to GLM. Weights that are not StatsBase.FrequencyWeights then sum to their Kish effective sample size. Uniform weights give the unweighted fit at the same strength. The scale of the weights moves the fit under FixedStrength alone, because MiddleOfPath scales with them.
When the columns are almost collinear, for example the levels of consecutive periods in KernelTrendPattern, the coordinate sweeps converge slowly. Each sweep therefore ends with an exact solve on the sign pattern that it leaves, see elastic_net_polish. The sweeps must still find the sign pattern of the optimum. On the 4 × 5 window of the test fixture of KernelTrendPattern they find it after between 2000 and 5000 sweeps, and a cap of 1000 stops more than 0.5 from the optimum in one coefficient. When the sweeps stop at tol or at iters before that, the result is the last sweep, which is not the optimum.
The penalty depends on the units of each column, so the fit is not invariant to the basis of the design, and is_basis_invariant returns false. StepwiseRegression accepts the target under :aic, :aicc and :bic alone, because these criteria read the effective degrees of freedom that ElasticNetFit states.
The effective degrees of freedom is the trace of the smoother on the active set of the fit. That trace is exact at a fixed strength. MiddleOfPath reads the strength off the response, so the divergence of its fit has one more term, and sdiv gates it. With sdiv = nothing, the default, the count leaves the term out. The gap was measured on 2000 draws of a design of 60 observations, an intercept and 5 factors, with the coefficients [0.5, 1, -0.5, 0, 0, 0.25] and a unit noise. There the trace was above the mean finite-difference divergence of the fit by 0.06 at theta = 0.5 and by 0.03 at theta = 0.99, with a standard error of 0.014. One factor costs about one degree of freedom, so the gap did not change a comparison of two models. StrengthDivergence adds the term. Under FixedStrength the term is zero, so StrengthDivergence() adds zero.
Mathematical definition
\[\begin{align} \mathbf{P} &= \mathbf{X} \mathbf{S}^{-1}\,,\\ \hat{\boldsymbol{z}} &= \underset{\boldsymbol{z}}{\arg\min} \left(\boldsymbol{y} - \mathbf{P} \boldsymbol{z}\right)^\intercal \mathbf{V} \left(\boldsymbol{y} - \mathbf{P} \boldsymbol{z}\right) + \gamma \sum_{k \in \mathcal{K}} \left( 2 \vartheta \lvert z_k \rvert + (1 - \vartheta) z_k^2 \right)\,,\\ \hat{\boldsymbol{b}} &= \mathbf{S}^{-1} \hat{\boldsymbol{z}}\,. \end{align}\]
Where:
- $\mathbf{X}$: Design matrix of the caller, one row per observation and one column per regressor.
- $\mathbf{S}$: Diagonal matrix of the standard deviations of the columns of $\mathbf{X}$ under
ve, with one in place of the intercept column and of a column of zero standard deviation. It is the identity whenveisnothing. - $\mathbf{P}$: Regressors of the elastic net, one row per observation and one column per regressor. Column $k$ is $\mathbf{P}_k$.
- $\boldsymbol{y}$: Response of the elastic net, one entry per observation.
- $\boldsymbol{z}$: Coefficients of the elastic net, one per column of $\mathbf{P}$. Coefficient $k$ is $z_k$.
- $\mathbf{V}$: Diagonal matrix of the weights of the observations of the elastic net, the identity when the fit has no observation weights. Its diagonal holds the frequency weights of
glm_weights. - $\mathcal{K}$: Penalised columns of the elastic net, every column of $\mathbf{P}$ but the intercept column that an estimator names.
- $\gamma$: Regularisation strength of the elastic net.
- $\vartheta$: Share of the $L_1$ penalty in the elastic net, the
thetafield ofElasticNet. - $\hat{\boldsymbol{b}}$: Coefficients of the fit on the columns of $\mathbf{X}$, which
StatsAPI.coefreturns.
Fields
theta: The share of the $L_1$ penalty,1for the lasso and0for the ridge.
strength: The rule that sets the regularisation strength.
sdiv: The rule that adds the term of a strength that reads the response to the effective degrees of freedom of the fit, ornothingto leave the term out.
ve: Variance estimator. It standardises each penalised column, ornothingto fit the columns as they are.
iters: Maximum number of coordinate-descent sweeps. Each sweep ends with the exact solve on its sign pattern.
tol: Tolerance on the largest change of a coefficient over one sweep. The sweeps stop below it when no exact solve has given the optimum.
w: Observation weights vectorobservations × 1.
ic: The index of the column of ones that the estimator names withInterceptColumn, ornothing. The fit does not penalise or standardise that column.
Constructors
ElasticNet(; theta::Real = 0.99, strength::AbstractPenaltyStrength = MiddleOfPath(), sdiv::Option{<:AbstractStrengthDivergence} = nothing, ve::Option{<:AbstractVarianceEstimator} = nothing, iters::Integer = 10_000, tol::Real = 1e-10, w::Option{<:ObsWeights} = nothing, ic::Option{<:Integer} = nothing) -> ElasticNetKeywords correspond to the struct's fields. The constructor passes w to ve through factory.
Validation
0 <= theta <= 1. ADomainErroris thrown otherwise.theta > 0whenstrengthis aMiddleOfPath. ADomainErroris thrown otherwise, because a ridge alone has no strength above which every coefficient is zero.iters >= 1. ADomainErroris thrown otherwise.tol > 0. ADomainErroris thrown otherwise.- If
wis a vector,!isempty(w). AnIsEmptyErroris thrown otherwise. - If
icis notnothing,ic >= 1. ADomainErroris thrown otherwise.
Examples
julia> ElasticNet()ElasticNet theta ┼ Float64: 0.99 strength ┼ MiddleOfPath │ ratio ┴ Float64: 0.001 sdiv ┼ nothing ve ┼ nothing iters ┼ Int64: 10000 tol ┼ Float64: 1.0e-10 w ┼ nothing ic ┴ nothingjulia> X = [1.0 0.5; 1.0 -0.2; 1.0 0.9; 1.0 0.1];julia> y = [1.2, 0.4, 1.9, 0.6];julia> tgt = factory(ElasticNet(; strength = FixedStrength(; gamma = 1e6)), InterceptColumn(1));julia> m = PortfolioOptimisers.StatsAPI.fit(tgt, X, y);julia> PortfolioOptimisers.StatsAPI.coef(m) ≈ [sum(y) / 4, 0.0]trueRelated
AbstractPenalisedRegressionTargetAbstractPenaltyStrengthMiddleOfPathFixedStrengthInterceptColumnLinearModelKernelTrendPatternStrengthDivergenceElasticNetFitStepwiseRegressionStatsAPI.fit(::ElasticNet, ::MatNum, ::VecNum)
References
- [108] J. Friedman, T. Hastie and R. Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software 33, 1–22 (2010). Section 2.5, Equations (1), (2) and (10).
- [110] H. Zou and T. Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 67, 301–320 (2005). Equation (3).
- [101] Z.-R. Lai, P.-Y. Yang, X. Wu and L. Fang. A kernel-based trend pattern tracking system for portfolio optimization. Data Mining and Knowledge Discovery 32, 1708–1734 (2018). Equation (15).
- [109] R. J. Tibshirani and J. Taylor. Degrees of freedom in lasso problems. The Annals of Statistics 40, 1198–1232 (2012). Remark (The elastic net) after Equation (28).
PortfolioOptimisers.factory — Method
factory(tgt::ElasticNet, w::ObsWeights) -> ElasticNetReturn a new ElasticNet whose fit weighs the observations by w.
Algorithm
- Rebuild
tgtwithwin place oftgt.wand every other field unchanged, throughrebuild_struct. The keyword constructor that it calls passeswtotgt.vethroughfactory, so the standardisation reads the same weights.
Arguments
tgt: The regression target.w: Observation weights vectorobservations × 1.
Returns
tgt::ElasticNet: A new regression target that holds the weights.
Related
PortfolioOptimisers.factory — Method
factory(tgt::ElasticNet, ic::InterceptColumn) -> ElasticNetReturn a new ElasticNet whose fit leaves the column of ones that an estimator names out of the penalty.
Algorithm
- Rebuild
tgtwithic.kin place oftgt.icand every other field unchanged, throughrebuild_struct.
Arguments
tgt: The regression target.ic: The intercept column.
Returns
tgt::ElasticNet: A new regression target that does not penalise or standardise columnic.k.
Related
PortfolioOptimisers.regression_target_weights — Method
regression_target_weights(tgt::ElasticNet)Return the observation weights that an ElasticNet carries into its fit, or nothing.
The method returns tgt.w as it is, so it does not resolve a DynamicAbstractWeights against a design.
Arguments
tgt: The regression target.
Returns
w::Option{ObsWeights}:tgt.w.
Related
StatsAPI.fit — Method
StatsAPI.fit(tgt::ElasticNet, X::MatNum, y::VecNum) -> ElasticNetFitFit an ElasticNet to the design X and the response y by cyclic coordinate descent.
ElasticNet states the objective. The fit solves it at one strength, which the strength rule gives. A design with few columns does not need the rest of the path as warm starts.
Algorithm
- Build the standardised design
P, the weighted designQ, the scalesand the weightsvwithelastic_net_design. - Mark the penalised columns in
pen: every column buttgt.ic. Computenrm2, the weighted squared norm of each column. - Start from the coefficients
zat zero and the residualrequal toy. With an intercept column, fit that column alone first, and take its residual asr. - Compute
g, the product of each penalised column ofQwithr, the strengthlamwithpenalty_strength, and the ridge termridgeof each column, zero at the intercept column. - When every entry of
gis zero, the start point is the optimum. Otherwise compute the thresholdaof each column, zero at the intercept column, and runelastic_net_sweepsfrom the start point. - Compute the counts of the fit with
elastic_net_counts. - Divide the coefficients by
s, and return them withlamand the counts in anElasticNetFit.
Arguments
tgt: The regression target.X: The design matrixobservations × regressors.y: The response vector.
Returns
m::ElasticNetFit: The fitted model.
Related
StatsAPI.predict — Method
StatsAPI.predict(m::ElasticNetFit, X::MatNum) -> VecNumReturn the fitted response of a fitted ElasticNet on a design.
Arguments
m: The fitted model.X: The design matrix,observations × regressors, with the columns of the design of the fit.
Returns
yhat::VecNum:X * m.b, one entry per row ofX.
Related
PortfolioOptimisers.residual_count — Method
residual_count(m::ElasticNetFit, X::MatNum) -> RealReturn the residual count of a fitted ElasticNet, m.rdof.
ElasticNetFit states the count and its derivation. The design does not change it, because the fit already holds the count of its own design.
Arguments
m: The fitted model.X: The design of the fit, which the method does not read.
Returns
nu::Real:m.rdof.
Related
PortfolioOptimisers.obs_weights_view — Method
obs_weights_view(tgt::ElasticNet, i) -> ElasticNetReturn an ElasticNet whose observation weights, and the weights of its variance estimator, are cut to the observations i.
The target is not @propagatable, so it needs this method of obs_weights_view. A cut of the observations of a fit, such as the drop of a Shared Series gap, then cuts the weights of the regression too, see shared_series_estimator.
Algorithm
- Index
tgt.wwithobs_weights_index. - Cut the variance estimator
tgt.vewithobs_weights_view. - Rebuild the target with the two cut fields through
rebuild_struct. The other fields keep their values.
Arguments
tgt: The regression target.i: The observations to keep: a mask of the full sample, or their positions.
Validation
- The rules of
obs_weights_index.
Returns
tgt::ElasticNet: The target at the kept observations.
Related
References
- [101]
- Z.-R. Lai, P.-Y. Yang, X. Wu and L. Fang. A kernel-based trend pattern tracking system for portfolio optimization. Data Mining and Knowledge Discovery 32, 1708–1734 (2018).
- [108]
- J. Friedman, T. Hastie and R. Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software 33, 1–22 (2010).
- [109]
- R. J. Tibshirani and J. Taylor. Degrees of freedom in lasso problems. The Annals of Statistics 40, 1198–1232 (2012), arXiv:1111.0653.
- [110]
- H. Zou and T. Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 67, 301–320 (2005).