Base moments

Abstract moment types and fallbacks

Some optimisations and constraints make use of summary statistics. These types and functions form the base for moment estimation in PortfolioOptimisers.jl.

They also provide generic fallbacks for the various functionality in the library.

PortfolioOptimisers.AbstractCovarianceEstimatorType
abstract type AbstractCovarianceEstimator <: CovarianceEstimator

Abstract supertype for all covariance estimator types.

All concrete and/or abstract types that implement covariance estimation should be subtypes of AbstractCovarianceEstimator.

Interfaces

In order to implement a new covariance estimator which will work seamlessly with the library, subtype AbstractCovarianceEstimator with all necessary parameters—including observation weights—as part of the struct, and implement the following methods:

Covariance and correlation

  • Statistics.cov(ce::AbstractCovarianceEstimator, X::MatNum; kwargs...) -> MatNum: Covariance matrix estimation.
  • Statistics.cor(ce::AbstractCovarianceEstimator, X::MatNum; kwargs...) -> MatNum: Correlation matrix estimation.

Arguments

  • ce: Covariance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • kwargs...: Additional keyword arguments passed to the underlying covariance estimator.

Returns

  • sigrho::MatNum: Covariance/correlation matrix assets x assets.

Factory

  • PortfolioOptimisers.factory(ce::AbstractCovarianceEstimator, w::PortfolioOptimisers.ObsWeights) -> AbstractCovarianceEstimator: Factory method for creating instances of the estimator with new observation weights.

Arguments

  • ce: Covariance estimator.
  • w: Observation weights vector observations × 1.

Returns

  • ce: New covariance estimator of the same type as the argument, with the new weights applied.

Examples

We can create a dummy covariance estimator as follows:

julia> struct MyCovarianceEstimator{T1} <: PortfolioOptimisers.AbstractCovarianceEstimator           w::T1           function MyCovarianceEstimator(w::PortfolioOptimisers.Option{<:PortfolioOptimisers.ObsWeights})               PortfolioOptimisers.assert_nonempty_nonneg_finite_val(w, :w)               return new{typeof(w)}(w)           end       endjulia> function MyCovarianceEstimator(;                                      w::PortfolioOptimisers.Option{<:PortfolioOptimisers.ObsWeights} = nothing)           return MyCovarianceEstimator(w)       endMyCovarianceEstimatorjulia> function PortfolioOptimisers.factory(::MyCovarianceEstimator,                                            w::PortfolioOptimisers.ObsWeights)           return MyCovarianceEstimator(; w = w)       endjulia> function Statistics.cov(est::MyCovarianceEstimator, X::PortfolioOptimisers.MatNum;                               dims::Int = 1, kwargs...)           X = PortfolioOptimisers.dims_oriented(dims, X)           w = ifelse(isnothing(est.w), StatsBase.fweights(fill(1.0, size(X, 1))), est.w)           X = X .* w           sigma = X * X'           return sigma       endjulia> function Statistics.cor(est::MyCovarianceEstimator, X::PortfolioOptimisers.MatNum;                               dims::Int = 1, kwargs...)           X = PortfolioOptimisers.dims_oriented(dims, X)           w = isnothing(est.w) ? StatsBase.fweights(fill(1.0, size(X, 1))) : est.w           X = X .* w           sigma = X * X'           d = LinearAlgebra.diag(sigma)           StatsBase.cov2cor!(sigma, sqrt.(d))           return sigma       endjulia> cov(MyCovarianceEstimator(), [1.0 2.0; 0.3 0.7; 0.5 1.1])3×3 Matrix{Float64}: 5.0  1.7   2.7 1.7  0.58  0.92 2.7  0.92  1.46julia> cor(MyCovarianceEstimator(), [1.0 2.0; 0.3 0.7; 0.5 1.1])3×3 Matrix{Float64}: 1.0       0.998274  0.999315 0.998274  1.0       0.999764 0.999315  0.999764  1.0julia> PortfolioOptimisers.factory(MyCovarianceEstimator(), StatsBase.Weights([1, 2, 3]))MyCovarianceEstimator  w ┴ StatsBase.Weights{Int64, Int64, Vector{Int64}}: [1, 2, 3]

Related

source
PortfolioOptimisers.AbstractVarianceEstimatorType
abstract type AbstractVarianceEstimator <: AbstractCovarianceEstimator

Abstract supertype for all variance estimator types.

All concrete and/or abstract types that implement variance estimation should be subtypes of AbstractVarianceEstimator.

Interfaces

In order to implement a new covariance estimator which will work seamlessly with the library, subtype AbstractVarianceEstimator with all necessary parameters—including observation weights—as part of the struct, and implement the following methods:

Variance and standard deviation

  • Statistics.var(ve::AbstractVarianceEstimator, X::MatNum; kwargs...) -> ArrNum: Variance estimation.
  • Statistics.std(ve::AbstractVarianceEstimator, X::MatNum; kwargs...) -> ArrNum: Standard deviation estimation.
  • Statistics.var(ve::AbstractVarianceEstimator, X::VecNum; kwargs...) -> Number: Variance estimation.
  • Statistics.std(ve::AbstractVarianceEstimator, X::VecNum; kwargs...) -> Number: Standard deviation estimation.

Arguments

  • ve: Variance estimator.

  • X

    • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
    • X: Data vector observations × 1.
  • kwargs...: Additional keyword arguments passed to the mean estimator.

Returns

  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.

    • res::ArrNum: Variance or standard deviation vector of X, reshaped to be consistent with the dimension along which the value is computed.
  • X: Data vector observations × 1.

    • res::Number: Variance or standard deviation X

Covariance and correlation

Statistics.cov(ve::AbstractVarianceEstimator, X::MatNum; kwargs...) and Statistics.cor(ve::AbstractVarianceEstimator, X::MatNum; kwargs...) always throw a MethodError. A variance estimator resolves one marginal variance per asset and holds no cross-asset structure, so it cannot answer either verb.

Factory

  • PortfolioOptimisers.factory(ve::AbstractVarianceEstimator, w::PortfolioOptimisers.ObsWeights) -> AbstractVarianceEstimator: Factory method for creating instances of the estimator with new observation weights.

Arguments

  • ve: Variance estimator.
  • w: Observation weights vector observations × 1.

Returns

  • ve: New variance estimator of the same type as the argument, with the new weights applied.

Examples

We can create a dummy variance estimator as follows:

julia> struct MyVarianceEstimator{T1} <: PortfolioOptimisers.AbstractVarianceEstimator           w::T1           function MyVarianceEstimator(w::PortfolioOptimisers.Option{<:PortfolioOptimisers.ObsWeights})               PortfolioOptimisers.assert_nonempty_nonneg_finite_val(w, :w)               return new{typeof(w)}(w)           end       endjulia> function MyVarianceEstimator(;                                    w::PortfolioOptimisers.Option{<:PortfolioOptimisers.ObsWeights} = nothing)           return MyVarianceEstimator(w)       endMyVarianceEstimatorjulia> function PortfolioOptimisers.factory(::MyVarianceEstimator,                                            w::PortfolioOptimisers.ObsWeights)           return MyVarianceEstimator(; w = w)       endjulia> function Statistics.var(est::MyVarianceEstimator, X::PortfolioOptimisers.MatNum;                               dims::Int = 1, kwargs...)           X = PortfolioOptimisers.dims_oriented(dims, X)           w = isnothing(est.w) ? StatsBase.fweights(fill(1.0, size(X, 1))) : est.w           X = X .* w           sigma = LinearAlgebra.diag(X * X')           return isone(dims) ? reshape(sigma, 1, :) : reshape(sigma, :, 2)       endjulia> function Statistics.std(est::MyVarianceEstimator, X::PortfolioOptimisers.MatNum;                               dims::Int = 1, kwargs...)           X = PortfolioOptimisers.dims_oriented(dims, X)           w = isnothing(est.w) ? StatsBase.fweights(fill(1.0, size(X, 1))) : est.w           X = X .* w           sigma = sqrt.(LinearAlgebra.diag(X * X'))           return isone(dims) ? reshape(sigma, 1, :) : reshape(sigma, :, 1)       endjulia> function Statistics.var(est::MyVarianceEstimator, X::PortfolioOptimisers.VecNum; kwargs...)           w = isnothing(est.w) ? StatsBase.fweights(fill(1.0, size(X, 1))) : est.w           X = X .* w           return mean(LinearAlgebra.diag(X' * X))       endjulia> function Statistics.std(est::MyVarianceEstimator, X::PortfolioOptimisers.VecNum; kwargs...)           w = isnothing(est.w) ? StatsBase.fweights(fill(1.0, size(X, 1))) : est.w           X = X .* w           return sqrt(mean(LinearAlgebra.diag(X' * X)))       endjulia> var(MyVarianceEstimator(), [1.0 2.0; 0.3 0.7; 0.5 1.1])1×3 Matrix{Float64}: 5.0  0.58  1.46julia> std(MyVarianceEstimator(), [1.0 2.0; 0.3 0.7; 0.5 1.1])1×3 Matrix{Float64}: 2.23607  0.761577  1.2083julia> PortfolioOptimisers.factory(MyVarianceEstimator(), StatsBase.Weights([1, 2, 3]))MyVarianceEstimator  w ┴ StatsBase.Weights{Int64, Int64, Vector{Int64}}: [1, 2, 3]

Related

source
PortfolioOptimisers.AbstractExpectedReturnsEstimatorType
abstract type AbstractExpectedReturnsEstimator <: AbstractEstimator

Abstract supertype for all expected returns estimator types.

All concrete and/or abstract types that implement expected returns estimation should be subtypes of AbstractExpectedReturnsEstimator.

Interfaces

In order to implement a new expected returns estimator which will work seamlessly with the library, subtype AbstractExpectedReturnsEstimator with all necessary parameters—including observation weights—as part of the struct, and implement the following methods:

Expected returns

  • Statistics.mean(me::AbstractExpectedReturnsEstimator, X::MatNum; kwargs...) -> ArrNum: Expected returns estimation.

Arguments

  • me: Expected returns estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • kwargs...: Additional keyword arguments passed to the mean estimator.

Returns

  • mu::ArrNum: Expected returns vector assets x 1 if the dims keyword does not exist or dims = 2, 1 x assets if dims = 1.

Factory

  • PortfolioOptimisers.factory(me::AbstractExpectedReturnsEstimator, w::PortfolioOptimisers.ObsWeights) -> AbstractExpectedReturnsEstimator: Factory method for creating instances of the estimator with new observation weights.

Arguments

  • me: Expected returns estimator.
  • w: Observation weights vector observations × 1.

Returns

  • me: New expected returns estimator of the same type as the argument, with the appropriate weights applied.

Examples

julia> struct MyExpectedReturnsEstimator{T1} <:              PortfolioOptimisers.AbstractExpectedReturnsEstimator           w::T1           function MyExpectedReturnsEstimator(w::PortfolioOptimisers.Option{<:PortfolioOptimisers.ObsWeights})               PortfolioOptimisers.assert_nonempty_nonneg_finite_val(w, :w)               return new{typeof(w)}(w)           end       endjulia> function MyExpectedReturnsEstimator(;                                           w::PortfolioOptimisers.Option{<:PortfolioOptimisers.ObsWeights} = nothing)           return MyExpectedReturnsEstimator(w)       endMyExpectedReturnsEstimatorjulia> function PortfolioOptimisers.factory(::MyExpectedReturnsEstimator,                                            w::PortfolioOptimisers.ObsWeights)           return MyExpectedReturnsEstimator(; w = w)       endjulia> function Statistics.mean(est::MyExpectedReturnsEstimator, X::PortfolioOptimisers.MatNum;                                dims::Int = 1, kwargs...)           X = PortfolioOptimisers.dims_oriented(dims, X)           w = isnothing(est.w) ? fill(one(eltype(X)), size(X, 1)) : est.w           X = X .* w           mu = sum(X; dims = 1) / sum(w)           return isone(dims) ? reshape(mu, 1, :) : reshape(mu, :, 1)       endjulia> mean(MyExpectedReturnsEstimator(), [1.0 2.0; 0.3 0.7; 0.5 1.1]; dims = 2)3×1 Matrix{Float64}: 1.5 0.5 0.8julia> PortfolioOptimisers.factory(MyExpectedReturnsEstimator(), StatsBase.Weights([1, 2, 3]))MyExpectedReturnsEstimator  w ┴ StatsBase.Weights{Int64, Int64, Vector{Int64}}: [1, 2, 3]

Related

source
PortfolioOptimisers.factoryMethod
factory(a::Union{Nothing, <:AbstractEstimator, <:AbstractAlgorithm,
                 <:AbstractResult}, args...; kwargs...) -> a
factory(a::AbstractVector{<:Union{Nothing, <:AbstractEstimator, <:AbstractAlgorithm,
                                  <:AbstractResult}}, args...; kwargs...) -> Vector

No-op factory function for constructing objects with a uniform interface.

Defining methods which dispatch on the first argument allows for a consistent factory interface across different types.

factory and port_opt_view are the two propagation mechanisms in this library. They are duals: factory threads runtime values (prior moments, observation weights, previous portfolio weights) down through a composed struct tree; port_opt_view threads an index selection (a subset of assets or observations) down through the same tree.

The vector method is the one forwarding contract for every vector-valued propagation field: it applies factory to each element and forwards args... and kwargs... unchanged, so a family that admits a vector of estimators, algorithms, or results needs no method of its own. A family that needs more than the forward, such as a concrete element type (concrete_typed_array_if_abstract), defines its own more specific method.

Algorithm

The scalar method:

  1. Return a unchanged, and drop args... and kwargs.... This method is the leaf of the recursion, and it is what makes an untagged type safe to call the verb on.

The vector method:

  1. For each element ai of a, call factory on ai, and forward args... and kwargs... unchanged.
  2. Collect the results into a new vector, in the order of a, and return it.

A @propagatable struct with at least one @fprop- or @wprop-tagged field carries a generated method that dominates the scalar method. That method rebuilds the struct with its keyword constructor, sending each @fprop field through factory_child and each @wprop field through _wprop.

Arguments

  • a: Indicates no object should be constructed, or a vector whose elements are rebuilt one by one.
  • args...: Arbitrary positional arguments (ignored by the scalar method, forwarded by the vector method).
  • kwargs...: Arbitrary keyword arguments (ignored by the scalar method, forwarded by the vector method).

Returns

  • a: The input unchanged.
  • v::Vector: The element-wise rebuilds, for the vector method.

Examples

julia> factory(nothing, 1, 2; x = 3)julia> factory(MeanValue())MeanValue  w ┴ nothing

Related

source
PortfolioOptimisers.factoryMethod
factory(a::Union{Nothing, <:AbstractEstimator, <:AbstractAlgorithm,
                 <:AbstractResult}, args...; kwargs...) -> a
factory(a::AbstractVector{<:Union{Nothing, <:AbstractEstimator, <:AbstractAlgorithm,
                                  <:AbstractResult}}, args...; kwargs...) -> Vector

No-op factory function for constructing objects with a uniform interface.

Defining methods which dispatch on the first argument allows for a consistent factory interface across different types.

factory and port_opt_view are the two propagation mechanisms in this library. They are duals: factory threads runtime values (prior moments, observation weights, previous portfolio weights) down through a composed struct tree; port_opt_view threads an index selection (a subset of assets or observations) down through the same tree.

The vector method is the one forwarding contract for every vector-valued propagation field: it applies factory to each element and forwards args... and kwargs... unchanged, so a family that admits a vector of estimators, algorithms, or results needs no method of its own. A family that needs more than the forward, such as a concrete element type (concrete_typed_array_if_abstract), defines its own more specific method.

Algorithm

The scalar method:

  1. Return a unchanged, and drop args... and kwargs.... This method is the leaf of the recursion, and it is what makes an untagged type safe to call the verb on.

The vector method:

  1. For each element ai of a, call factory on ai, and forward args... and kwargs... unchanged.
  2. Collect the results into a new vector, in the order of a, and return it.

A @propagatable struct with at least one @fprop- or @wprop-tagged field carries a generated method that dominates the scalar method. That method rebuilds the struct with its keyword constructor, sending each @fprop field through factory_child and each @wprop field through _wprop.

Arguments

  • a: Indicates no object should be constructed, or a vector whose elements are rebuilt one by one.
  • args...: Arbitrary positional arguments (ignored by the scalar method, forwarded by the vector method).
  • kwargs...: Arbitrary keyword arguments (ignored by the scalar method, forwarded by the vector method).

Returns

  • a: The input unchanged.
  • v::Vector: The element-wise rebuilds, for the vector method.

Examples

julia> factory(nothing, 1, 2; x = 3)julia> factory(MeanValue())MeanValue  w ┴ nothing

Related

source
Statistics.covMethod
Statistics.cov(ce::AbstractCovarianceEstimator, X::MatNum; dims::Int = 1, kwargs...)

Generic covariance fallback assembling the covariance matrix from the estimator's correlation matrix and the marginal standard deviations of its variance estimator ce.ve.

This fallback lets a correlation-style covariance estimator define only its Statistics.cor method; the covariance is then recovered by rescaling the correlation matrix with the standard deviations obtained from std(ce.ve, X). Estimators whose covariance is not a plain rescaling of their correlation (for example ImpliedVolatility) override this method.

Mathematical definition

\[\begin{align} \hat{\mathbf{\Sigma}}_{ij} &= \hat{\rho}_{ij}\,\hat{\sigma}_i\,\hat{\sigma}_j\,. \end{align}\]

Where:

  • $\hat{\mathbf{\Sigma}}$: Estimated covariance matrix.
  • $\hat{\rho}_{ij}$: Correlation between assets $i$ and $j$.
  • $\hat{\sigma}_i$: Standard deviation of asset $i$.

Algorithm

  1. Estimate the correlation matrix with Statistics.cor(ce, X).
  2. Estimate the marginal standard deviations with Statistics.std(ce.ve, X).
  3. Rescale the correlation matrix into a covariance matrix with StatsBase.cor2cov!. The call is in place, so it consumes the matrix that step 1 returned.

Arguments

  • ce: Covariance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • dims: Dimension along which to perform the computation.
  • kwargs...: Additional keyword arguments passed to cor and to the variance estimator.

Returns

  • sigma::MatNum: Covariance matrix assets x assets.

Related

source
Statistics.covMethod
cov(
    ce::AbstractCovarianceEstimator,
    state::SampleBufferState
) -> Any

Reads a covariance matrix out of a SampleBufferState, by running the batch verb over the observations the buffer holds.

The buffer read-out arm of Statistics.cov. An estimator wrapped in Online carries a buffer rather than a family state, so it takes no exact fold, and its estimate is whatever a batch fit over the rows the buffer holds gives — every observation folded so far when the buffer is uncapped, and the last max_history of them when it is capped. It runs no assert_partial_fittable check of its own: the batch verb answers configurations an incremental fold cannot, and a wrapper is how a caller reaches them. The buffer also holds the per-observation masks it was folded with, so the batch verb is given the mask that explains the rows and a wrapped estimator under a CoveragePolicy answers what an unwrapped one answers.

Arguments

  • ce: Covariance estimator.
  • state: The buffer to read.

Returns

  • sigma::MatNum: Covariance matrix of the observations the buffer holds.

Related

source
Statistics.corMethod
cor(
    ce::AbstractCovarianceEstimator,
    state::SampleBufferState
) -> Any

Reads a correlation matrix out of a SampleBufferState, by running the batch verb over the observations the buffer holds.

The buffer read-out arm of Statistics.cor. An estimator wrapped in Online carries a buffer rather than a family state, so it takes no exact fold, and its estimate is whatever a batch fit over the rows the buffer holds gives — every observation folded so far when the buffer is uncapped, and the last max_history of them when it is capped. It runs no assert_partial_fittable check of its own: the batch verb answers configurations an incremental fold cannot, and a wrapper is how a caller reaches them. The buffer also holds the per-observation masks it was folded with, so the batch verb is given the mask that explains the rows and a wrapped estimator under a CoveragePolicy answers what an unwrapped one answers.

Arguments

  • ce: Covariance estimator.
  • state: The buffer to read.

Returns

  • rho::MatNum: Correlation matrix of the observations the buffer holds.

Related

source
Statistics.varMethod
var(
    ve::AbstractVarianceEstimator,
    state::SampleBufferState
) -> Any

Reads a variance out of a SampleBufferState, by running the batch verb over the observations the buffer holds.

The buffer read-out arm of Statistics.var. An estimator wrapped in Online carries a buffer rather than a family state, so it takes no exact fold, and its estimate is whatever a batch fit over the rows the buffer holds gives — every observation folded so far when the buffer is uncapped, and the last max_history of them when it is capped. It runs no assert_partial_fittable check of its own: the batch verb answers configurations an incremental fold cannot, and a wrapper is how a caller reaches them. The buffer also holds the per-observation masks it was folded with, so the batch verb is given the mask that explains the rows and a wrapped estimator under a CoveragePolicy answers what an unwrapped one answers.

Arguments

  • ve: Variance estimator.
  • state: The buffer to read.

Returns

  • sigma2::ArrNum: Variance of the observations the buffer holds, one entry per asset.

Related

source
Statistics.stdMethod
std(
    ve::AbstractVarianceEstimator,
    state::SampleBufferState
) -> Any

Reads a standard deviation out of a SampleBufferState, by running the batch verb over the observations the buffer holds.

The buffer read-out arm of Statistics.std. An estimator wrapped in Online carries a buffer rather than a family state, so it takes no exact fold, and its estimate is whatever a batch fit over the rows the buffer holds gives — every observation folded so far when the buffer is uncapped, and the last max_history of them when it is capped. It runs no assert_partial_fittable check of its own: the batch verb answers configurations an incremental fold cannot, and a wrapper is how a caller reaches them. The buffer also holds the per-observation masks it was folded with, so the batch verb is given the mask that explains the rows and a wrapped estimator under a CoveragePolicy answers what an unwrapped one answers.

Arguments

  • ve: Variance estimator.
  • state: The buffer to read.

Returns

  • sigma::ArrNum: Standard deviation of the observations the buffer holds, one entry per asset.

Related

source
Statistics.meanMethod
mean(
    me::AbstractExpectedReturnsEstimator,
    state::SampleBufferState
) -> Any

Reads an expected returns vector out of a SampleBufferState, by running the batch verb over the observations the buffer holds.

The buffer read-out arm of Statistics.mean. An estimator wrapped in Online carries a buffer rather than a family state, so it takes no exact fold, and its estimate is whatever a batch fit over the rows the buffer holds gives — every observation folded so far when the buffer is uncapped, and the last max_history of them when it is capped. It runs no assert_partial_fittable check of its own: the batch verb answers configurations an incremental fold cannot, and a wrapper is how a caller reaches them. The buffer also holds the per-observation masks it was folded with, so the batch verb is given the mask that explains the rows and a wrapped estimator under a CoveragePolicy answers what an unwrapped one answers.

Arguments

  • me: Expected returns estimator.
  • state: The buffer to read.

Returns

  • mu::VecNum: Expected returns of the observations the buffer holds, assets × 1.

Related

source

FullMoment and semi moments

Moments other than the expected return can be estimated using the entire spectrum of deviations (full), or only the deviations below a target (semi/downside). These types allow us to provide such functionality.

PortfolioOptimisers.FullMomentType
struct FullMoment <: AbstractMomentAlgorithm

Keeps every deviation from the target, so the moment is two-sided.

Mathematical definition

\[\begin{align} \boldsymbol{D} &= \boldsymbol{X} - t \end{align}\]

Where:

  • $\boldsymbol{X}$: Data vector observations × 1.
  • $t$: Target value, usually the unweighted (or weighted) expected value $E[\boldsymbol{X}]$.

Constructors

FullMoment() -> FullMoment

Examples

julia> FullMoment()FullMoment()

Related

source
PortfolioOptimisers.SemiMomentType
struct SemiMoment <: AbstractMomentAlgorithm

Clips every deviation above the target to zero, so the moment reads the downside alone.

Mathematical definition

\[\begin{align} \boldsymbol{D} &= \min\left(\boldsymbol{X} - t,\, 0\right) \end{align}\]

Where:

  • $\boldsymbol{X}$: Data vector observations × 1.
  • $t$: Target value, usually the unweighted (or weighted) expected value $E[\boldsymbol{X}]$.

Constructors

SemiMoment() -> SemiMoment

Examples

julia> SemiMoment()SemiMoment()

Related

source