The source files can be found in user_guide/.

Data and priors

The first stage turns prices into a prior, which holds the expected returns vector and the covariance matrix that most optimisers use. Two calls cover the common path, prices_to_returns and prior. For the other moment estimators and the priors that take views, see the moments and priors examples.

using PortfolioOptimisers, CSV, TimeSeries, DataFrames, PrettyTables, LinearAlgebra,      StatsPlots, GraphRecipesresfmt = (v, i, j) -> begin    return if j == 1        v    else        isa(v, AbstractFloat) ? "$(round(v*100, digits=3)) %" : v    endend;

1. Prices to returns

Market data usually arrives as prices, and an optimiser needs returns. prices_to_returns converts a TimeArray of prices into a ReturnsResult that holds the asset names nx and the return matrix X. Before it converts its inputs, it checks that their names, dates and shapes match.

A real price table has gaps. An asset that lists or delists inside the sample has no price before or after it trades, a halted asset has no quotes, and two exchanges close on different holidays. price_ingestion reads the first and the last quote of each asset, its listing span. The conversion keeps every gap in the returns, and it returns an AssetPanel on rd.pnl that states which assets have data on each date. To fill a gap inside the listing span, use PriceGapFill. To drop the assets or the dates with too much missing data, use MissingDataFilter. The page Data preprocessing and the ingestion layer covers these steps in depth.

X = TimeArray(CSV.File(joinpath(@__DIR__, "../examples/SP500.csv.gz")); timestamp = :Date)[(end - 252):end]rd = prices_to_returns(X)

ReturnsResult
    nx ┼ 20-element Vector{String}
     X ┼ 252×20 Matrix{Float64}
    nf ┼ nothing
     F ┼ nothing
    nb ┼ nothing
     B ┼ nothing
    ne ┼ nothing
     E ┼ nothing
    ts ┼ 252-element Vector{Date}
    iv ┼ nothing
  ivpa ┼ nothing
   pnl ┼ AssetPanel
       │     pf ┼ Vector{AbstractPanelField}: AbstractPanelField[]
       │   amsk ┼ 252×20 AllTrueMask
       │   emsk ┴ 252×20 AllTrueMask

On a price table, prices_to_returns(X) is the same call as prices_to_returns(price_ingestion(PriceIngestion(), X)). Write the two calls when you have more than one table. price_ingestion takes factor prices as F, benchmark prices as B, and implied volatilities as iv. By default it aligns them on the dates of the asset prices. The PriceIngestion estimator also sets how to collapse the prices to a lower frequency. The conversion then puts the factor returns in rd.F and the benchmark returns in rd.B, on the same ReturnsResult as the asset returns. The point-in-time universe runs these two calls on a price table with gaps.

2. Returns to a prior

prior applies a prior estimator to a ReturnsResult and returns the moments. The JuMP, clustering and meta-optimisers use EmpiricalPrior by default. It computes the sample mean and the sample covariance, and then repairs the covariance to the nearest positive definite matrix if it needs to. We compute it and print the mean and the volatility of each asset.

pr = prior(EmpiricalPrior(), rd)pretty_table(DataFrame("Asset" => rd.nx, "Expected return" => pr.mu,                       "Volatility" => sqrt.(diag(pr.sigma))); formatters = [resfmt],             title = "Empirical prior: per-asset mean and volatility")

Empirical prior: per-asset mean and volatility
┌────────┬─────────────────┬────────────┐
│  Asset │ Expected return │ Volatility │
│ String │         Float64 │    Float64 │
├────────┼─────────────────┼────────────┤
│   AAPL │        -0.113 % │    2.237 % │
│    AMD │        -0.281 % │     3.84 % │
│    BAC │        -0.093 % │     2.04 % │
│    BBY │        -0.028 % │     2.85 % │
│    CVX │         0.195 % │     2.07 % │
│     GE │        -0.034 % │    2.192 % │
│     HD │        -0.071 % │    1.968 % │
│    JNJ │         0.031 % │    1.096 % │
│    JPM │        -0.042 % │    1.877 % │
│     KO │          0.05 % │    1.237 % │
│    LLY │         0.131 % │    1.714 % │
│    MRK │         0.167 % │    1.254 % │
│   MSFT │        -0.121 % │    2.214 % │
│    PEP │         0.039 % │    1.223 % │
│    PFE │        -0.026 % │    1.698 % │
│     PG │        -0.008 % │    1.382 % │
│    RRC │         0.182 % │    3.957 % │
│    UNH │         0.036 % │    1.533 % │
│    WMT │         0.016 % │    1.681 % │
│    XOM │         0.264 % │    2.207 % │
└────────┴─────────────────┴────────────┘

3. Other priors

You call every prior estimator with prior(pe, rd), and you change the prior by changing its first argument. The common alternatives are these:

• FactorPrior computes the moments from a factor model. See Factor Priors.
• BlackLittermanPrior combines the moments of a market equilibrium with your views. See Black-Litterman. Three more priors of the same family are BayesianBlackLittermanPrior, FactorBlackLittermanPrior, which takes views on the factors, and AugmentedBlackLittermanPrior, which takes views on the assets and on the factors together. See Advanced Black-Litterman.
• EntropyPoolingPrior changes the probability of each historical scenario until the scenarios satisfy your views, and it takes views on any moment. See Entropy Pooling.
• OpinionPoolingPrior combines several entropy pooling priors, each with its own views, into one probability for each scenario, and computes the moments under those probabilities. See Opinion Pooling.
• CrossSectionalFactorPrior computes the moments from a factor model that it fits across the assets on each date, not through time. On each date it regresses the returns of the assets on their exposures of the date before, by default. It therefore needs no factor returns, and the set of assets can change from date to date.

A cross-sectional prior needs more input than the others. It does not read F. It reads an AssetPanel of data for each asset on each date, such as a market capitalisation, a book equity or an industry label, which the returns hold in rd.pnl. You give it a list of pairs. Each pair names a factor and the estimator that computes the exposures of the assets to that factor from the panel. An industry label gives one factor for each industry, and one pair then gives many factors. The page Cross-sectional factor model, end to end fits one, and Cross-sectional factor model through a Pipeline gets the same weights through a Pipeline.

EmpiricalPrior holds a covariance estimator in its field ce, and you can change it for another, such as a denoised covariance or the Gerber covariance. The page Covariance Estimation compares them. A moment estimator can also use only the last part of the sample, or weight the recent returns more than the old ones. Use this when the recent returns describe the market better than the full sample does, as the page Windowed Estimators shows. RegimeAdjustedExpWeightedVariance and RegimeAdjustedExpWeightedCovariance multiply an exponentially weighted estimate by a factor that measures how far the recent standardised returns are from their usual size.

4. A first look at the data

plot_prior shows a prior in one figure: the expected returns, the volatility of each asset and the correlation matrix. Look at it before you optimise, to find an asset with an expected return or a volatility that you do not expect.

plot_prior(pr, rd)

This page was generated using Literate.jl.