The source files can be found in examples/.
Cross-sectional factor model, end to end
Apart from the two pages after this one, every factor example on this site fits a factor model through time. It regresses each asset's return series on a set of observed factor series, one regression per asset. This page fits one across the cross-section instead. At each observation it regresses that day's returns of every asset on the lagged traits of those assets, such as their size, their value and their industry. The coefficients it recovers are the factor returns of that day. Nobody supplies a factor series, because the fit produces one.
A point-in-time asset panel is what lets the fit run in that direction. A panel is a stack of named fields, such as market capitalisation, book equity and an industry label, each indexed by observation and asset. It comes with the returns as rd.pnl. Because the factor exposures come from the panel, the model can include a factor that nobody publishes a series for.
We run the whole route on a synthetic panel drawn from a factor model we wrote, so you can check every number below against an answer fixed before the estimator ran.
- Build the panel, and the traits it was drawn from.
- Fit a
CrossSectionalFactorPrioron it. - Measure what the fit recovered, and the three quantities it reproduces to machine precision.
- Hand the prior's own factor model to two orthogonal uncertainty sets, and watch the book leave the directions the factors do not span.
- Run the same optimiser through a
WalkForwardunder a factor mandate. - Compute a predicted and a realised factor attribution of the book.
Reach for a cross-sectional factor model when your view is about asset traits rather than factor series, so that you can say that cheap companies beat expensive ones but hold no published value-factor return to regress on. It is also the one route here that admits a universe whose membership changes, as assets list and delist. If you already hold factor series, FactorPrior is the simpler tool.
using PortfolioOptimisers, StableRNGs, Statistics, LinearAlgebra, Dates, PrettyTables, DataFrames, Clarabelnumfmt = (v, i, j) -> begin return isa(v, AbstractFloat) ? round(v; sigdigits = 4) : vend;1. A synthetic asset panel, and the model it was drawn from
The generator below draws nine factors, one market factor, four industries and four styles. It gives every asset a fixed trait vector of a market beta, four industry columns and four style loadings. The industry columns are 1 for the asset's own industry and 0 for the other three. An asset's return is that trait vector through the factor returns, plus an idiosyncratic shock and a small alpha of its own.
The fields are then built as noisy functions of the same traits. Log market capitalisation tracks the size trait, book-to-price tracks the value trait, and so on for the rest. Because the truth is fixed before the fit runs, you can name what the fit recovers rather than take the output on trust.
A fifth of the assets list late. The panel's active mask says which pair of an observation and an asset is live. Here it is not all true, and the universe changes membership inside the sample.
function synthetic_panel(; T = 500, N = 80, seed = 661_001) rng = StableRNG(seed) industries = ["Energy", "Financials", "Health Care", "Technology"] Kind = length(industries) ind = rand(rng, 1:Kind, N) beta = 1.0 .+ 0.35 .* randn(rng, N) Ltrue = randn(rng, N, 4) alpha = 0.0004 .* randn(rng, N) onehot = Float64[ind[i] == k for i in 1:N, k in 1:Kind] Btrue = hcat(beta, onehot, Ltrue) ftrue = hcat(0.009 .* randn(rng, T), 0.005 .* randn(rng, T, Kind), 0.004 .* randn(rng, T, 4)) ivol = 0.008 .+ 0.012 .* rand(rng, N) X = ftrue * transpose(Btrue) + randn(rng, T, N) .* transpose(ivol) .+ transpose(alpha) logcap = 20.0 .+ 1.4 .* transpose(view(Ltrue, :, 1)) .+ 0.3 .* cumsum(randn(rng, T, N); dims = 1) ./ sqrt(T) mcap = exp.(logcap) shares = mcap ./ 50.0 fields = ["market_cap" => mcap, "book_equity" => mcap .* exp.(-0.7 .+ 0.5 .* transpose(view(Ltrue, :, 2)) .+ 0.05 .* randn(rng, T, N)), "net_income_ttm" => mcap .* (0.05 .+ 0.02 .* transpose(view(Ltrue, :, 3)) .+ 0.004 .* randn(rng, T, N)), "adj_shares_outstanding" => shares, "adj_volume" => shares .* exp.(-4.5 .+ 0.6 .* transpose(view(Ltrue, :, 4)) .+ 0.15 .* randn(rng, T, N)), "signal" => transpose(alpha) .+ 0.0002 .* randn(rng, T, N)] listed = [i <= N ÷ 5 ? rand(rng, 20:120) : 1 for i in 1:N] amsk = [t >= listed[i] for t in 1:T, i in 1:N] for (_, a) in fields a[.!amsk] .= NaN end inputs = [[NumericPanelInput(; name = n, vals = a, alg = ForwardPanelFill(; val = 0.0)) for (n, a) in fields] CategoricalPanelInput(; name = "industry", vals = repeat(reshape(industries[ind], 1, N), T, 1), levels = industries)] days = filter(d -> Dates.dayofweek(d) <= 5, Date(2015, 1, 1):Day(1):(Date(2015, 1, 1) + Day(2 * T + 10)))[1:T] rd = ReturnsResult(; nx = ["A" * lpad(i, 3, '0') for i in 1:N], X = ifelse.(amsk, X, NaN), ts = days, pnl = asset_panel(inputs; amsk = amsk, emsk = amsk)) nf = vcat("market", ["industry=" * l for l in industries], ["size", "value", "earnings_yield", "liquidity"]) return (; rd = rd, B = Btrue, f = ftrue, ivar = ivol .^ 2, nf = nf)endsyn = synthetic_panel()rd = syn.rdT, N = size(rd.X)(500, 80)The panel comes with the returns. rd.X is the returns, rd.pnl is the panel, and rd.pnl.amsk is the active mask.
pretty_table(DataFrame("Assets" => N, "Observations" => T, "Panel fields" => length(rd.pnl.pf), "Share of active cells" => count(rd.pnl.amsk) / (T * N)); formatters = [numfmt], title = "The synthetic asset panel") The synthetic asset panel
┌────────┬──────────────┬──────────────┬───────────────────────┐
│ Assets │ Observations │ Panel fields │ Share of active cells │
│ Int64 │ Int64 │ Int64 │ Float64 │
├────────┼──────────────┼──────────────┼───────────────────────┤
│ 80 │ 500 │ 7 │ 0.969 │
└────────┴──────────────┴──────────────┴───────────────────────┘2. The prior
You specify a CrossSectionalFactorPrior by its factor exposures, one per factor, each naming the family it belongs to. Four kinds ship. The list below names three, and the fit uses two of them, OneHotExposure and CompositeExposure.
ConstantExposureis a column of ones, the market intercept. We use aCompositeExposureoverEWMarketBetainstead, so the market exposure is the asset's own estimated beta rather than one.OneHotExposureturns a categorical field into a block of indicator columns, one per level.CompositeExposurescores one or more descriptors across the cross-section. A descriptor is the trait itself, such asLogMarketCaporBookToPrice, and the exposure standardises it.
Three further parts of the specification change the answer.
families = ["industry" => nothing]constrains the industry factor returns, weighted by the benchmark weight of each industry, to sum to zero at every observation. An industry block sums to one for every asset, so it is close to collinear with a market factor whose exposure is a beta near one. Without the constraint the regression cannot separate the industry factor returns from the market factor return.neutralise = ["style" => "industry"]removes the benchmark-weighted overlap of the style family with the industry family, so a style factor return is not an industry bet under another name.wa = BlendedInverseVarianceWeights(...)makes the regression a two-pass fit. The first pass estimates the residual variances, and the second re-weights by them.
rfe supplies an alpha forecast. It scores the signal field, which no exposure uses. The prior splits the forecast into the part the factors span and the part they do not. lambda shrinks the fitted factor mean towards the spanned part, and at lambda = 1.0 the prior keeps the fitted factor mean. This example states lambda = 1.0. The default is PrecisionBlend(), a rule that computes lambda from the fit and gives more weight to the estimate with the smaller error. c scales the part the factors do not span. That part lands in rr.b, and it is what the orthogonal sets of section 4 act on.
style(d) = CompositeExposure(; descriptors = [d], family = "style")factors = ["market" => CompositeExposure(; descriptors = [EWMarketBeta()], outlier = nothing, scoring = nothing, family = "market"), "industry" => OneHotExposure(; field = "industry", family = "industry"), "size" => style(LogMarketCap()), "value" => style(BookToPrice()), "earnings_yield" => style(EarningsToPrice()), "liquidity" => style(EWShareTurnover())]forecast = FixedWeightedReturnForecast(; scores = DescriptorScores(; descriptors = [Passthrough(; field = "signal")], outlier = nothing, scoring = nothing), scale = 1.0)pe = CrossSectionalFactorPrior(; factors = factors, families = ["industry" => nothing], neutralise = ["style" => "industry"], wa = BlendedInverseVarianceWeights(; lambda = 0.5), rfe = forecast, lambda = 1.0, c = 1.0)pr = prior(pe, rd)rr = pr.rrpretty_table(DataFrame("Factor" => rr.nf, "Family" => rr.fam); title = "The nine factors the fit produced, and their families")The nine factors the fit produced, and their families
┌──────────────────────┬──────────┐
│ Factor │ Family │
│ String │ String │
├──────────────────────┼──────────┤
│ market │ market │
│ industry=Energy │ industry │
│ industry=Financials │ industry │
│ industry=Health Care │ industry │
│ industry=Technology │ industry │
│ size │ style │
│ value │ style │
│ earnings_yield │ style │
│ liquidity │ style │
└──────────────────────┴──────────┘The result is an ordinary LowOrderPrior over the whole asset universe, so every optimiser and estimator that takes a prior takes this one unchanged. The new part is the field rr. It is a CrossSectionalFactorModel, and it holds the exposure history Ms, assets × factors × observations, the realised factor returns, the idiosyncratic returns and variances, the regression and benchmark weights, and the family basis.
The fit keeps the tail of the observation axis, which is the window left after the warm-up of the descriptors and the exposure lag.
pretty_table(DataFrame("Fit rows" => size(pr.X, 1), "Warm-up rows" => T - size(pr.X, 1), "Factors" => size(rr.Ms, 2), "Investable assets" => count(isfinite, pr.mu)); formatters = [numfmt], title = "What the fit kept") What the fit kept
┌──────────┬──────────────┬─────────┬───────────────────┐
│ Fit rows │ Warm-up rows │ Factors │ Investable assets │
│ Int64 │ Int64 │ Int64 │ Int64 │
├──────────┼──────────────┼─────────┼───────────────────┤
│ 440 │ 60 │ 9 │ 80 │
└──────────┴──────────────┴─────────┴───────────────────┘3. What the fit recovered, and what it reproduces exactly
The recovery is statistical. Because the fields are noisy functions of the traits, a fitted exposure correlates with the truth rather than equalling it. The industry block is the exception. A one-hot exposure of a known classification is the classification itself, and the fitted block and the true block hold the same numbers.
active = findall(view(rd.pnl.amsk, T, :))loading_corr = [cor(view(rr.M, active, k), view(syn.B, active, k)) for k in eachindex(rr.nf)]industry_k = findall(isequal("industry"), rr.fam)pretty_table(DataFrame("Factor" => rr.nf, "corr(fitted, true) loading" => loading_corr); formatters = [numfmt], title = "Correlation of each fitted loading with its true loading")pretty_table(DataFrame("One-hot industry loadings recovered exactly" => rr.M[active, industry_k] == syn.B[active, industry_k]))Correlation of each fitted loading with its true loading
┌──────────────────────┬────────────────────────────┐
│ Factor │ corr(fitted, true) loading │
│ String │ Float64 │
├──────────────────────┼────────────────────────────┤
│ market │ 0.6564 │
│ industry=Energy │ 1.0 │
│ industry=Financials │ 1.0 │
│ industry=Health Care │ 1.0 │
│ industry=Technology │ 1.0 │
│ size │ 0.9051 │
│ value │ 0.9201 │
│ earnings_yield │ 0.9011 │
│ liquidity │ 0.7713 │
└──────────────────────┴────────────────────────────┘
┌─────────────────────────────────────────────┐
│ One-hot industry loadings recovered exactly │
│ Bool │
├─────────────────────────────────────────────┤
│ true │
└─────────────────────────────────────────────┘The systematic return of one observation and one asset is the quantity the model states about that asset. A single factor return is identified only up to the basis the family constraint chose, and Ms[t - lag] · f_t does not depend on that basis, so you can compare it with the generator's own.
function systematic_pairs(pr, B, f) rr = pr.rr Tf = size(rr.csr.eps, 1) ftail = view(f, (size(f, 1) - Tf + 1):size(f, 1), :) fitted = Float64[] truth = Float64[] for t in (1 + rr.lag):Tf, i in axes(rr.csr.eps, 2) u = dot(view(rr.Ms, i, :, t - rr.lag), view(pr.fpr.X, t, :)) v = dot(view(B, i, :), view(ftail, t, :)) if isfinite(u) && isfinite(v) push!(fitted, u) push!(truth, v) end end return fitted, truthendsys_fit, sys_true = systematic_pairs(pr, syn.B, syn.f)idio_ok = findall(isfinite, rr.esigma)pretty_table(DataFrame("corr(systematic return)" => cor(sys_fit, sys_true), "Pairs" => length(sys_fit), "corr(idio variance)" => cor(rr.esigma[idio_ok], syn.ivar[idio_ok]), "median ratio" => median(rr.esigma[idio_ok] ./ syn.ivar[idio_ok])); formatters = [numfmt], title = "Fitted systematic returns and idiosyncratic variances against the true ones")Fitted systematic returns and idiosyncratic variances against the true ones
┌─────────────────────────┬───────┬─────────────────────┬──────────────┐
│ corr(systematic return) │ Pairs │ corr(idio variance) │ median ratio │
│ Float64 │ Int64 │ Float64 │ Float64 │
├─────────────────────────┼───────┼─────────────────────┼──────────────┤
│ 0.9145 │ 33895 │ 0.9177 │ 1.005 │
└─────────────────────────┴───────┴─────────────────────┴──────────────┘The next three quantities are constructions rather than estimates, so the residual the cell prints for each one is at the level of machine precision.
- The lagged exposures through the factor returns, plus the idiosyncratic return, reproduce the asset's return.
- The covariance square root reproduces the covariance.
muis the loadings through the factor mean, plus the orthogonal part of the alpha forecast.
function reconciliation(pr, X) rr = pr.rr Tf = size(rr.csr.eps, 1) Xtail = view(X, (size(X, 1) - Tf + 1):size(X, 1), :) worst = 0.0 n = 0 for t in (1 + rr.lag):Tf, i in axes(rr.csr.eps, 2) v = dot(view(rr.Ms, i, :, t - rr.lag), view(pr.fpr.X, t, :)) + rr.csr.eps[t, i] if isfinite(v) && isfinite(Xtail[t, i]) worst = max(worst, abs(v - Xtail[t, i])) n += 1 end end return worst, nendrecon, recon_pairs = reconciliation(pr, rd.X)inv_i = findall(isfinite, pr.mu)pretty_table(DataFrame("max |Ms*f + eps - X|" => recon, "Pairs" => recon_pairs, "max |chol'chol - S|" => maximum(abs, transpose(pr.chol[:, inv_i]) * pr.chol[:, inv_i] - pr.sigma[inv_i, inv_i]), "max |mu - M*f - b|" => maximum(abs, view(pr.mu, inv_i) - view(rr.M * pr.fpr.mu + rr.b, inv_i))); formatters = [numfmt], title = "Three residuals of the fit") Three residuals of the fit
┌──────────────────────┬───────┬─────────────────────┬────────────────────┐
│ max |Ms*f + eps - X| │ Pairs │ max |chol'chol - S| │ max |mu - M*f - b| │
│ Float64 │ Int64 │ Float64 │ Float64 │
├──────────────────────┼───────┼─────────────────────┼────────────────────┤
│ 1.388e-17 │ 33895 │ 1.626e-19 │ 4.337e-19 │
└──────────────────────┴───────┴─────────────────────┴────────────────────┘The zero-sum constraint on the industry factor returns is a construction as well. At every observation we compute the weighted sum of the industry factor returns, relative to the size of its own terms, and the cell prints the largest of those relative sums.
function family_zero_sum(pr, family) rr = pr.rr worst = 0.0 for u in axes(pr.fpr.X, 1) #! The factor return of row u pairs with the exposures of row u - lag. t = u - rr.lag if t < 1 continue end total = 0.0 magnitude = 0.0 for k in family c = 0.0 for a in axes(rr.bw, 2) e = rr.Ms[a, k, t] isfinite(e) && (c += rr.bw[t, a] * e) end total += c * pr.fpr.X[u, k] magnitude += abs(c * pr.fpr.X[u, k]) end worst = max(worst, abs(total) / max(magnitude, eps())) end return worstendzero_sum = family_zero_sum(pr, industry_k)pretty_table(DataFrame("max relative benchmark-weighted industry return" => zero_sum); formatters = [numfmt], title = "Largest relative weighted sum of the industry factor returns")Largest relative weighted sum of the industry factor returns
┌─────────────────────────────────────────────────┐
│ max relative benchmark-weighted industry return │
│ Float64 │
├─────────────────────────────────────────────────┤
│ 4.887e-16 │
└─────────────────────────────────────────────────┘4. Two orthogonal uncertainty sets
The factor model splits the asset space in two. The loadings span one part, where the model has something to say. The rest is the orthogonal subspace, the directions the loadings do not reach, where the model says nothing. A portfolio that bets in the second part bets on estimation error.
OrthogonalUncertaintySet takes the factor model from the prior result of the optimisation it runs in, and confines the uncertainty to that second part. One estimator gives two sets.
- A low-rank norm ball on the mean. By default its radius is the square root of the 0.95 quantile of a chi-squared distribution whose degrees of freedom are the dimension
rof the orthogonal subspace. - A compact covariance set,
(κ, C, Q), whose worst-case variance isw'Σw + κ·min_z ‖Cw − Qz‖². The optimiser writes the variance and the penalty with two second-order cone constraints and needs no semidefinite constraint.Qis an orthonormal basis of the subspace the penalty leaves alone, which is the weighted factor span.
We fit both from the prior we already have.
mu_set = mu_ucs(OrthogonalUncertaintySet(), pr)sigma_set = sigma_ucs(OrthogonalUncertaintySet(), pr)pretty_table(DataFrame("Rank of the factor span" => size(sigma_set.Q, 2), "Rank of the orthogonal subspace" => size(mu_set.L, 2), "Mean-ball radius" => mu_set.kappa); formatters = [numfmt], title = "Ranks of the two subspaces and the radius of the mean set") Ranks of the two subspaces and the radius of the mean set
┌─────────────────────────┬─────────────────────────────────┬──────────────────┐
│ Rank of the factor span │ Rank of the orthogonal subspace │ Mean-ball radius │
│ Int64 │ Int64 │ Float64 │
├─────────────────────────┼─────────────────────────────────┼──────────────────┤
│ 8 │ 72 │ 9.634 │
└─────────────────────────┴─────────────────────────────────┴──────────────────┘The number that shows the effect of the set on a book is how much of the book lies outside the subspace the penalty leaves alone. That number is ‖(I − QQᵀ)Cw‖ / ‖Cw‖, and the two sweeps below move it.
universe = UniverseSets(; dict = Dict("nx" => rd.nx, "ncf" => rr.nf))solver = Solver(; name = :clarabel, solver = Clarabel.Optimizer, check_sol = (; allow_local = true, allow_almost = true), settings = Dict("verbose" => false))function orthogonal_share(w, set) v = set.C .* w return norm(v - set.Q * (transpose(set.Q) * v)) / norm(v)endfunction book(; radius = 0.0, kappa = 1.0, obj = MinimumRisk(), constraint = nothing, data = rd) u = OrthogonalUncertaintySet(; kappa = kappa, method = radius) return optimise(MeanRisk(; r = UncertaintySetVariance(; ucs = u), obj = obj, opt = JuMPOptimiser(; pe = pe, slv = solver, bgt = 1.0, wb = WeightBounds(; lb = 0.0, ub = 0.1), sets = universe, lcse = constraint, ret = ArithmeticReturn(; ucs = u))), data)endbook (generic function with 1 method)The covariance radius moves the book by degrees
κ scales a quadratic penalty on the orthogonal component, so raising it moves the book out of the orthogonal subspace one step at a time. The minimum-risk book below starts with 89% of its metric-scaled weight outside the factor span and ends with 0.15% of it.
kappa_grid = [0.0, 1.0, 10.0, 100.0, 1_000.0, 10_000.0]kappa_books = [book(; kappa = k) for k in kappa_grid]pretty_table(DataFrame("kappa" => kappa_grid, "Outside the span" => [orthogonal_share(r.w, sigma_set) for r in kappa_books], "Volatility" => [sqrt(dot(r.w, pr.sigma * r.w)) for r in kappa_books], "Expected return" => [dot(r.w, pr.mu) for r in kappa_books], "Names" => [count(>(1e-6), r.w) for r in kappa_books]); formatters = [(v, i, j) -> if j == 2 "$(round(v * 100, digits = 2)) %" elseif j == 3 || j == 4 "$(round(v * 10_000, digits = 2)) bp" else v end], title = "The compact covariance radius") The compact covariance radius
┌─────────┬──────────────────┬────────────┬─────────────────┬───────┐
│ kappa │ Outside the span │ Volatility │ Expected return │ Names │
│ Float64 │ Float64 │ Float64 │ Float64 │ Int64 │
├─────────┼──────────────────┼────────────┼─────────────────┼───────┤
│ 0.0 │ 89.15 % │ 55.23 bp │ 1.84 bp │ 38 │
│ 1.0 │ 84.58 % │ 55.83 bp │ 1.93 bp │ 49 │
│ 10.0 │ 45.52 % │ 65.72 bp │ 1.14 bp │ 68 │
│ 100.0 │ 7.84 % │ 73.93 bp │ 1.16 bp │ 72 │
│ 1000.0 │ 1.21 % │ 75.91 bp │ 1.43 bp │ 75 │
│ 10000.0 │ 0.15 % │ 76.36 bp │ 1.45 bp │ 77 │
└─────────┴──────────────────┴────────────┴─────────────────┴───────┘Calibrating the covariance radius instead of stating it
The sweep above explores κ by hand, and the numbers in kappa_grid came from nowhere in the data. You can size κ from the sample instead, by putting a rule of AbstractCompactRadiusAlgorithm in the same field. The estimator resolves the rule inside the fit, where the metric, the loadings and the factor span are all available, and the set it returns has a plain number as its radius.
Two rules ship, and they answer two different questions.
ResidualInflationderivesκfrom a confidence level. The penalty acts on the idiosyncratic variance. The rule asks how far the estimate of that variance can be from the truth, and a variance has a chi-squared bound. Under this model's default metric the answer has no unit, because it is the relative inflation itself.VarianceFractiontreatsκas a magnitude with a unit. It sizes the penalty so that a reference portfolio pays a stated fraction of its nominal variance.
calibrated = ["Stated" => 100.0, "ResidualInflation()" => ResidualInflation(), "ResidualInflation(; q = 0.01)" => ResidualInflation(; q = 0.01), "VarianceFraction(; f = 0.1)" => VarianceFraction(; f = 0.1), "VarianceFraction(; f = 0.5)" => VarianceFraction(; f = 0.5)]calibrated_sets = [sigma_ucs(OrthogonalUncertaintySet(; kappa = k), rd, pr) for (_, k) in calibrated]calibrated_books = [book(; kappa = k) for (_, k) in calibrated]pretty_table(DataFrame("kappa" => first.(calibrated), "Resolved" => [s.kappa for s in calibrated_sets], "Outside the span" => [orthogonal_share(r.w, sigma_set) for r in calibrated_books], "Volatility" => [sqrt(dot(r.w, pr.sigma * r.w)) for r in calibrated_books], "Names" => [count(>(1e-6), r.w) for r in calibrated_books]); formatters = [(v, i, j) -> if j == 2 round(v; sigdigits = 4) elseif j == 3 "$(round(v * 100, digits = 2)) %" elseif j == 4 "$(round(v * 10_000, digits = 2)) bp" else v end], title = "Stated and calibrated covariance radii, and the books they give") Stated and calibrated covariance radii, and the books they give
┌───────────────────────────────┬──────────┬──────────────────┬────────────┬───────┐
│ kappa │ Resolved │ Outside the span │ Volatility │ Names │
│ String │ Float64 │ Float64 │ Float64 │ Int64 │
├───────────────────────────────┼──────────┼──────────────────┼────────────┼───────┤
│ Stated │ 100.0 │ 7.84 % │ 73.93 bp │ 72 │
│ ResidualInflation() │ 0.4266 │ 86.9 % │ 55.4 bp │ 41 │
│ ResidualInflation(; q = 0.01) │ 0.5842 │ 86.17 % │ 55.51 bp │ 48 │
│ VarianceFraction(; f = 0.1) │ 9.077 │ 48.27 % │ 65.11 bp │ 68 │
│ VarianceFraction(; f = 0.5) │ 45.39 │ 14.7 % │ 72.31 bp │ 72 │
└───────────────────────────────┴──────────┴──────────────────┴────────────┴───────┘The two rules give very different radii. Here ResidualInflation returns about 0.43, more than two orders of magnitude below the 100.0 the sweep above needed to move the book. A chi-squared bound on a residual variance measures estimation error, and over this sample that error is small. The fit records how many observations each variance is worth, and the default exponentially weighted variance is worth about 115 of them, not the 440 rows of the fit, so the bound is wider than the row count suggests. A radius of that size still moves the book very little, and the table shows about 87% of the metric-scaled weight still outside the factor span. Reach for VarianceFraction when you want the book to move, because it is sized against the nominal variance rather than against the sampling error. It is also linear in f, so the resolved radius of the f = 0.5 row is five times that of the f = 0.1 row.
ResidualInflation sizes the radius from the sampling error of the residual variance. VarianceFraction sizes it from the nominal variance. A stated 100.0 comes from neither, and it stays the same on every fold.
VarianceFraction needs a reference portfolio. Its w0 takes a weight vector, or any optimiser that is not a finite allocation. The fit needs nothing more, because an optimiser has its own solver. nothing uses the equal-weight book.
The two rules also apply to different factor models. ResidualInflation needs the idiosyncratic variances in rr.esigma, and it raises an error on a factor model fitted without them. VarianceFraction does not use them, but the set's default metric, InverseIdiosyncraticVarianceMetric, inverts them and raises the same error. On such a model, give the set metric = nothing as well, which weighs every asset equally. The q of ResidualInflation defaults to the q of the OrthogonalUncertaintySet, so one confidence level sizes both the mean set and the covariance set unless you set them apart. The two qs are tail probabilities of different errors, and a smaller q gives a larger radius in both sets.
vf_book = book(; kappa = VarianceFraction(; f = 0.1, w0 = InverseVolatility()))pretty_table(DataFrame("Reference" => ["Equal weight (default)", "InverseVolatility()"], "Outside the span" => [orthogonal_share(calibrated_books[4].w, sigma_set), orthogonal_share(vf_book.w, sigma_set)]); formatters = [(v, i, j) -> j == 2 ? "$(round(v * 100, digits = 2)) %" : v], title = "Share outside the span under VarianceFraction(; f = 0.1), at two reference portfolios")[ Info: 16 asset(s) left the investable universe and are excluded from this optimisation: ["A001", "A002", "A003", "A004", "A005", "A006", "A007", "A008", "A009", "A010", "A011", "A012", "A013", "A014", "A015", "A016"]. A constraint, bound or rate stated for one of them is dropped, and a liquidation charge is priced over them.
Share outside the span under VarianceFraction(; f = 0.1), at two reference portfolios
┌────────────────────────┬──────────────────┐
│ Reference │ Outside the span │
│ String │ Float64 │
├────────────────────────┼──────────────────┤
│ Equal weight (default) │ 48.27 % │
│ InverseVolatility() │ 60.64 % │
└────────────────────────┴──────────────────┘The radius is also searchable
You do not have to choose any of this in advance. kappa is a plain field, so a search grid can range over it with the key "r.ucs.kappa" on the strategy optimiser of section 5, and one grid can hold rules and numbers together. Each candidate is fitted per fold, and the walk-forward score decides between them. That is the third route, after stating a size and after calibrating one.
grid = ["r.ucs.kappa" => [0.0, 1.0, 100.0, ResidualInflation(), VarianceFraction(; f = 0.1)]]search_cross_validation(strategy, GridSearchCrossValidation(grid; cv = IndexWalkForward(252, 63)), rd)The mean radius has a threshold, and past it the radius changes nothing
The mean set behaves differently. Its penalty is the norm −κ‖Lᵀw‖, and a norm is not differentiable at zero. Once the radius passes a threshold, the orthogonal component of the book is exactly zero, and a larger radius changes nothing. On this sample the threshold is below 0.5, the first radius above zero in the grid. The maximum-return book below pays 6 bp of expected return for it, and it goes from 10 names to 73.
radius_grid = [0.0, 0.5, 1.0, 2.0, 4.0, mu_set.kappa]radius_books = [book(; radius = rad, obj = MaximumReturn()) for rad in radius_grid]pretty_table(DataFrame("Radius" => radius_grid, "Outside the span" => [orthogonal_share(r.w, sigma_set) for r in radius_books], "Volatility" => [sqrt(dot(r.w, pr.sigma * r.w)) for r in radius_books], "Expected return" => [dot(r.w, pr.mu) for r in radius_books], "Names" => [count(>(1e-6), r.w) for r in radius_books]); formatters = [(v, i, j) -> if j == 1 round(v; digits = 3) elseif j == 2 "$(round(v * 100, digits = 4)) %" elseif j == 3 || j == 4 "$(round(v * 10_000, digits = 2)) bp" else v end], title = "The mean norm-ball radius") The mean norm-ball radius
┌─────────┬──────────────────┬────────────┬─────────────────┬───────┐
│ Radius │ Outside the span │ Volatility │ Expected return │ Names │
│ Float64 │ Float64 │ Float64 │ Float64 │ Int64 │
├─────────┼──────────────────┼────────────┼─────────────────┼───────┤
│ 0.0 │ 67.0121 % │ 105.54 bp │ 22.09 bp │ 10 │
│ 0.5 │ 0.0 % │ 93.4 bp │ 16.06 bp │ 73 │
│ 1.0 │ 0.0 % │ 93.4 bp │ 16.06 bp │ 73 │
│ 2.0 │ 0.0 % │ 93.4 bp │ 16.06 bp │ 73 │
│ 4.0 │ 0.0 % │ 93.4 bp │ 16.06 bp │ 73 │
│ 9.634 │ 0.0 % │ 93.4 bp │ 16.06 bp │ 73 │
└─────────┴──────────────────┴────────────┴─────────────────┴───────┘MaximumRatio solves a homogenised problem in a scaled variable k. A mean uncertainty set wide enough that no feasible portfolio's worst case beats rf leaves nothing to pin that scale. The objective is then non-positive along every ray, and its supremum is at the origin. MaximumRatio writes a floor k >= kmin for this case, so the constraints keep their meaning and the recovered weights keep the mandate. A k that comes back on the floor tells you that there was no tangency portfolio to find, and that the weights beside it maximise the return expression at that scale rather than the ratio. The books on this page use MinimumRisk and MaximumReturn, whose scale is fixed at one, so the question does not arise for them.
5. A walk-forward, under a factor mandate
Everything above was fitted on one sample. A WalkForward refits all of it per fold. The prior refits on the fold's own rows, the two uncertainty sets refit against that fold's factor model, and a factor-exposure constraint written in a factor name uses the loadings that the fold fitted.
The mandate below is one line of the constraint grammar, "size >= 0.10", wrapped in an ExposureConstraintEstimator that names the space the factor name lives in.
mandate = ExposureConstraintEstimator(; lce = LinearConstraintEstimator(; val = "size >= 0.10"), space = FactorSpace())ucs_wf = OrthogonalUncertaintySet(; kappa = 100.0)strategy = MeanRisk(; r = UncertaintySetVariance(; ucs = ucs_wf), obj = MinimumRisk(), opt = JuMPOptimiser(; pe = pe, slv = solver, bgt = 1.0, wb = WeightBounds(; lb = 0.0, ub = 0.1), sets = universe, lcse = mandate, ret = ArithmeticReturn(; ucs = ucs_wf)))walk = IndexWalkForward(252, 63)folds = cross_val_predict(strategy, rd, walk)size_k = findfirst(isequal("size"), rr.nf)fold_priors = [p.res.jor.pr for p in folds.pred]fold_sets = [sigma_ucs(OrthogonalUncertaintySet(), prf) for prf in fold_priors]pretty_table(DataFrame("Fold" => eachindex(folds.pred), "Fit rows" => [size(prf.X, 1) for prf in fold_priors], "Size exposure" => [(transpose(fold_priors[i].rr.M) * folds.pred[i].res.w)[size_k] for i in eachindex(folds.pred)], "norm(M_fold - M_full)" => [norm(prf.rr.M - rr.M) for prf in fold_priors], "Outside the span" => [orthogonal_share(folds.pred[i].res.w, fold_sets[i]) for i in eachindex(folds.pred)], "Names" => [count(>(1e-6), p.res.w) for p in folds.pred]); formatters = [numfmt], title = "The walk-forward book, fold by fold") The walk-forward book, fold by fold
┌───────┬──────────┬───────────────┬───────────────────────┬──────────────────┬───────┐
│ Fold │ Fit rows │ Size exposure │ norm(M_fold - M_full) │ Outside the span │ Names │
│ Int64 │ Int64 │ Float64 │ Float64 │ Float64 │ Int64 │
├───────┼──────────┼───────────────┼───────────────────────┼──────────────────┼───────┤
│ 1 │ 192 │ 0.1 │ 3.185 │ 0.4562 │ 37 │
│ 2 │ 192 │ 0.1 │ 3.292 │ 0.4588 │ 36 │
│ 3 │ 192 │ 0.1 │ 2.956 │ 0.4859 │ 36 │
└───────┴──────────┴───────────────┴───────────────────────┴──────────────────┴───────┘The norm column is not zero, which shows that each fold refit its loadings. The size exposure is 0.1 in every fold, which is the bound of the mandate. The share outside the span is measured against the covariance set of each fold's own prior.
6. Factor attribution
The last question is where the book's risk and return came from. factor_attribution answers it twice. The predicted decomposition uses the moments of the prior. The realised one takes a return series and decomposes what happened.
final = book(; kappa = 100.0, constraint = mandate)predicted = factor_attribution(final.w, final.jor.pr; assets = true)FactorAttributionResult
sys ┼ AttributionComponent
│ vol ┼ Float64: 0.01056434271345619
│ vol_contrib ┼ Float64: 0.010255402861846554
│ pct_var ┼ Float64: 0.9423679074553059
│ mu_contrib ┼ Float64: 0.00104685125624161
│ corr ┼ Float64: 0.9707563584418625
│ mu_se ┴ nothing
idio ┼ AttributionComponent
│ vol ┼ Float64: 0.0026125489234783764
│ vol_contrib ┼ Float64: 0.0006271863909426404
│ pct_var ┼ Float64: 0.057632092544694105
│ mu_contrib ┼ Float64: 1.0064611233530387e-5
│ corr ┼ Float64: 0.24006685015781354
│ mu_se ┴ nothing
unattr ┼ AttributionComponent
│ vol ┼ Float64: NaN
│ vol_contrib ┼ Float64: 2.335013095446582e-19
│ pct_var ┼ Float64: 2.1456411164724614e-17
│ mu_contrib ┼ Float64: 1.4907779871675686e-19
│ corr ┼ Float64: NaN
│ mu_se ┴ nothing
total ┼ AttributionComponent
│ vol ┼ Float64: 0.010882589252789197
│ vol_contrib ┼ Float64: 0.010882589252789197
│ pct_var ┼ Float64: 1.0
│ mu_contrib ┼ Float64: 0.0010569158674751406
│ corr ┼ Float64: 1.0
│ mu_se ┴ nothing
fbd ┼ AttributionBreakdown
│ labels ┼ nothing
│ exposure ┼ 9-element Vector{Float64}
│ exposure_std ┼ nothing
│ vol ┼ 9-element Vector{Float64}
│ corr ┼ 9-element Vector{Float64}
│ vol_contrib ┼ 9-element Vector{Float64}
│ pct_var ┼ 9-element Vector{Float64}
│ mu ┼ 9-element Vector{Float64}
│ mu_contrib ┼ 9-element Vector{Float64}
│ mu_se ┴ nothing
fmbd ┼ AttributionBreakdown
│ labels ┼ Vector{String}: ["industry", "market", "style"]
│ exposure ┼ Vector{Float64}: [1.0, 0.9674644832305037, -0.08600037745247105]
│ exposure_std ┼ nothing
│ vol ┼ nothing
│ corr ┼ nothing
│ vol_contrib ┼ Vector{Float64}: [0.0003954570520682828, 0.009775509151340005, 8.443665843826823e-5]
│ pct_var ┼ Vector{Float64}: [0.036338507581449665, 0.8982705240698625, 0.007758875803993723]
│ mu ┼ nothing
│ mu_contrib ┼ Vector{Float64}: [0.00035130440686532267, 0.0006469039958920698, 4.864285348421773e-5]
│ mu_se ┴ nothing
abd ┼ AssetAttributionBreakdown
│ weight ┼ 80-element Vector{Float64}
│ weight_std ┼ nothing
│ sys_vol_contrib ┼ 80-element Vector{Float64}
│ sys_mu_contrib ┼ 80-element Vector{Float64}
│ idio_vol_contrib ┼ 80-element Vector{Float64}
│ idio_mu_contrib ┼ 80-element Vector{Float64}
│ vol ┼ 80-element Vector{Float64}
│ corr ┼ 80-element Vector{Float64}
│ vol_contrib ┼ 80-element Vector{Float64}
│ pct_var ┼ 80-element Vector{Float64}
│ mu ┼ 80-element Vector{Float64}
│ mu_contrib ┴ 80-element Vector{Float64}
afc ┼ AssetFactorContribution
│ vol_contrib ┼ 80×9 Matrix{Float64}
│ mu_contrib ┴ 80×9 Matrix{Float64}
realised ┼ Bool: false
ppy ┴ Int64: 1
The realised call needs returns. Our panel lists a fifth of its assets late, and a held asset can have a non-finite return at an observation before it listed. Under strict = false the call warns and zeroes those pairs, and under strict = true it raises. The warning names the assets and counts the pairs, so you can see when the total is understated.
realised = factor_attribution(final.w, final.jor.pr, rd.X; assets = true, strict = false)FactorAttributionResult
sys ┼ AttributionComponent
│ vol ┼ Float64: 0.011157738946340525
│ vol_contrib ┼ Float64: 0.011064180112188054
│ pct_var ┼ Float64: 0.9428980552063736
│ mu_contrib ┼ Float64: 0.0007947260608330455
│ corr ┼ Float64: 0.991614893070862
│ mu_se ┴ nothing
idio ┼ AttributionComponent
│ vol ┼ Float64: 0.0010858501847667173
│ vol_contrib ┼ Float64: 0.0002466126216648694
│ pct_var ┼ Float64: 0.021016519886638513
│ mu_contrib ┼ Float64: 6.4734722180881284e-6
│ corr ┼ Float64: 0.22711477616762696
│ mu_se ┴ nothing
unattr ┼ AttributionComponent
│ vol ┼ Float64: 0.001067407004828637
│ vol_contrib ┼ Float64: 0.0004234345785222408
│ pct_var ┼ Float64: 0.036085424906987934
│ mu_contrib ┼ Float64: 8.546625175277071e-5
│ corr ┼ Float64: 0.39669458473360825
│ mu_se ┴ nothing
total ┼ AttributionComponent
│ vol ┼ Float64: 0.011734227312375164
│ vol_contrib ┼ Float64: 0.011734227312375164
│ pct_var ┼ Float64: 1.0
│ mu_contrib ┼ Float64: 0.0008866657848039042
│ corr ┼ Float64: 1.0
│ mu_se ┴ nothing
fbd ┼ AttributionBreakdown
│ labels ┼ nothing
│ exposure ┼ 9-element Vector{Float64}
│ exposure_std ┼ 9-element Vector{Float64}
│ vol ┼ 9-element Vector{Float64}
│ corr ┼ 9-element Vector{Float64}
│ vol_contrib ┼ 9-element Vector{Float64}
│ pct_var ┼ 9-element Vector{Float64}
│ mu ┼ 9-element Vector{Float64}
│ mu_contrib ┼ 9-element Vector{Float64}
│ mu_se ┴ nothing
fmbd ┼ AttributionBreakdown
│ labels ┼ Vector{String}: ["industry", "market", "style"]
│ exposure ┼ Vector{Float64}: [0.9687760966714908, 0.9489987834227565, -0.03892545392276419]
│ exposure_std ┼ Vector{Float64}: [0.06958126704779771, 0.06402842424977426, 0.041370326926070955]
│ vol ┼ nothing
│ corr ┼ nothing
│ vol_contrib ┼ Vector{Float64}: [0.0005239564806874252, 0.010526874784423193, 1.3348847077438807e-5]
│ pct_var ┼ Vector{Float64}: [0.04465197978011298, 0.8971084762710647, 0.0011375991551962551]
│ mu ┼ nothing
│ mu_contrib ┼ Vector{Float64}: [0.00034603696832041, 0.00046667235248771155, -1.7983259975076223e-5]
│ mu_se ┴ nothing
abd ┼ AssetAttributionBreakdown
│ weight ┼ 80-element Vector{Float64}
│ weight_std ┼ 80-element Vector{Float64}
│ sys_vol_contrib ┼ 80-element Vector{Float64}
│ sys_mu_contrib ┼ 80-element Vector{Float64}
│ idio_vol_contrib ┼ 80-element Vector{Float64}
│ idio_mu_contrib ┼ 80-element Vector{Float64}
│ vol ┼ 80-element Vector{Float64}
│ corr ┼ 80-element Vector{Float64}
│ vol_contrib ┼ 80-element Vector{Float64}
│ pct_var ┼ 80-element Vector{Float64}
│ mu ┼ 80-element Vector{Float64}
│ mu_contrib ┴ 80-element Vector{Float64}
afc ┼ AssetFactorContribution
│ vol_contrib ┼ 80×9 Matrix{Float64}
│ mu_contrib ┴ 80×9 Matrix{Float64}
realised ┼ Bool: true
ppy ┴ Int64: 1
The cell below prints five residuals per decomposition. It measures the systematic, idiosyncratic and unattributed parts against the portfolio's own volatility and return, and the per-factor contributions against the systematic part. Each residual is at the level of machine precision.
function attribution_residuals(a) return (a.sys.vol_contrib + a.idio.vol_contrib + a.unattr.vol_contrib - a.total.vol_contrib, a.sys.mu_contrib + a.idio.mu_contrib + a.unattr.mu_contrib - a.total.mu_contrib, a.sys.pct_var + a.idio.pct_var + a.unattr.pct_var - 1, sum(a.fbd.vol_contrib) - a.sys.vol_contrib, sum(a.fmbd.vol_contrib) - a.sys.vol_contrib)endresiduals = [attribution_residuals(predicted), attribution_residuals(realised)]pretty_table(DataFrame("Decomposition" => ["Predicted", "Realised"], "sigma parts - sigma" => [r[1] for r in residuals], "mu parts - mu" => [r[2] for r in residuals], "variance shares - 1" => [r[3] for r in residuals], "factors - systematic" => [r[4] for r in residuals], "families - systematic" => [r[5] for r in residuals]); formatters = [numfmt], title = "Residuals of the two decompositions")pretty_table(DataFrame("Component" => ["Systematic", "Idiosyncratic", "Unattributed", "Total"], "Predicted sigma contribution" => [predicted.sys.vol_contrib, predicted.idio.vol_contrib, predicted.unattr.vol_contrib, predicted.total.vol_contrib], "Predicted % variance" => [predicted.sys.pct_var, predicted.idio.pct_var, predicted.unattr.pct_var, 1.0], "Realised sigma contribution" => [realised.sys.vol_contrib, realised.idio.vol_contrib, realised.unattr.vol_contrib, realised.total.vol_contrib], "Realised mu contribution" => [realised.sys.mu_contrib, realised.idio.mu_contrib, realised.unattr.mu_contrib, realised.total.mu_contrib]); formatters = [numfmt], title = "Where the book's risk and return came from")pretty_table(DataFrame("Family" => rr.fam, "Factor" => rr.nf, "Predicted sigma contribution" => predicted.fbd.vol_contrib, "Predicted % variance" => predicted.fbd.pct_var, "Realised sigma contribution" => realised.fbd.vol_contrib); formatters = [numfmt], title = "By factor") Residuals of the two decompositions
┌───────────────┬─────────────────────┬───────────────┬─────────────────────┬──────────────────────┬───────────────────────┐
│ Decomposition │ sigma parts - sigma │ mu parts - mu │ variance shares - 1 │ factors - systematic │ families - systematic │
│ String │ Float64 │ Float64 │ Float64 │ Float64 │ Float64 │
├───────────────┼─────────────────────┼───────────────┼─────────────────────┼──────────────────────┼───────────────────────┤
│ Predicted │ -1.735e-18 │ 0.0 │ 0.0 │ 1.735e-18 │ 0.0 │
│ Realised │ 0.0 │ 1.084e-19 │ 0.0 │ 3.469e-18 │ 3.469e-18 │
└───────────────┴─────────────────────┴───────────────┴─────────────────────┴──────────────────────┴───────────────────────┘
Where the book's risk and return came from
┌───────────────┬──────────────────────────────┬──────────────────────┬─────────────────────────────┬──────────────────────────┐
│ Component │ Predicted sigma contribution │ Predicted % variance │ Realised sigma contribution │ Realised mu contribution │
│ String │ Float64 │ Float64 │ Float64 │ Float64 │
├───────────────┼──────────────────────────────┼──────────────────────┼─────────────────────────────┼──────────────────────────┤
│ Systematic │ 0.01026 │ 0.9424 │ 0.01106 │ 0.0007947 │
│ Idiosyncratic │ 0.0006272 │ 0.05763 │ 0.0002466 │ 6.473e-6 │
│ Unattributed │ 2.335e-19 │ 2.146e-17 │ 0.0004234 │ 8.547e-5 │
│ Total │ 0.01088 │ 1.0 │ 0.01173 │ 0.0008867 │
└───────────────┴──────────────────────────────┴──────────────────────┴─────────────────────────────┴──────────────────────────┘
By factor
┌──────────┬──────────────────────┬──────────────────────────────┬──────────────────────┬─────────────────────────────┐
│ Family │ Factor │ Predicted sigma contribution │ Predicted % variance │ Realised sigma contribution │
│ String │ String │ Float64 │ Float64 │ Float64 │
├──────────┼──────────────────────┼──────────────────────────────┼──────────────────────┼─────────────────────────────┤
│ market │ market │ 0.009776 │ 0.8983 │ 0.01053 │
│ industry │ industry=Energy │ -0.0001385 │ -0.01272 │ -0.0001598 │
│ industry │ industry=Financials │ 0.0003099 │ 0.02848 │ 0.0002827 │
│ industry │ industry=Health Care │ 7.333e-5 │ 0.006739 │ 0.0002988 │
│ industry │ industry=Technology │ 0.0001507 │ 0.01384 │ 0.0001023 │
│ style │ size │ 1.091e-5 │ 0.001002 │ -4.1e-5 │
│ style │ value │ 1.492e-6 │ 0.0001371 │ -2.15e-6 │
│ style │ earnings_yield │ 5.417e-5 │ 0.004978 │ 3.844e-5 │
│ style │ liquidity │ 1.787e-5 │ 0.001642 │ 1.807e-5 │
└──────────┴──────────────────────┴──────────────────────────────┴──────────────────────┴─────────────────────────────┘Where to go next
- Cross-sectional factor model through a Pipeline reaches the same weights with the panel passed into the pipeline as data.
- Reading a return forecast before an optimiser sees it scores the
rfeof section 2 out of sample, against what happened next, and compares it with another. - Factor priors is the counterpart of section 2 through time.
- Uncertainty sets covers the box, ellipsoidal and norm-ball shapes that the orthogonal sets of section 4 specialise.
- Factor exposure constraints is the full grammar behind the one-line mandate of section 5.
This page was generated using Literate.jl.