Scoped config: private API

Six settings apply to the whole package. Each is in a ScopedConfig, which many threads can read at once.

  • COMPACT_SHOW sets how far the printout collapses a large nested type.
  • SHOW_NOTHING_FIELDS sets whether the printout shows a field that holds nothing.
  • STRING_DISTANCE sets how close a misspelled name must be for an error to suggest a correction.
  • EQUATION_LIMITS sets the largest equation that the parser accepts.
  • RESOURCE_LIMITS sets the largest draw counts, subset counts and grids that the sampling and sweep estimators accept.
  • LOG_LIMITS sets how many times one log site repeats one message, for each log level.

The set_*! and with_* functions that change these settings are public, see Scoped config. You can also set the defaults of a project at load time through Preferences.jl.

PortfolioOptimisers.RESOURCE_LIMITS — Constant
RESOURCE_LIMITS = ScopedConfig(ResourceLimits())

Default global resource caps for the estimators that sample or sweep. The caps stop a configuration value that is large enough to use all of the memory, or to keep the process busy for days. Each estimator reads RESOURCE_LIMITS[], and passes the cap to assert_resource_cap. The "max_n_sim", "max_n_subsets", "max_frontier", "max_bins", "max_hop_count", "max_search_grid" and "max_ep_grid" preferences can set the defaults for a project at load time, through apply_preferences!.

Related

source
PortfolioOptimisers.COMPACT_SHOW — Constant

Global control for the collapse of large nested structs in the output of @define_pretty_show.

It holds one of these values:

  • false: The collapse is off, and a nested struct always prints in full.
  • true: The collapse is on, with a line budget that the size of the terminal sets.
  • n::Int: The collapse is on, with a fixed line budget of n.

It is a ScopedConfig. set_compact_show! sets the global default, and with_compact_show sets an override for one scope. compact_show_budget reads it together with the per-call :po_compact IO property. The "compact_show" preference can set the default for a project at load time, through apply_preferences!.

source
PortfolioOptimisers.SHOW_NOTHING_FIELDS — Constant
SHOW_NOTHING_FIELDS = ScopedConfig(ShowNothingFields(false, Dict{Symbol, Bool}()))

Global control for whether @define_pretty_show renders a field that holds nothing.

The package default hides such a field, because a reader at the REPL wants the fields that hold a value. The documentation build shows these fields with set_show_nothing_fields!(true), next to its set_compact_show!(false) call, so that a rendered docstring shows the complete type. pretty_show_fields reads SHOW_NOTHING_FIELDS[]. The "show_nothing_fields" and "show_nothing_fields_by_type" preferences can set the default for a project at load time, through apply_preferences!.

Related

source
PortfolioOptimisers.STRING_DISTANCE — Constant
STRING_DISTANCE = ScopedConfig(StringDistanceConfig(StringDistances.Levenshtein(), 0.7))

Default string distance configuration for the fuzzy "did you mean?" suggestions that did_you_mean adds to a "variable not in asset universe" message. did_you_mean reads STRING_DISTANCE[]. The "suggestion_distance" and "suggestion_min_score" preferences can set the defaults for a project at load time, through apply_preferences!.

Related

source
PortfolioOptimisers.EQUATION_LIMITS — Constant
EQUATION_LIMITS = ScopedConfig(EquationLimits(4096, 256))

Default global resource caps for the parse of an equation. The caps stop a string whose expression tree is deep enough to use all of the stack. parse_equation and each other parser of an equation string read EQUATION_LIMITS[]. The "equation_max_length" and "equation_max_depth" preferences can set the defaults for a project at load time, through apply_preferences!.

Related

source
PortfolioOptimisers.ScopedConfig — Type
mutable struct ScopedConfig{T}

Thread-safe holder for a package-level configuration value. It holds a global default that persists, and a task-scoped override.

A read is cfg[]. Inside a with_* block it returns the innermost active scoped override, and in other code it returns the global default. The default is an @atomic field, and a set_*! call replaces the whole value in one atomic store. A concurrent reader thus never sees a configuration that is half written. The FLoops.@floop loops inside the meta-optimisers are such readers. The scoped override is a Base.ScopedValues.ScopedValue. A task spawned inside the scope gets the override, the override goes away when the scope exits, and a concurrent task outside the scope does not see it.

Each configuration of this kind holds an immutable struct or a bits value. A change to one setting builds a new value and stores it in place of the old value. No call mutates the value in place.

COMPACT_SHOW, SHOW_NOTHING_FIELDS, STRING_DISTANCE, EQUATION_LIMITS, RESOURCE_LIMITS and LOG_LIMITS are of this type. A set_*! function sets the global default of each, and a with_* function sets a scoped override. Preferences.jl sets a default for each project at load time, through apply_preferences!.

Related

source
PortfolioOptimisers.ResourceLimits — Type
struct ResourceLimits

Global resource caps for the estimators that sample or sweep. The caps stop a configuration value that is large enough to use all of the memory, or to keep the process busy for days.

A draw count, a subset count, the length of a frontier sweep and a histogram bin count are untrusted configuration integers. They can come from a configuration file, a tuning grid or a user interface. Each one multiplies the size of an allocation, and a subset count or a frontier length multiplies the number of optimisations. The constructors of these estimators bound the values only from below, with n_sim > 0, n_subsets >= 2, N > 0 and bins > 0. A value that is much too large, from an extra digit or a sweep with the wrong scale, passes these checks. The out-of-memory killer then stops the process, and the caller does not get a message. These caps raise a DomainError at the point where the library resolves the value.

Each allocation or loop that a value drives has its own cap, and the name of the cap follows the field that it checks. The caps are not shared, because a linear cap cannot bound a quadratic allocation. The bins × bins histogram has its own max_bins, and does not share the linear cap on draws.

The defaults are fixed values far above real use. A cap turns a kill by the out-of-memory killer into an error with a type. It does not judge the size that the caller chose. The value is immutable. The ScopedConfig RESOURCE_LIMITS holds it. set_resource_limits! sets the global default, and with_resource_limits sets an override for one scope. Use the keyword constructor ResourceLimits(; …). The seven caps have one type and four of them have the value 100_000, so a positional call can easily put a value in the wrong place.

Fields

  • max_n_sim: The maximum number of Monte-Carlo or bootstrap draws, n_sim, that NormalUncertaintySet and ARCHUncertaintySet accept. The default is 1_000_000. Each draw stores an N × N covariance, so the array holds N² · n_sim elements. At 20 assets the default cap permits a request of 3.2 GB, and the default n_sim is 3_000. This cap bounds memory.
  • max_n_subsets: The maximum number of resampled asset subsets, n_subsets, that SubsetResampling and MultipleRandomised accept. The default is 100_000. This cap bounds compute much more than memory, because each subset runs a full inner optimisation. The cap is far above a real sweep, where the default is 2. It is far below a value that keeps a session busy for days.
  • max_frontier: The maximum number of points of an efficient-frontier sweep that the Frontier algorithm of MeanRisk and NearOptimalCentering accepts. The default is 100_000. Each point runs a full inner optimise_JuMP_model! solve, so this cap bounds compute, and it has the same value as max_n_subsets. The default of Frontier is N = 20. The library checks this cap at two points. The constructor of Frontier caps the N of one bound. At Model Assembly, assert_frontier_sweep_cap caps the product over every swept return term and every swept risk measure. The sweep is an Iterators.product, so k bounds of N points cost N^k solves.
  • max_bins: The maximum number of histogram bins that MutualInfoCovariance and VariationInfoDistance accept. The default is 10_000. The joint histogram is a bins × bins weights matrix for each pair of assets, so this cap bounds a quadratic allocation of memory. A histogram of 10_000² cells takes about 800 MB. That size is below the limit of the out-of-memory killer, and far above the 50 bins, approximately, that a real binning gives.
  • max_hop_count: The maximum hop count, n, that HopCount accepts. The default is 100_000. Three readers sum A^i over i in 0:n, so the cost is linear in n, at N³ floating point operations for each power. A large n keeps the session busy on compute, and not on memory. This cap bounds compute, and it has the same value as max_n_subsets. The default hop count is n = 1. The constructor of HopCount reads the cap, and resolve_separation also builds a HopCount from the value that a rule computes. One check thus applies to a stated value and to a computed value.
  • max_search_grid: The maximum number of candidates in a search grid that GridSearchCrossValidation and RandomisedSearchCrossValidation accept. The default is 100_000. Each candidate runs a full cross-validated fit, so this cap bounds compute, as max_n_subsets does. collect builds the grid from an Iterators.product, so k parameters of N values give N^k candidates. assert_search_grid_cap checks the cap on that product where the grid forms, because a check on each parameter cannot see the product.
  • max_ep_grid: The maximum number of grid points, K, that GridTailView accepts. The default is 10_000. Each grid point is one binary variable of the mixed-integer program that an upper-bound or equality view builds. It is also one dense row over the T posterior probabilities. The memory is linear in K, at K · T coefficients, and the compute is a branch-and-bound tree over K binary variables. The cap is far below the compute caps that bound a plain solve, and far above the default K = 11.

Constructors

ResourceLimits(max_n_sim::Integer, max_n_subsets::Integer, max_frontier::Integer,               max_bins::Integer, max_hop_count::Integer, max_search_grid::Integer,               max_ep_grid::Integer) -> ResourceLimitsResourceLimits(;    max_n_sim::Integer = 1_000_000,    max_n_subsets::Integer = 100_000,    max_frontier::Integer = 100_000,    max_bins::Integer = 10_000,    max_hop_count::Integer = 100_000,    max_search_grid::Integer = 100_000,    max_ep_grid::Integer = 10_000) -> ResourceLimits

Each keyword sets the field of the same name.

Validation

  • max_n_sim > 0 && max_n_subsets > 0 && max_frontier > 0 && max_bins > 0 && max_hop_count > 0 && max_search_grid > 0 && max_ep_grid > 0.

Related

source
PortfolioOptimisers.ShowNothingFields — Type
struct ShowNothingFields

Configuration for whether @define_pretty_show renders a field that holds nothing.

The ScopedConfig SHOW_NOTHING_FIELDS holds it, and pretty_show_fields reads it. set_show_nothing_fields! sets the global default, and with_show_nothing_fields sets an override for one scope. No call mutates a value. A change builds a new value with a copy of by_type, and stores it in place of the old value. A reader thus never sees a table that is half written.

Fields

  • default: Whether a field that holds nothing prints, for a type that no per-name entry names. The package default is false, which hides a nothing field.
  • by_type: The per-name overrides. The key is the name of a type without its module. A true entry prints every declared field of that type, and a false entry hides its nothing fields. An entry overrides default and the show_fields method of the type.

Constructors

ShowNothingFields(default::Bool, by_type::Dict{Symbol, Bool}) -> ShowNothingFieldsShowNothingFields(cfg::ShowNothingFields, name::Symbol,                  x::Union{Nothing, Bool}) -> ShowNothingFields

The second form copies cfg and changes one per-name entry. A Bool x sets the entry for name, and nothing removes it.

Related

source
PortfolioOptimisers.StringDistanceConfig — Type
struct StringDistanceConfig

Global configuration for the fuzzy "did you mean?" suggestions that did_you_mean adds to a "variable not in asset universe" message.

The value is immutable. The ScopedConfig STRING_DISTANCE holds it, and did_you_mean reads it. set_string_distance! sets the global default, and with_string_distance sets an override for one scope.

Fields

  • dist: The distance that scores a candidate name against the unknown name. The default is StringDistances.Levenshtein().
  • min_score: The minimum normalised similarity of a candidate that the function suggests. The default is 0.7. A value near 1 keeps only an almost exact match, and a value above 1 turns all suggestions off. The inner loop of a meta-optimiser needs that value. A cluster or a subset that does not hold an asset name is correct, the name is not a typo, and a suggestion for it is wrong.

Constructors

StringDistanceConfig(dist::StringDistances.StringDistance,                     min_score::Real) -> StringDistanceConfig

Validation

  • min_score > 0. StringDistances.findnearest never suggests a candidate with a score of exactly 0. But a threshold at or below 0 admits every candidate with a nonzero similarity, so 0 and a negative value have the same effect. Both let a guessed name that is near a real asset name get that real name back, and the suggestion then leaks data.

Related

source
PortfolioOptimisers.EquationLimits — Type
struct EquationLimits

Global resource caps for the parse of an equation. The caps stop a string whose expression tree is deep enough to use all of the stack.

A constraint string, a Black-Litterman view string and an entropy-pooling view string are untrusted input. They can come from a configuration file, a spreadsheet or a user interface. All of them go through parse_equation, which calls Meta.parse. eval_numeric_functions, collect_terms! and has_invalid_plus then walk the expression tree recursively. With no bound, a deeply nested string, such as tens of thousands of parentheses, gives a tree that uses all of the stack and stops the Julia process. These caps raise a Meta.ParseError long before that point.

The values are fixed defaults with a wide margin. A fixed value is the same on the build machine and on the deployment machine, and a value that precompilation measures is not. The value is immutable. The ScopedConfig EQUATION_LIMITS holds it. set_equation_limits! sets the global default, and with_equation_limits sets an override for one scope.

Fields

  • max_length: The maximum number of characters in an equation string that the library passes to Meta.parse. The default is 4096. A real linear constraint is short, so the bound is far above each real constraint and far below the nesting depth that can use all of the stack. A nesting depth d needs at least d characters, so the length cap also bounds the depth of the tree of the string form.
  • max_depth: The maximum depth of an expression tree that the Expr form of parse_equation accepts. The default is 256. That form gets a tree that is already built, and no length cap applies to it.

Constructors

EquationLimits(max_length::Integer, max_depth::Integer) -> EquationLimits

Validation

  • max_length > 0 && max_depth > 0.

Related

source
PortfolioOptimisers.set_default! — Function
set_default!(cfg::ScopedConfig{T}, x) -> Any

Replace the global default of a ScopedConfig with x in one atomic store, and return x. An active scoped override does not change.

Algorithm

  1. Convert x to T, the element type of cfg. A failed conversion raises before the function stores a value.
  2. Store the converted x into cfg.default in one atomic write. A concurrent reader sees the old value or the new value, and never a partial write.

Arguments

  • cfg: The configuration holder whose global default the function replaces.
  • x: The new value. The function converts it to the element type T of cfg.

Returns

  • x::T: The stored value.

Related

source
PortfolioOptimisers.with_config — Function
with_config(f, cfg::ScopedConfig{T}, x) -> Any

Run f() with the value of the ScopedConfig cfg set to x for the dynamic extent of the call. The previous value comes back when the call exits. The override is task-scoped, so it is thread-safe. A task spawned inside f gets the override, and a concurrent task outside f does not see it.

Algorithm

  1. Convert x to T, the element type of cfg.
  2. Bind cfg.scoped to the converted x, and run f() inside that binding. The binding ends when f returns and when f raises.

Arguments

  • f: The zero-argument function to run under the override.
  • cfg: The configuration holder to override.
  • x: The override value. The function converts it to the element type T of cfg.

Returns

  • The value that f() returns.

Related

source
PortfolioOptimisers.compact_show_budget — Function
compact_show_budget(io::IO) -> Any

Resolve the line budget above which @define_pretty_show collapses a nested struct.

The per-call :po_compact IO property overrides the global COMPACT_SHOW setting. Both take false for no collapse, true for the automatic budget, or an Int for a fixed budget. The automatic budget is max(8, displaysize(io)[1] - 4), so only a subtree that almost fills the terminal, or is taller than it, collapses.

Algorithm

  1. Read v, the :po_compact property of io.
  2. When io sets no such property, the collapse applies only to output with a limited height. Return nothing when :limit is false. Otherwise read the global COMPACT_SHOW setting into v.
  3. Return nothing when v is false, because that value turns the collapse off.
  4. Return Int(v) when v is an integer that is not a Bool, because that value is the fixed budget.
  5. Otherwise v is true, so return the automatic budget max(8, displaysize(io)[1] - 4).

Arguments

  • io: The stream whose :po_compact, :limit and :displaysize properties the function reads.

Returns

  • nothing when the collapse is off.
  • budget::Int in the other cases. It is the maximum number of lines that a nested struct can print before it collapses.

Related

source
PortfolioOptimisers.LogLimits — Type
struct LogLimits

Global caps on how many times one log site of the library repeats one message, one cap per log level.

A cross-validation, a walk-forward, an online selection and a meta-optimiser call the same verb many times, and each call that meets the same condition logs the same message again. So every log site of the library passes maxlog and _id to its log macro. maxlog comes from log_limit, which reads the cap of the level from this struct. _id comes from @log_id, which keys the count on the site and on the stable parts of the configuration. A logger counts each _id separately, so a different configuration at the same site logs again.

A cap of nothing sets no cap. Each site then keeps the limit that it states, and most sites state none. An integer cap replaces the limit of every site of its level. A cap is at least 1, so a cap never hides the first message of a site.

The value is immutable. The ScopedConfig LOG_LIMITS holds it. set_log_limits! sets the global default, and with_log_limits sets an override for one scope.

Fields

  • max_warn: Maximum number of times one @warn site logs one _id. The default is nothing, and each site then keeps its own limit.
  • max_info: Maximum number of times one @info site logs one _id. The default is nothing, and each site then keeps its own limit.
  • max_debug: Maximum number of times one @debug site logs one _id. The default is nothing, and each site then keeps its own limit.

Constructors

LogLimits(max_warn::Union{Nothing, Integer}, max_info::Union{Nothing, Integer},          max_debug::Union{Nothing, Integer}) -> LogLimitsLogLimits(;    max_warn::Union{Nothing, Integer} = nothing,    max_info::Union{Nothing, Integer} = nothing,    max_debug::Union{Nothing, Integer} = nothing) -> LogLimits

Each keyword sets the field of the same name.

Validation

  • Each cap is nothing or a positive integer that is not a Bool.

Related

source
PortfolioOptimisers.log_limit — Function
log_limit(level::Symbol) -> Union{Nothing, Int64}
log_limit(
    level::Symbol,
    site::Union{Nothing, Integer}
) -> Union{Nothing, Integer}

Give the maxlog of one log site. It is the cap of the level in LOG_LIMITS when that cap is set, and the limit that the site states in the other case.

Every log site of the library passes maxlog = log_limit(level), or maxlog = log_limit(level, n) when the site limits itself to n repeats. A maxlog of nothing sets no limit, because a logger limits a record only when its maxlog is an integer.

Algorithm

  1. Read the active value of LOG_LIMITS, and take the cap of level from it.
  2. Return the cap when it is an integer, and site when it is nothing.

Arguments

  • level: The level of the log macro, :warn, :info or :debug.
  • site: The limit that the site states, or nothing for no limit of its own.

Validation

  • level is :warn, :info or :debug. Each other symbol raises an ArgumentError.

Returns

  • maxlog::Union{Nothing, Integer}: The value to pass as the maxlog of the log macro.

Related

source
PortfolioOptimisers.@log_id — Macro
@log_id(parts...)

Give the _id of one log site. It is a hash of the site and of the stable parts of its message.

A logger counts the maxlog of each _id separately. The default _id of a log macro is one id for each source line. A limit on a line that logs different messages hides every later message, and a limit inside a shared helper such as strict_diagnostic hides the messages of every caller. An _id made from the full message never limits a message that holds a value that changes on each call. @log_id therefore keys the count on the module, the file and the line of its call, and on parts.

parts are the stable parts of the configuration that the message reports. They are the types of the estimators, and the keys and names that the caller wrote. The values that the data gives, such as an asset list, a count, an index, a float or a timestamp, stay in the message text and out of parts. The same condition at the same site under the same configuration counts as one message, and a different configuration logs again.

Algorithm

  1. At expansion, build a Symbol from the module, the file and the line of the call.
  2. At run time, return the hash of the tuple of that symbol and the values of parts.

Arguments

  • parts: The stable parts of the message, each evaluated in the scope of the call.

Returns

  • id::UInt: The value to pass as the _id of the log macro, or as the id of strict_diagnostic.

Related

source