Scoped config: private API
Six settings apply to the whole package. Each is in a ScopedConfig, which many threads can read at once.
COMPACT_SHOWsets how far the printout collapses a large nested type.SHOW_NOTHING_FIELDSsets whether the printout shows a field that holdsnothing.STRING_DISTANCEsets how close a misspelled name must be for an error to suggest a correction.EQUATION_LIMITSsets the largest equation that the parser accepts.RESOURCE_LIMITSsets the largest draw counts, subset counts and grids that the sampling and sweep estimators accept.LOG_LIMITSsets how many times one log site repeats one message, for each log level.
The set_*! and with_* functions that change these settings are public, see Scoped config. You can also set the defaults of a project at load time through Preferences.jl.
PortfolioOptimisers.RESOURCE_LIMITS — Constant
RESOURCE_LIMITS = ScopedConfig(ResourceLimits())Default global resource caps for the estimators that sample or sweep. The caps stop a configuration value that is large enough to use all of the memory, or to keep the process busy for days. Each estimator reads RESOURCE_LIMITS[], and passes the cap to assert_resource_cap. The "max_n_sim", "max_n_subsets", "max_frontier", "max_bins", "max_hop_count", "max_search_grid" and "max_ep_grid" preferences can set the defaults for a project at load time, through apply_preferences!.
Related
PortfolioOptimisers.COMPACT_SHOW — Constant
Global control for the collapse of large nested structs in the output of @define_pretty_show.
It holds one of these values:
false: The collapse is off, and a nested struct always prints in full.true: The collapse is on, with a line budget that the size of the terminal sets.n::Int: The collapse is on, with a fixed line budget ofn.
It is a ScopedConfig. set_compact_show! sets the global default, and with_compact_show sets an override for one scope. compact_show_budget reads it together with the per-call :po_compact IO property. The "compact_show" preference can set the default for a project at load time, through apply_preferences!.
PortfolioOptimisers.SHOW_NOTHING_FIELDS — Constant
SHOW_NOTHING_FIELDS = ScopedConfig(ShowNothingFields(false, Dict{Symbol, Bool}()))Global control for whether @define_pretty_show renders a field that holds nothing.
The package default hides such a field, because a reader at the REPL wants the fields that hold a value. The documentation build shows these fields with set_show_nothing_fields!(true), next to its set_compact_show!(false) call, so that a rendered docstring shows the complete type. pretty_show_fields reads SHOW_NOTHING_FIELDS[]. The "show_nothing_fields" and "show_nothing_fields_by_type" preferences can set the default for a project at load time, through apply_preferences!.
Related
PortfolioOptimisers.STRING_DISTANCE — Constant
STRING_DISTANCE = ScopedConfig(StringDistanceConfig(StringDistances.Levenshtein(), 0.7))Default string distance configuration for the fuzzy "did you mean?" suggestions that did_you_mean adds to a "variable not in asset universe" message. did_you_mean reads STRING_DISTANCE[]. The "suggestion_distance" and "suggestion_min_score" preferences can set the defaults for a project at load time, through apply_preferences!.
Related
PortfolioOptimisers.EQUATION_LIMITS — Constant
EQUATION_LIMITS = ScopedConfig(EquationLimits(4096, 256))Default global resource caps for the parse of an equation. The caps stop a string whose expression tree is deep enough to use all of the stack. parse_equation and each other parser of an equation string read EQUATION_LIMITS[]. The "equation_max_length" and "equation_max_depth" preferences can set the defaults for a project at load time, through apply_preferences!.
Related
PortfolioOptimisers.ScopedConfig — Type
mutable struct ScopedConfig{T}Thread-safe holder for a package-level configuration value. It holds a global default that persists, and a task-scoped override.
A read is cfg[]. Inside a with_* block it returns the innermost active scoped override, and in other code it returns the global default. The default is an @atomic field, and a set_*! call replaces the whole value in one atomic store. A concurrent reader thus never sees a configuration that is half written. The FLoops.@floop loops inside the meta-optimisers are such readers. The scoped override is a Base.ScopedValues.ScopedValue. A task spawned inside the scope gets the override, the override goes away when the scope exits, and a concurrent task outside the scope does not see it.
Each configuration of this kind holds an immutable struct or a bits value. A change to one setting builds a new value and stores it in place of the old value. No call mutates the value in place.
COMPACT_SHOW, SHOW_NOTHING_FIELDS, STRING_DISTANCE, EQUATION_LIMITS, RESOURCE_LIMITS and LOG_LIMITS are of this type. A set_*! function sets the global default of each, and a with_* function sets a scoped override. Preferences.jl sets a default for each project at load time, through apply_preferences!.
Related
PortfolioOptimisers.ResourceLimits — Type
struct ResourceLimitsGlobal resource caps for the estimators that sample or sweep. The caps stop a configuration value that is large enough to use all of the memory, or to keep the process busy for days.
A draw count, a subset count, the length of a frontier sweep and a histogram bin count are untrusted configuration integers. They can come from a configuration file, a tuning grid or a user interface. Each one multiplies the size of an allocation, and a subset count or a frontier length multiplies the number of optimisations. The constructors of these estimators bound the values only from below, with n_sim > 0, n_subsets >= 2, N > 0 and bins > 0. A value that is much too large, from an extra digit or a sweep with the wrong scale, passes these checks. The out-of-memory killer then stops the process, and the caller does not get a message. These caps raise a DomainError at the point where the library resolves the value.
Each allocation or loop that a value drives has its own cap, and the name of the cap follows the field that it checks. The caps are not shared, because a linear cap cannot bound a quadratic allocation. The bins × bins histogram has its own max_bins, and does not share the linear cap on draws.
The defaults are fixed values far above real use. A cap turns a kill by the out-of-memory killer into an error with a type. It does not judge the size that the caller chose. The value is immutable. The ScopedConfig RESOURCE_LIMITS holds it. set_resource_limits! sets the global default, and with_resource_limits sets an override for one scope. Use the keyword constructor ResourceLimits(; …). The seven caps have one type and four of them have the value 100_000, so a positional call can easily put a value in the wrong place.
Fields
max_n_sim: The maximum number of Monte-Carlo or bootstrap draws,n_sim, thatNormalUncertaintySetandARCHUncertaintySetaccept. The default is1_000_000. Each draw stores anN × Ncovariance, so the array holdsN² · n_simelements. At 20 assets the default cap permits a request of 3.2 GB, and the defaultn_simis3_000. This cap bounds memory.
max_n_subsets: The maximum number of resampled asset subsets,n_subsets, thatSubsetResamplingandMultipleRandomisedaccept. The default is100_000. This cap bounds compute much more than memory, because each subset runs a full inner optimisation. The cap is far above a real sweep, where the default is2. It is far below a value that keeps a session busy for days.
max_frontier: The maximum number of points of an efficient-frontier sweep that theFrontieralgorithm ofMeanRiskandNearOptimalCenteringaccepts. The default is100_000. Each point runs a full inneroptimise_JuMP_model!solve, so this cap bounds compute, and it has the same value asmax_n_subsets. The default ofFrontierisN = 20. The library checks this cap at two points. The constructor ofFrontiercaps theNof one bound. At Model Assembly,assert_frontier_sweep_capcaps the product over every swept return term and every swept risk measure. The sweep is anIterators.product, sokbounds ofNpoints costN^ksolves.
max_bins: The maximum number of histogram bins thatMutualInfoCovarianceandVariationInfoDistanceaccept. The default is10_000. The joint histogram is abins × binsweights matrix for each pair of assets, so this cap bounds a quadratic allocation of memory. A histogram of10_000²cells takes about 800 MB. That size is below the limit of the out-of-memory killer, and far above the 50 bins, approximately, that a real binning gives.
max_hop_count: The maximum hop count,n, thatHopCountaccepts. The default is100_000. Three readers sumA^ioveri in 0:n, so the cost is linear inn, atN³floating point operations for each power. A largenkeeps the session busy on compute, and not on memory. This cap bounds compute, and it has the same value asmax_n_subsets. The default hop count isn = 1. The constructor ofHopCountreads the cap, andresolve_separationalso builds aHopCountfrom the value that a rule computes. One check thus applies to a stated value and to a computed value.
max_search_grid: The maximum number of candidates in a search grid thatGridSearchCrossValidationandRandomisedSearchCrossValidationaccept. The default is100_000. Each candidate runs a full cross-validated fit, so this cap bounds compute, asmax_n_subsetsdoes.collectbuilds the grid from anIterators.product, sokparameters ofNvalues giveN^kcandidates.assert_search_grid_capchecks the cap on that product where the grid forms, because a check on each parameter cannot see the product.
max_ep_grid: The maximum number of grid points,K, thatGridTailViewaccepts. The default is10_000. Each grid point is one binary variable of the mixed-integer program that an upper-bound or equality view builds. It is also one dense row over theTposterior probabilities. The memory is linear inK, atK · Tcoefficients, and the compute is a branch-and-bound tree overKbinary variables. The cap is far below the compute caps that bound a plain solve, and far above the defaultK = 11.
Constructors
ResourceLimits(max_n_sim::Integer, max_n_subsets::Integer, max_frontier::Integer, max_bins::Integer, max_hop_count::Integer, max_search_grid::Integer, max_ep_grid::Integer) -> ResourceLimitsResourceLimits(; max_n_sim::Integer = 1_000_000, max_n_subsets::Integer = 100_000, max_frontier::Integer = 100_000, max_bins::Integer = 10_000, max_hop_count::Integer = 100_000, max_search_grid::Integer = 100_000, max_ep_grid::Integer = 10_000) -> ResourceLimitsEach keyword sets the field of the same name.
Validation
max_n_sim > 0 && max_n_subsets > 0 && max_frontier > 0 && max_bins > 0 && max_hop_count > 0 && max_search_grid > 0 && max_ep_grid > 0.
Related
PortfolioOptimisers.ShowNothingFields — Type
struct ShowNothingFieldsConfiguration for whether @define_pretty_show renders a field that holds nothing.
The ScopedConfig SHOW_NOTHING_FIELDS holds it, and pretty_show_fields reads it. set_show_nothing_fields! sets the global default, and with_show_nothing_fields sets an override for one scope. No call mutates a value. A change builds a new value with a copy of by_type, and stores it in place of the old value. A reader thus never sees a table that is half written.
Fields
default: Whether a field that holdsnothingprints, for a type that no per-name entry names. The package default isfalse, which hides anothingfield.
by_type: The per-name overrides. The key is the name of a type without its module. Atrueentry prints every declared field of that type, and afalseentry hides itsnothingfields. An entry overridesdefaultand theshow_fieldsmethod of the type.
Constructors
ShowNothingFields(default::Bool, by_type::Dict{Symbol, Bool}) -> ShowNothingFieldsShowNothingFields(cfg::ShowNothingFields, name::Symbol, x::Union{Nothing, Bool}) -> ShowNothingFieldsThe second form copies cfg and changes one per-name entry. A Bool x sets the entry for name, and nothing removes it.
Related
PortfolioOptimisers.StringDistanceConfig — Type
struct StringDistanceConfigGlobal configuration for the fuzzy "did you mean?" suggestions that did_you_mean adds to a "variable not in asset universe" message.
The value is immutable. The ScopedConfig STRING_DISTANCE holds it, and did_you_mean reads it. set_string_distance! sets the global default, and with_string_distance sets an override for one scope.
Fields
dist: The distance that scores a candidate name against the unknown name. The default isStringDistances.Levenshtein().
min_score: The minimum normalised similarity of a candidate that the function suggests. The default is0.7. A value near1keeps only an almost exact match, and a value above1turns all suggestions off. The inner loop of a meta-optimiser needs that value. A cluster or a subset that does not hold an asset name is correct, the name is not a typo, and a suggestion for it is wrong.
Constructors
StringDistanceConfig(dist::StringDistances.StringDistance, min_score::Real) -> StringDistanceConfigValidation
min_score > 0.StringDistances.findnearestnever suggests a candidate with a score of exactly0. But a threshold at or below0admits every candidate with a nonzero similarity, so0and a negative value have the same effect. Both let a guessed name that is near a real asset name get that real name back, and the suggestion then leaks data.
Related
PortfolioOptimisers.EquationLimits — Type
struct EquationLimitsGlobal resource caps for the parse of an equation. The caps stop a string whose expression tree is deep enough to use all of the stack.
A constraint string, a Black-Litterman view string and an entropy-pooling view string are untrusted input. They can come from a configuration file, a spreadsheet or a user interface. All of them go through parse_equation, which calls Meta.parse. eval_numeric_functions, collect_terms! and has_invalid_plus then walk the expression tree recursively. With no bound, a deeply nested string, such as tens of thousands of parentheses, gives a tree that uses all of the stack and stops the Julia process. These caps raise a Meta.ParseError long before that point.
The values are fixed defaults with a wide margin. A fixed value is the same on the build machine and on the deployment machine, and a value that precompilation measures is not. The value is immutable. The ScopedConfig EQUATION_LIMITS holds it. set_equation_limits! sets the global default, and with_equation_limits sets an override for one scope.
Fields
max_length: The maximum number of characters in an equation string that the library passes toMeta.parse. The default is4096. A real linear constraint is short, so the bound is far above each real constraint and far below the nesting depth that can use all of the stack. A nesting depthdneeds at leastdcharacters, so the length cap also bounds the depth of the tree of the string form.
max_depth: The maximum depth of an expression tree that theExprform ofparse_equationaccepts. The default is256. That form gets a tree that is already built, and no length cap applies to it.
Constructors
EquationLimits(max_length::Integer, max_depth::Integer) -> EquationLimitsValidation
max_length > 0 && max_depth > 0.
Related
PortfolioOptimisers.set_default! — Function
set_default!(cfg::ScopedConfig{T}, x) -> Any
Replace the global default of a ScopedConfig with x in one atomic store, and return x. An active scoped override does not change.
Algorithm
- Convert
xtoT, the element type ofcfg. A failed conversion raises before the function stores a value. - Store the converted
xintocfg.defaultin one atomic write. A concurrent reader sees the old value or the new value, and never a partial write.
Arguments
cfg: The configuration holder whose global default the function replaces.x: The new value. The function converts it to the element typeTofcfg.
Returns
x::T: The stored value.
Related
PortfolioOptimisers.with_config — Function
with_config(f, cfg::ScopedConfig{T}, x) -> Any
Run f() with the value of the ScopedConfig cfg set to x for the dynamic extent of the call. The previous value comes back when the call exits. The override is task-scoped, so it is thread-safe. A task spawned inside f gets the override, and a concurrent task outside f does not see it.
Algorithm
- Convert
xtoT, the element type ofcfg. - Bind
cfg.scopedto the convertedx, and runf()inside that binding. The binding ends whenfreturns and whenfraises.
Arguments
f: The zero-argument function to run under the override.cfg: The configuration holder to override.x: The override value. The function converts it to the element typeTofcfg.
Returns
- The value that
f()returns.
Related
PortfolioOptimisers.compact_show_budget — Function
compact_show_budget(io::IO) -> Any
Resolve the line budget above which @define_pretty_show collapses a nested struct.
The per-call :po_compact IO property overrides the global COMPACT_SHOW setting. Both take false for no collapse, true for the automatic budget, or an Int for a fixed budget. The automatic budget is max(8, displaysize(io)[1] - 4), so only a subtree that almost fills the terminal, or is taller than it, collapses.
Algorithm
- Read
v, the:po_compactproperty ofio. - When
iosets no such property, the collapse applies only to output with a limited height. Returnnothingwhen:limitisfalse. Otherwise read the globalCOMPACT_SHOWsetting intov. - Return
nothingwhenvisfalse, because that value turns the collapse off. - Return
Int(v)whenvis an integer that is not aBool, because that value is the fixed budget. - Otherwise
vistrue, so return the automatic budgetmax(8, displaysize(io)[1] - 4).
Arguments
io: The stream whose:po_compact,:limitand:displaysizeproperties the function reads.
Returns
nothingwhen the collapse is off.budget::Intin the other cases. It is the maximum number of lines that a nested struct can print before it collapses.
Related
PortfolioOptimisers.LOG_LIMITS — Constant
LOG_LIMITS = ScopedConfig(LogLimits())Default global caps on the repeats of one log site, one cap for each log level. A log site reads LOG_LIMITS[] through log_limit. The "max_warn", "max_info" and "max_debug" preferences can set the defaults for a project at load time, through apply_preferences!.
Related
PortfolioOptimisers.LogLimits — Type
struct LogLimitsGlobal caps on how many times one log site of the library repeats one message, one cap per log level.
A cross-validation, a walk-forward, an online selection and a meta-optimiser call the same verb many times, and each call that meets the same condition logs the same message again. So every log site of the library passes maxlog and _id to its log macro. maxlog comes from log_limit, which reads the cap of the level from this struct. _id comes from @log_id, which keys the count on the site and on the stable parts of the configuration. A logger counts each _id separately, so a different configuration at the same site logs again.
A cap of nothing sets no cap. Each site then keeps the limit that it states, and most sites state none. An integer cap replaces the limit of every site of its level. A cap is at least 1, so a cap never hides the first message of a site.
The value is immutable. The ScopedConfig LOG_LIMITS holds it. set_log_limits! sets the global default, and with_log_limits sets an override for one scope.
Fields
max_warn: Maximum number of times one@warnsite logs one_id. The default isnothing, and each site then keeps its own limit.
max_info: Maximum number of times one@infosite logs one_id. The default isnothing, and each site then keeps its own limit.
max_debug: Maximum number of times one@debugsite logs one_id. The default isnothing, and each site then keeps its own limit.
Constructors
LogLimits(max_warn::Union{Nothing, Integer}, max_info::Union{Nothing, Integer}, max_debug::Union{Nothing, Integer}) -> LogLimitsLogLimits(; max_warn::Union{Nothing, Integer} = nothing, max_info::Union{Nothing, Integer} = nothing, max_debug::Union{Nothing, Integer} = nothing) -> LogLimitsEach keyword sets the field of the same name.
Validation
- Each cap is
nothingor a positive integer that is not aBool.
Related
PortfolioOptimisers.log_limit — Function
log_limit(level::Symbol) -> Union{Nothing, Int64}
log_limit(
level::Symbol,
site::Union{Nothing, Integer}
) -> Union{Nothing, Integer}
Give the maxlog of one log site. It is the cap of the level in LOG_LIMITS when that cap is set, and the limit that the site states in the other case.
Every log site of the library passes maxlog = log_limit(level), or maxlog = log_limit(level, n) when the site limits itself to n repeats. A maxlog of nothing sets no limit, because a logger limits a record only when its maxlog is an integer.
Algorithm
- Read the active value of
LOG_LIMITS, and take the cap oflevelfrom it. - Return the cap when it is an integer, and
sitewhen it isnothing.
Arguments
level: The level of the log macro,:warn,:infoor:debug.site: The limit that the site states, ornothingfor no limit of its own.
Validation
levelis:warn,:infoor:debug. Each other symbol raises anArgumentError.
Returns
maxlog::Union{Nothing, Integer}: The value to pass as themaxlogof the log macro.
Related
PortfolioOptimisers.@log_id — Macro
@log_id(parts...)Give the _id of one log site. It is a hash of the site and of the stable parts of its message.
A logger counts the maxlog of each _id separately. The default _id of a log macro is one id for each source line. A limit on a line that logs different messages hides every later message, and a limit inside a shared helper such as strict_diagnostic hides the messages of every caller. An _id made from the full message never limits a message that holds a value that changes on each call. @log_id therefore keys the count on the module, the file and the line of its call, and on parts.
parts are the stable parts of the configuration that the message reports. They are the types of the estimators, and the keys and names that the caller wrote. The values that the data gives, such as an asset list, a count, an index, a float or a timestamp, stay in the message text and out of parts. The same condition at the same site under the same configuration counts as one message, and a different configuration logs again.
Algorithm
- At expansion, build a
Symbolfrom the module, the file and the line of the call. - At run time, return the hash of the tuple of that symbol and the values of
parts.
Arguments
parts: The stable parts of the message, each evaluated in the scope of the call.
Returns
id::UInt: The value to pass as the_idof the log macro, or as theidofstrict_diagnostic.
Related