Panel round trip: private API

Functions

PortfolioOptimisers.panel_frame_axes — Function

panel_frame_axes(pnl::AssetPanel, nx::Option{<:VecStr}, ts::Option{<:AbstractVector}) -> Tuple

Name the asset axis and the observation axis of an AssetPanel for a table.

The panel holds no asset names and no observation labels, because the PricesResult or the ReturnsResult that holds the panel names them. panel_dataframe and panel_manifest take them from the caller, and name each entry by its position when the caller gives nothing. Both functions name the axes through this function, so a table and its manifest name them in the same way.

Algorithm

The asset names are nx, or the positions "1", "2", … when nx is nothing.
The observation labels are ts, or the positions 1, 2, … when ts is nothing or the panel is static.
Check each length against the axes of the panel.

Arguments

pnl: The Asset Panel.
nx: The asset names of the universe of the panel, or nothing. See VecStr.
ts: The observation labels, or nothing. A static panel does not read it.

Validation

length(nx) is the length of the asset axis of the panel. Raises a DimensionMismatch.
length(ts) is the length of the observation axis of the panel, when the panel is time-varying. Raises a DimensionMismatch.

Returns

(nxa, tsa): The asset names and the observation labels.

Related

PortfolioOptimisers.panel_manifest_row! — Function

panel_manifest_row!(mf::DataFrames.DataFrame, kind::AbstractString, field, label;
                    group = missing, observed = missing,
                    placeholder = missing) -> nothing

Add one row to a manifest that panel_manifest writes.

A cell that the kind of the row does not use holds missing. A text reader writes missing as an empty cell, so a person who reads the manifest sees only the cells that the row uses.

Arguments

mf: The manifest. This function adds the row to it.
kind: The kind of the row.
field: The name of the Panel Field of the row, or missing.
label: The label of the row, or missing.
group: The group of a trailing-axis label, "grouped" or "ungrouped" on a "tensor" row, or missing.
observed: Whether the Panel Field carries an observed mask, or missing.
placeholder: Whether the Panel Field carries a placeholder mask, or missing.

Returns

nothing. mf holds the new row.

Related

PortfolioOptimisers.panel_manifest_rows! — Function

panel_manifest_rows!(mf::DataFrames.DataFrame, f::NumericPanelField, decode::Bool) -> nothing
panel_manifest_rows!(mf::DataFrames.DataFrame, f::CategoricalPanelField, decode::Bool) -> nothing
panel_manifest_rows!(mf::DataFrames.DataFrame, f::TensorPanelField, decode::Bool) -> nothing

Add the rows that describe one Panel Field to a manifest that panel_manifest writes.

The table that panel_dataframe writes gives the values of a Panel Field. These rows give the parts of the Panel Field that a table column cannot hold: its kind, whether it carries an observed mask and a placeholder mask, whether the column of a categorical Panel Field holds its levels or its codes, the order of its levels, and the axis name, the labels and the groups of a tensor Panel Field.

Algorithm

The method that Julia selects is the algorithm.

NumericPanelField: one "numeric" row.
CategoricalPanelField: one "categorical" row with the label "levels" when decode is true and "codes" when it is false, then one "level" row per level, in the order of the codes. A level that no cell holds keeps its row.
TensorPanelField: one "tensor" row with the axis name as its label, and the group "grouped" when the Panel Field carries groups and "ungrouped" when it does not. Then one "label" row per trailing-axis label with its group.

The first row of each Panel Field records in observed whether it carries an observed mask, and in placeholder whether it carries a placeholder mask.

The "categorical" row records decode, so that asset_panel reads the column as the table holds it. The "tensor" row records the groups flag because an empty group and no group are both an empty cell in a text file. The flag is a word, so the group column always holds one value that is not a number. A reader that guesses column types, such as CSV.jl, then reads the column as text, and the group "01" stays "01".

Arguments

mf: The manifest. This function adds the rows to it.
f: The Panel Field.
decode: The decode of the panel_dataframe call. Only the method of a categorical Panel Field reads it.

Returns

nothing. mf holds the new rows.

Related

PortfolioOptimisers.panel_text — Function

panel_text(x) -> String

Return a table cell as text, with missing as the empty string.

A reader that guesses the type of a column can give a cell as a number, a date or missing where the writer wrote text: CSV.jl reads the asset name "7" as the integer 7, and an empty string as missing. asset_panel compares every name and every label of a table and its manifest as text, so the two tables agree however the reader typed them.

Arguments

x: The cell.

Returns

s::String: string(x), or "" when x is missing.

Related

PortfolioOptimisers.panel_table_column — Function

panel_table_column(df::DataFrames.DataFrame, name::AbstractString) -> AbstractVector

Return the column of a table with the name name, and refuse a name that the table does not hold.

Arguments

df: The table.
name: The column name.

Validation

df holds a column named name. Raises a KeyError.

Returns

col::AbstractVector: The column, not a copy.

Related

PortfolioOptimisers.panel_table_cell — Function

panel_table_cell(col::AbstractVector, r::Integer, name::AbstractString)

Return the value in row r of the table column col, and refuse a missing value.

A Panel Field holds a finite value in every cell, so a table from panel_dataframe holds a value in every cell. A missing value is a change that the table got after the export.

Arguments

col: The column.
r: The row.
name: The column name, for the error message.

Validation

The cell is not missing. Raises an ArgumentError.

Returns

x: The cell.

Related

panel_table_read!

PortfolioOptimisers.panel_table_read! — Function

panel_table_read!(A::AbstractArray, lin::AbstractVector{<:Integer}, tab::NamedTuple, name::AbstractString, g) -> nothing
panel_table_read!(A::AbstractArray, lin::Nothing, tab::NamedTuple, name::AbstractString, g) -> nothing

Write the cells of one Panel Field column, or of one mask, from a table into the array A.

A has the shape of the column in the panel: assets on a static panel and observations × assets on a time-varying one. The caller fills it before the call, so a cell that the table does not hold keeps that value.

Algorithm

The method that Julia selects is the algorithm. lin says the layout.

The long layout gives lin, the linear index into A of the cell of each table row. Write g of the value of row r into A[lin[r]].
The wide layout gives nothing. The table holds one column per asset, named "<name>@<asset>", with one row per observation. Write g of each value into the cell of its observation and asset.

Arguments

A: The array. It is changed in place.
lin: The linear index of the cell of each row of a long table, or nothing for a wide table.
tab: The named tuple (; df, nx) of the table and the asset names of the manifest.
name: The column name in the long layout, and the part of the column name before "@" in the wide layout.
g: The function that converts a cell to the value that A holds.

Validation

The table holds each column that the call reads. Raises a KeyError. See panel_table_column.
No cell that the call reads is missing. Raises an ArgumentError. See panel_table_cell.

Returns

nothing. A holds the values.

Related

asset_panel
panel_table_column
panel_table_cell

PortfolioOptimisers.panel_table_eltype — Function

panel_table_eltype(lin::AbstractVector{<:Integer}, tab::NamedTuple, cols::VecStr) -> Type
panel_table_eltype(lin::Nothing, tab::NamedTuple, cols::VecStr) -> Type

Return the element type that holds every value of the table columns of one Panel Field.

The type is derived from the columns, and never forced: it is the promoted element type of the columns with Missing removed, which panel_table_types reads. A reader that gives an integer column and a floating-point column for two labels of one tensor Panel Field gets the floating-point type.

Arguments

lin: The linear indices of a long table, or nothing for a wide table. See panel_table_read!.
tab: The named tuple (; df, nx) of the table and the asset names of the manifest.
cols: The column names of the Panel Field in the long layout. See VecStr.

Validation

The table holds each column. Raises a KeyError. See panel_table_column.
The columns state a type that is not Missing. A reader such as CSV.jl types a column with no value as Missing, for example the columns of a long table with no row. Raises an ArgumentError.

Returns

T::Type: The element type.

Related

asset_panel
panel_table_read!
panel_table_types

PortfolioOptimisers.panel_table_types — Function

panel_table_types(lin::AbstractVector{<:Integer}, tab::NamedTuple, cols::VecStr) -> Type
panel_table_types(lin::Nothing, tab::NamedTuple, cols::VecStr) -> Type

Return the promoted element type of the table columns of one Panel Field, with Missing removed.

A column that holds no value states no type, so the answer is Union{} when no column holds a value. panel_table_eltype refuses that answer, and panel_code_eltype reads it as a column that states no code type.

Algorithm

The method that Julia selects is the algorithm.

The long layout reads the columns cols.
The wide layout reads the column "<col>@<asset>" of each column and each asset.

Arguments

lin: The linear indices of a long table, or nothing for a wide table. See panel_table_read!.
tab: The named tuple (; df, nx) of the table and the asset names of the manifest.
cols: The column names of the Panel Field in the long layout. See VecStr.

Validation

The table holds each column. Raises a KeyError. See panel_table_column.

Returns

T::Type: The promoted element type, or Union{}.

Related

panel_table_eltype
panel_code_eltype

PortfolioOptimisers.panel_code_eltype — Function

panel_code_eltype(s::NamedTuple, lin, tab::NamedTuple) -> Type

Return the element type of the codes of a categorical Panel Field that asset_panel reads from a table.

The type is derived from the table when the table holds it. A column of codes that a reader typed as an integer type states the type of the codes, so an Int32 code column gives Int32 codes. A column of levels holds text, and a code is then the position of its level in the levels. A position in Julia is an Int, so such codes are Int. A column of codes that holds text or no value states no code type either, and its codes are Int too.

Algorithm

When the manifest entry s records that the column holds the levels, return Int.
Otherwise read the type of the column with panel_table_types. Return it when it is an integer type, and Int when it is not, or when the column holds no value.

Arguments

s: The manifest entry of the categorical Panel Field, from panel_manifest_fields.
lin: The linear indices of a long table, or nothing for a wide table. See panel_table_read!.
tab: The named tuple (; df, nx) of the table and the asset names of the manifest.

Returns

T::Type{<:Integer}: The element type of the codes.

Related

panel_read_field
panel_code_reader
panel_table_types

PortfolioOptimisers.panel_table_mask — Function

panel_table_mask(lin, tab::NamedTuple, name::AbstractString, sz::Tuple) -> BitArray

Read one mask of the shape sz from a table, with false in every cell that the table does not hold.

Arguments

lin: The linear indices of a long table, or nothing for a wide table. See panel_table_read!.
tab: The named tuple (; df, nx) of the table and the asset names of the manifest.
name: The column name of the mask. See panel_table_read!.
sz: The shape of the mask.

Returns

M::BitArray: The mask.

Related

PortfolioOptimisers.panel_field_table_mask — Function

panel_field_table_mask(on::Bool, lin, tab::NamedTuple, name::AbstractString, sz::Tuple) -> Option{BitArray}
panel_field_table_mask(on::Bool, lin, tab::NamedTuple, cols::VecStr, sz::Tuple) -> Option{BitArray}

Read one mask of a Panel Field from a table, the observed mask or the placeholder mask, when its manifest entry records one.

The mask of the column "<column>" is the column "<column>::observed" or "<column>::placeholder". A numeric or a categorical Panel Field has one mask column, and its mask has the shape sz. A tensor Panel Field has one mask column per label, and its mask has a trailing label axis.

Arguments

on: Whether the manifest records the mask. With false, the answer is nothing.
lin: The linear indices of a long table, or nothing for a wide table. See panel_table_read!.
tab: The named tuple (; df, nx) of the table and the asset names of the manifest.
name, cols: The mask column of the Panel Field, or the mask column of each label.
sz: The shape of one column of the panel.

Returns

M::Option{BitArray}: The mask, or nothing.

Related

panel_read_field
panel_table_mask
Option
VecStr

PortfolioOptimisers.panel_read_field — Function

panel_read_field(s::NamedTuple, lin, tab::NamedTuple, sz::Tuple) -> AbstractPanelField

Read one Panel Field from a table, as its manifest entry s describes it.

A cell that a long table does not hold is outside the universe, and no consumer reads it. It gets the value zero, the first level of a categorical Panel Field, and false in the observed mask and in the placeholder mask.

Algorithm

"numeric": read the column named after the Panel Field into an array of the element type of the column, through panel_table_read!.
"categorical": read the same column into an array of the code type of panel_code_eltype. When the manifest records that the column holds the levels, find the code of the level in each cell. When it records the codes, each cell holds the code.
"tensor": read the column "<field>=<label>" of each label into its slice of the trailing axis.
Read the observed mask from the columns named "<column>::observed" when the manifest records one, and the placeholder mask from the columns named "<column>::placeholder" in the same way.

Arguments

s: The manifest entry of the Panel Field, from panel_manifest_fields.
lin: The linear indices of a long table, or nothing for a wide table. See panel_table_read!.
tab: The named tuple (; df, nx) of the table and the asset names of the manifest.
sz: The shape of one column of the panel: (assets,) or (observations, assets).

Validation

When the column holds the levels, each categorical cell is a level of the manifest. Raises a KeyError.
When the column holds the codes, each categorical cell is an integer. Raises an ArgumentError.
The constructor of the Panel Field checks the values. See NumericPanelField, CategoricalPanelField and TensorPanelField.

Returns

f::AbstractPanelField: The Panel Field.

Related

asset_panel
panel_table_read!
panel_table_eltype
panel_code_eltype
panel_table_mask
panel_field_table_mask

PortfolioOptimisers.panel_code_reader — Function

panel_code_reader(s::NamedTuple, ::Type{T}) where {T <: Integer}

Return the function that converts a cell of a categorical column to its integer code of the type T.

Algorithm

When the manifest entry s records that the column holds the levels, the cell holds a level. Return the position of the level, as text, in the levels of s.
When s records the codes, the cell holds the code. Return it as a T. A reader that gives the cell as text gets it parsed.

Arguments

s: The manifest entry of the categorical Panel Field, from panel_manifest_fields. Its decode tells whether the cell holds the level or the code.
T: The element type of the codes, from panel_code_eltype.

Validation

When the column holds the levels, the cell is a level of the manifest. Raises a KeyError.
When the column holds the codes, the cell is an integer that T holds. Raises an ArgumentError.

Returns

g: The function.

Related

panel_read_field
panel_code_eltype
panel_text

PortfolioOptimisers.panel_manifest_axes — Function

panel_manifest_axes(mf::DataFrames.DataFrame) -> NamedTuple

Read the shape, the observation labels and the asset names from a manifest that panel_manifest wrote.

Algorithm

Read the "panel" row. Its label is "static" or "time-varying".
Read the label of each "observation" row and each "asset" row as text, in order.

Arguments

mf: The manifest.

Validation

mf holds the columns kind and label. Raises a KeyError.
mf holds one "panel" row, with the label "static" or "time-varying". Raises an ArgumentError.
A static manifest holds no "observation" row. Raises an ArgumentError.
No asset name repeats. Raises an ArgumentError.

Returns

(; static, ts, nx): Whether the panel is static, the observation labels and the asset names, as text.

Related

PortfolioOptimisers.panel_manifest_fields — Function

panel_manifest_fields(mf::DataFrames.DataFrame) -> Vector{<:NamedTuple}

Read the entry of each Panel Field from a manifest that panel_manifest wrote.

An entry is the named tuple (; kind, name, observed, placeholder, decode, grouped, axis, labels, groups). labels holds the levels of a categorical Panel Field and the trailing-axis labels of a tensor Panel Field. decode belongs to a categorical Panel Field, and is false for the other kinds. axis, grouped and groups belong to a tensor Panel Field, and are "", false and empty for the other kinds.

Algorithm

Skip the "panel", "observation" and "asset" rows.
A "numeric", "categorical" or "tensor" row starts a new entry. The label of a "categorical" row gives decode, and the group of a "tensor" row gives grouped.
A "level" or a "label" row adds its label, and its group, to the entry of its Panel Field.

Arguments

mf: The manifest.

Validation

mf holds the six columns of a manifest. Raises a KeyError.
The label of a "categorical" row is "levels" or "codes", and the group of a "tensor" row is "grouped" or "ungrouped". Raises an ArgumentError.
A "level" row follows the "categorical" row of its Panel Field, and a "label" row follows the "tensor" row of its Panel Field. Raises an ArgumentError.
Each row has a kind that a manifest holds. Raises an ArgumentError.

Returns

specs::Vector{<:NamedTuple}: One entry per Panel Field, in the order of the manifest.

Related

asset_panel
panel_manifest
panel_manifest_rows!
panel_manifest_flag
panel_manifest_attach!
panel_text

PortfolioOptimisers.panel_manifest_flag — Function

panel_manifest_flag(r::Integer, k::AbstractString, label::AbstractString,
                    group::AbstractString) -> nothing

Check the flag that the first row of a categorical or a tensor Panel Field records as a word.

The label of a "categorical" row is "levels" or "codes": what the table column of the Panel Field holds. The group of a "tensor" row is "grouped" or "ungrouped": whether the Panel Field carries groups. A "numeric" row records no flag.

Arguments

r: The row of the manifest, for the error message.
k: The kind of the row.
label: The label of the row.
group: The group of the row.

Validation

The flag of a "categorical" or a "tensor" row is one of its two words. Raises an ArgumentError.

Returns

nothing.

Related

panel_manifest_fields
panel_manifest_rows!

PortfolioOptimisers.panel_manifest_attach! — Function

panel_manifest_attach!(specs::AbstractVector{<:NamedTuple}, r::Integer, k::AbstractString,
                       field::AbstractString, label::AbstractString,
                       group::AbstractString) -> nothing

Add the label, and the group, of a "level" or a "label" row of a manifest to the entry of its Panel Field.

The entry of the Panel Field is the last entry of specs, because panel_manifest writes the level rows and the label rows of a Panel Field after its first row.

Arguments

specs: The entries that panel_manifest_fields read so far. The last one gets the label.
r: The row of the manifest, for the error message.
k: The kind of the row.
field: The name of the Panel Field of the row.
label: The level or the trailing-axis label.
group: The group of a trailing-axis label, or "".

Validation

k is "level" or "label", and the last entry is the "categorical" or the "tensor" entry of the Panel Field field. Raises an ArgumentError.

Returns

nothing. The last entry of specs holds the label.

Related

panel_manifest_fields
panel_manifest_rows!

PortfolioOptimisers.panel_long_cells — Function

panel_long_cells(df::DataFrames.DataFrame, ax::NamedTuple) -> Vector{Int}

Find the cell of the panel that each row of a long table holds.

A long table from panel_dataframe holds one row per asset of a static panel, and one row per pair of observation and asset in the universe of a time-varying panel. Its "asset" column and, on a time-varying panel, its "observation" column name the cell. This function finds the position of each name in the manifest, compared as text, and returns the linear index of the cell in an assets or an observations × assets array.

Algorithm

Find the position of the asset of each row in ax.nx.
On a time-varying panel, find the position of the observation of each row in ax.ts, and combine the two positions into a linear index.
Check that no cell repeats, and that a static table holds every asset.

Arguments

df: The long table.
ax: The axes of the manifest, from panel_manifest_axes.

Validation

df holds the key columns. Raises a KeyError.
The manifest names no observation twice, when the panel is time-varying. Raises an ArgumentError.
The manifest names the asset, and the observation, of each row. Raises a KeyError.
No two rows hold one cell. Raises an ArgumentError.
A static table holds one row per asset. Raises an ArgumentError.

Returns

lin::Vector{Int}: The linear index of the cell of each row.

Related

asset_panel
panel_table_read!
panel_text

PortfolioOptimisers.panel_long_position — Function

panel_long_position(pos::AbstractDict, x, key::AbstractString) -> Int

Return the position of the name x, as text, on one axis of the manifest.

Arguments

pos: The position of each name on the axis.
x: The cell of the key column.
key: The name of the key column, for the error message.

Validation

pos holds the name. Raises a KeyError.

Returns

i::Int: The position.

Related

panel_long_cells
panel_text

PortfolioOptimisers.panel_table_cells — Function

panel_table_cells(df::DataFrames.DataFrame, ax::NamedTuple) -> Option{Vector{Int}}

Tell the layout of a table from panel_dataframe, and find where its rows go in the panel.

Only the long layout holds an "asset" column. A column of the wide layout is named "<name>@<asset>", so no column of it is named "asset".

Algorithm

The table holds an "asset" column: it is a long table. Return the cell of each row, from panel_long_cells.
Otherwise it is a wide table, whose rows are the observations in order. Check the row count and, on a time-varying panel, the labels of its "observation" column against the manifest, as text and in order. Return nothing.

Arguments

df: The table.
ax: The axes of the manifest, from panel_manifest_axes.

Validation

A wide table has one row on a static panel, and one row per observation on a time-varying one. Raises a DimensionMismatch.
The "observation" column of a wide table, when the table holds one, names the observations of the manifest in order. Raises an ArgumentError.
The rows of a long table name cells of the manifest, each once. See panel_long_cells.

Returns

lin::Option{Vector{Int}}: The linear index of the cell of each row of a long table, or nothing for a wide table. See panel_table_read!.

Related

PortfolioOptimisers.panel_active_mask — Function

panel_active_mask(lin::AbstractVector{<:Integer}, tab::NamedTuple, sz::Tuple) -> BitMatrix
panel_active_mask(lin::Nothing, tab::NamedTuple, sz::Tuple) -> BitMatrix

Read the active mask of a time-varying panel from a table.

Algorithm

The method that Julia selects is the algorithm.

A long table holds a row for each cell in the universe and no other row, so the active mask is true at the cell of each row.
A wide table holds the active mask in its "amsk@<asset>" columns. Read them with panel_table_mask.

Arguments

lin: The linear indices of a long table, or nothing for a wide table. See panel_table_read!.
tab: The named tuple (; df, nx) of the table and the asset names of the manifest.
sz: The shape (observations, assets) of the mask.

Validation

A wide table holds each "amsk@<asset>" column. Raises a KeyError. See panel_table_column.

Returns

amsk::BitMatrix: The active mask.

Related

PortfolioOptimisers.panel_manifest_select — Function

panel_manifest_select(specs::AbstractVector{<:NamedTuple}, fields::Nothing)
panel_manifest_select(specs::AbstractVector{<:NamedTuple}, fields::AbstractString)
panel_manifest_select(specs::AbstractVector{<:NamedTuple}, fields)

Keep the manifest entries of the Panel Fields that an asset_panel call names, in the order of the call.

Algorithm

The method that Julia selects is the algorithm.

fields is nothing: return every entry, in the order of the manifest.
fields is one string: return its entry, as a one-entry vector.
Otherwise: find the entry of each name with panel_manifest_entry, in the order of fields.

Arguments

specs: The entries, from panel_manifest_fields.
fields: One Panel Field name as a string, a collection of names, or nothing. An entry can be a String or a Symbol.

Validation

specs holds an entry for each name. Raises a KeyError. See panel_manifest_entry.

Returns

specs: The entries, in the order of the call.

Related

asset_panel
panel_manifest_entry

PortfolioOptimisers.panel_manifest_entry — Function

panel_manifest_entry(specs::AbstractVector{<:NamedTuple}, name::AbstractString) -> NamedTuple

Return the manifest entry of the Panel Field named name, and refuse a name that the manifest does not hold.

Arguments

specs: The entries, from panel_manifest_fields.
name: The name of the Panel Field.

Validation

specs holds an entry with the name. Raises a KeyError.

Returns

s::NamedTuple: The entry.

Related

asset_panel
panel_manifest_fields