GitHub

Panel round trip

Read a panel back from its tables

panel_dataframe writes the values and the masks of an AssetPanel as a table. panel_manifest writes the parts that a table column cannot hold: the axes, the kind of each field, the order of the levels, and the axis name, the labels and the groups of a tensor field. asset_panel reads the panel back from the two tables, after they go through any table format, for example CSV.jl or Arrow.jl.

PortfolioOptimisers.panel_manifest — Function
panel_manifest(pnl::AssetPanel; nx::Option{<:VecStr} = nothing,
               ts::Option{<:AbstractVector} = nothing, fields = nothing,
               assets = nothing, decode::Bool = true) -> DataFrames.DataFrame

Describe an AssetPanel in a table, so that asset_panel can read the panel back from a panel_dataframe table.

The table of panel_dataframe holds the values and the masks. It cannot hold the order of the levels of a categorical Panel Field, a level that no cell holds, whether a categorical column holds the levels or the codes, or the axis name and the groups of a tensor Panel Field. The long layout also drops an asset that is never in the universe and an observation with no asset in the universe. The manifest holds these parts. Write the two tables with a package that writes a DataFrame, for example CSV.jl or Arrow.jl, and give both to asset_panel to read the panel back.

Call this function with the nx, ts, fields, assets and decode of the panel_dataframe call, so that the two tables describe the same panel.

The manifest has six columns: kind, field, label, group, observed and placeholder. Its rows are, in order:

  • One "panel" row. Its label is "static" or "time-varying".
  • One "observation" row per observation of a time-varying panel. Its label is the observation label as text.
  • One "asset" row per selected asset. Its label is the asset name.
  • The rows of each selected Panel Field, from panel_manifest_rows!.

Algorithm

  1. Name the axes with panel_frame_axes, and find the positions of the selected assets with panel_frame_assets.
  2. Add the "panel" row, the "observation" rows and the "asset" rows.
  3. Find the Panel Fields with panel_frame_fields, and add the rows of each with panel_manifest_rows!.

Arguments

  • pnl: The Asset Panel.
  • nx: The asset names of the universe of the panel, or nothing to name each asset by its position. See VecStr.
  • ts: The observation labels, or nothing to name each observation by its position. A static panel has no observation axis and does not read it.
  • fields: One Panel Field name as a string, a collection of names, or nothing for all Panel Fields of the panel.
  • assets: One asset label as a string, a collection of labels, or nothing for all assets of the universe.
  • decode::Bool = true: Whether the categorical columns of the table hold the levels or the integer codes. The manifest records it, and asset_panel reads it.

Validation

  • length(nx) and length(ts) match the axes of the panel. Raises a DimensionMismatch. See panel_frame_axes.
  • The panel holds every named Panel Field, and nx holds every named asset. Raises a KeyError.
  • No asset is named twice in assets. Raises an ArgumentError.

Returns

  • mf::DataFrames.DataFrame: The manifest.

Examples

julia> pnl = AssetPanel(;                        pf = [CategoricalPanelField(; name = "sector",                                                    levels = ["Tech", "Energy", "Retail"],                                                    codes = [1, 2, 1])]);julia> panel_manifest(pnl; nx = ["A", "B", "C"])8×6 DataFrame Row │ kind         field    label    group    observed  placeholder     │ String       String?  String?  String?  Bool?     Bool?─────┼───────────────────────────────────────────────────────────────   1 │ panel        missing  static   missing   missing      missing   2 │ asset        missing  A        missing   missing      missing   3 │ asset        missing  B        missing   missing      missing   4 │ asset        missing  C        missing   missing      missing   5 │ categorical  sector   levels   missing     false        false   6 │ level        sector   Tech     missing   missing      missing   7 │ level        sector   Energy   missing   missing      missing   8 │ level        sector   Retail   missing   missing      missing

Related

source
PortfolioOptimisers.asset_panel — Method
asset_panel(df::DataFrames.DataFrame, mf::DataFrames.DataFrame;
            fields = nothing) -> AssetPanel

Read an AssetPanel from a table that panel_dataframe wrote and the manifest that panel_manifest wrote. This is the inverse of the export.

The two tables can go through any table format and come back, for example through CSV.jl or Arrow.jl, so a panel makes a round trip through a file with no file format of its own. Arrow.jl can map a file into memory, and a column selection before the read gives a subset of the Panel Fields.

The function reads the long and the wide layouts, and tells them apart by the "asset" column, which only the long layout holds. It does not read the single-field shape, which holds no mask.

  • Wide: the table holds every cell and both universe masks, so the panel comes back whole.
  • Long: the table holds only the rows where the active mask is true. The function makes the active mask from the rows that the table holds, and the manifest gives the axes. The value of a cell outside the universe does not come back: the cell gets zero, the first level of a categorical Panel Field, and an observed mask of false. No consumer reads a cell outside the universe, so this is lossless for every consumer.

The function compares names and labels as text, so a reader that types a column still matches the manifest. A reader that changes the text of a name does not match it: CSV.jl reads the asset name "007" as the integer 7. Read such a column as text. The manifest records whether each categorical column holds the levels or the codes, so the function reads the column as panel_dataframe wrote it.

The codes of a categorical column that holds the codes keep the integer type of the column, so Int32 codes come back as Int32 codes when the reader keeps the column type. A column that holds the levels carries no code type, and its codes are the positions of the levels, which are Int. See panel_code_eltype.

A static Panel Field that a time-varying panel lifts, see RepeatedLeading, comes back as an ordinary array with the same values.

Algorithm

  1. Read the axes and the Panel Field entries of the manifest with panel_manifest_axes and panel_manifest_fields. Keep the entries that fields names with panel_manifest_select.
  2. Tell the layout with panel_table_cells. In the long layout, it finds the cell of each row. In the wide layout, it checks the row count: one row on a static panel, one row per observation on a time-varying one.
  3. Read each Panel Field with panel_read_field.
  4. On a time-varying panel, read the masks. panel_active_mask reads the active mask. The estimation mask is the "emsk" column of a long table, and the "emsk@<asset>" columns of a wide one.
  5. Make the panel with the AssetPanel constructor, which checks it.

Arguments

  • df: The table, in the long or the wide layout of panel_dataframe.
  • mf: The manifest, from panel_manifest.
  • fields: One Panel Field name as a string, a collection of names, or nothing for all Panel Fields of the manifest.

Validation

  • The manifest is one that panel_manifest wrote. Raises an ArgumentError or a KeyError. See panel_manifest_axes and panel_manifest_fields.
  • The manifest holds every named Panel Field. Raises a KeyError.
  • The table holds each column that the manifest and fields name. Raises a KeyError.
  • A wide table has one row, or one row per observation. Raises a DimensionMismatch.
  • The "observation" column of a wide table names the observations of the manifest, as text and in order. Raises an ArgumentError. See panel_table_cells.
  • The rows of a long table name cells of the manifest, each once. See panel_long_cells.
  • No cell that the function reads is missing. Raises an ArgumentError.
  • The value columns of a Panel Field state a type. A long table with no row, read back by CSV.jl, types each column as Missing. Raises an ArgumentError. See panel_table_eltype.
  • The AssetPanel constructor and the constructor of each Panel Field check the result.

Returns

  • pnl::AssetPanel: The Asset Panel.

Examples

julia> pnl = AssetPanel(;                        pf = [NumericPanelField(; name = "mcap", vals = [1.0 2.0; 3.0 4.0]),                              CategoricalPanelField(; name = "sector",                                                    levels = ["Tech", "Energy", "Retail"],                                                    codes = [1 2; 1 1])], amsk = Bool[1 0; 1 1],                        emsk = Bool[1 0; 0 1]);julia> df = panel_dataframe(pnl; nx = ["A", "B"]);julia> mf = panel_manifest(pnl; nx = ["A", "B"]);julia> pnl2 = asset_panel(df, mf);julia> pnl2.amsk == pnl.amsk, pnl2.emsk == pnl.emsk(true, true)julia> panel_field(pnl2, "sector").levels3-element Vector{String}: "Tech" "Energy" "Retail"

Related

source