Absorb

Each table instance is stored in its own directory on disk. The directory contains the following files:

• table_metadata.json: Contains metadata including table_version, table_parameters, table_recipe, and update_schedule.
• Parquet files: The actual content of the table is stored in Parquet format.
• collection.log: A log file related to the collection process.

In absorb, data is organized using Table classes. A single Table class acts as a blueprint (a recipe) that can represent multiple specific table instances by providing different values for its table parameters.

• Table Class: The definition/blueprint (e.g., ChainFees).
• Table Instance: A specific realization of a class with a set of parameter values (e.g., chain_fees_top_100).
• Naming: The name of a table instance is automatically derived from the Table class and its parameters using the Table.name_template attribute.

The relationship between sources and tables follows this hierarchy:

1. A Source contains multiple Table classes.
2. A Table class contains one Table recipe.
3. A table recipe can contain multiple templates.
4. A Table class produces multiple Table instances.

Warning on Identification: Because different parameter sets can theoretically result in the same table name (if the variable is not included in the name_template), a table name alone is not always sufficient to uniquely identify a specific table instance. Use the full context of the parameters when necessary.

absorb uses the filesystem as its database. Datasets are stored as collections of Parquet files, either locally or in the cloud. absorb uses symlinks to organize these files within the ABSORB_ROOT tree.

The ABSORB_ROOT directory structure is as follows:

{ABSORB_ROOT}/
    datasets/
        <source>/
            tables/
                <datatype>/
                    {filename}.parquet
                table_metadata.json
            repos/
                {repo_name}/
    absorb_config.json

You can install absorb using uv. Choose the installation method that fits your needs:

Basic installation:

uv tool install paradigm_absorb

Install with all extras (recommended for full functionality):

uv tool install paradigm_absorb[test,datasources,interactive]

Install from source:

git clone git@github.com:paradigmxyz/absorb.git
uv tool install --editable .[test,datasources,interactive]

absorb uses a configuration file named absorb_config.json to specify which datasets to track.

Schema for absorb_config.json:

{
    'version': str,
    'tracked_tables': list[TableDict],
    'use_git': bool,
    'default_bucket': {
        'rclone_remote': str | None,
        'bucket_name': str | None,
        'path_prefix': str | None,
        'provider': str | None,
    },
}

The setup command allows you to control whether Absorb uses Git to track changes in your datasets.

• Enable/Disable Tracking: Use --enable-git or --disable-git to toggle the use_git setting in your configuration.
• Setup Tracking: If use_git is enabled in the config, running setup will trigger absorb.ops.setup_git(track_tables=datasets), which initializes Git tracking for the specified datasets.

If your local directory structure or configuration becomes out of sync, use the following flags with setup:

• --regenerate-metadata: Re-runs the table directory setup for the specified datasets (or all tracked datasets if none are provided). This is useful if table directories need to be recreated.
• --regenerate-config: Rebuilds the configuration file from defaults but attempts to migrate existing tracked_tables into the new configuration structure.

The cd subcommand allows the absorb CLI to change your current shell's working directory to a path returned by a command. Because a child process (the CLI) cannot change the working directory of its parent process (your shell), this command requires a shell function wrapper to work correctly.

To enable this functionality, you must add a specific shell snippet to your shell configuration file (e.g., ~/.bashrc, ~/.zshrc, or ~/.profile).

function absorb {
    local tempfile="$(mktemp -t tmp.XXXXXX)"
    command absorb "$@" --cd-destination-tempfile "$tempfile"
    if [[ -s "$tempfile" ]]; then
        cd "$(realpath $(cat "$tempfile"))"
    fi
    rm -f "$tempfile" 2>/dev/null
}

function abs() {
    absorb "$@"
}