DOCSEARCHbyAlgolia
Submit Docs

TabArena Benchmarking System

repository·main·Indexed 43 days ago

https://github.com/autogluon/tabarena

A living benchmarking system for tabular machine learning that evaluates individual models and AutoML systems. It features TabArena-v0.1 for curated IID datasets and BeyondArena for complex, non-IID scenarios including temporal and grouped splits. The ecosystem includes bencheval v0.1.0, an evaluation engine for generating Elo, win-rate, and rank leaderboards from raw performance results.

Tokens
39.1K
Snippets
94
Records
141
Agent score
56%

What's inside TabArena

  1. What is bencheval?

    main

    bencheval is a benchmark evaluation engine for tabular machine learning. It transforms raw per-task performance results into statistical leaderboards. It is the engine used by TabArena but is decoupled from the tabarena package, meaning it can be used with any benchmark that produces (method, task, metric_error) rows.

    Note: The API is experimental and subject to change. It is recommended to pin the version you use for evaluation, as tabarena releases are matched to specific bencheval releases.

  2. Manage memory estimates in ModelJob

    main
    When running GPU-based models, you can provide a fake_memory_for_estimates key within the resources dictionary of a ModelJob. This value (in GB) is used by the AutoGluon wrapper to calculate static memory estimates. This allows the scheduler to prevent Out-of-Memory (OOM) errors by controlling how many folds or tasks run in parallel based on the available VRAM of the target partition.

  3. How models and systems differ in TabArena

    main

    TabArena distinguishes between two types of entrants:

    1. Model: A single method that TabArena tunes using its own shared protocol. This includes shared preprocessing, a validation split provided by TabArena, a search space (up to 200 configurations), and bagging under the official validation protocol.

      • Code location: packages/tabarena/src/tabarena/models/<key>/
      • Testing: pytest -m models -k <Key>

    2. System: A complete pipeline that manages its own preprocessing, validation, tuning, and ensembling within a budget provided by TabArena. Examples include AutoML frameworks like AutoGluon or LLM agents.

      • Code location: packages/tabarena/src/tabarena/systems/<key>/
      • Testing: pytest tests/tabarena/systems/

  4. How TabArena benchmark plans work

    main

    A TabArenaBenchmarkPlan orchestrates the transition from high-level model/task requirements to executable cluster jobs.

    The Process:

    1. Composition: You define the plan using PathSetup, ResourcesSetup, SchedulerSetup, an arena context, and an experiment bundle.
    2. Resolution: setup_jobs() identifies which (experiment, dataset, fold, repeat) units need to run by checking cache hits and model constraints.
    3. Bundling: Work units are grouped into ModelJob sets and bundled into SLURM array tasks.
    4. Artifact Creation: The system writes a JobBatch artifact and a job JSON file.
    5. Execution: The system prints sbatch commands. Each array task runs a runner script that loads the JobBatch and executes the experiment, caching results in the specified workspace.

  5. Understand differences in C++ metric implementations

    main

    The custom C++ metric kernels in TabArena differ from standard reference implementations (like scikit-learn) in the following ways:

    1. No sample_weights support: To maximize computational efficiency, these kernels do not support sample weighting.
    2. Enhanced Precision: The kernels return values as double to provide higher precision compared to some standard implementations.

  6. Understand the vendored LimiX implementation

    main

    The LimiXModel in tabarena uses a vendored version of the LimiX library. Because LimiX is not a pip-installable package, tabarena includes the necessary inference-time source files directly within the repository under the tabarena.models.limix._vendor namespace.

    Key characteristics of the vendored code:

    • Namespace Rewriting: All internal imports have been rewritten from bare names (e.g., from model.X import ...) to the full path from tabarena.models.limix._vendor.{inference|model|utils}.X import ... to prevent collisions on sys.path.
    • Pickle Compatibility: The source has been modified to replace inline lambda functions in sklearn FunctionTransformer steps with named module-level helpers (like _nan_inf_to_nan and _add_epsilon). This ensures that fitted LimiXPredictor pipelines can be successfully pickled by AutoGluon.
    • Bug Fixes: A .squeeze() bug in SubSampleData.fit was removed to prevent crashes when processing datasets with exactly one feature.

  7. Understand the vendored TabSwift implementation

    main

    TabSwift is a tabular in-context-learning foundation model (similar to TabPFN or TabICL). Because it is not available as a standalone pip package, TabArena includes a vendored, self-contained version of the tabswift library from the TALENT benchmark framework.

    Key Characteristics:

    • Self-contained: The vendored library has no dependencies on the TALENT framework.
    • Dependencies: It relies on torch, numpy, scikit-learn, scipy, psutil, tqdm, huggingface_hub, and packaging.
    • Namespace: The library resides under the tabarena.models.tabswift._vendor namespace.
    • Workflow: While the upstream TALENT Method adapter is not vendored, its intended workflow (NaN handling, ordinal categorical encoding, target standardization, etc.) is implemented via the TabArena wrapper located at ../model.py.

  8. Optimize SLURM job efficiency with bundle_size

    main

    In GCPSlurmSetup, the bundle_size parameter determines how many model fits are grouped into a single SLURM array task.

    • Small bundle_size (e.g., 1): Each task is a single fit. This is safer for very slow models but can lead to high overhead and slower overall execution due to frequent node provisioning.
    • Large bundle_size (e.g., 5 or 10): Multiple fits are run within one array task. This amortizes the time spent waiting for nodes to boot (especially on 'flex' partitions) and is generally more efficient for faster models.
    • Escape Hatch: If a model is extremely slow and risks hitting SLURM time limits, use bundle_size_per_dataset to ensure contiguous runs of the same dataset stay within the allotted time.

  9. Configure model constraints and HPO search spaces

    main

    When running benchmarks with specific model requirements (e.g., memory limits or sample size constraints), use the custom_model_constraints parameter. This is particularly useful for Hyperparameter Optimization (HPO) runs where you might want to split a single benchmark into multiple jobs based on data scale.

    Key constraint keys include:

    • max_n_samples_train_per_fold
    • min_n_samples_train_per_fold
    • max_n_features
    • max_n_classes
    • regression_support (boolean)

    Example of splitting a benchmark into a 'small data' run and a 'large data' run using the same benchmark_name:

    # Run 1 — small data (<= 50k train), all configs
BenchmarkSetup(
    benchmark_name="tabpfnv25_hpo_14112025",
    models=[("RealTabPFN-v2.5", "all")],
    num_gpus=1,
    configs_per_job=10,
    custom_model_constraints={
        "REALTABPFN-V2.5": {
            "max_n_samples_train_per_fold": 50_000,
            "max_n_features": 2000,
            "max_n_classes": 10,
        }
    },
).setup_jobs()

# Run 2 — large data (50k-100k train), 25 configs, big GPUs
BenchmarkSetup(
    benchmark_name="tabpfnv25_hpo_14112025",
    models=[("RealTabPFN-v2.5", 25)],
    configs_per_job=1,
    custom_model_constraints={
        "REALTABPFN-V2.5": {
            "max_n_samples_train_per_fold": 100_000,
            "max_n_features": 2000,
            "max_n_classes": 10,
            "min_n_samples_train_per_fold": 50_001,
        }
    },
    slurm_gpu_partition="alldlc2_gpu-h200",
    fake_memory_for_estimates=140,
    parallel_benchmark_fix="_large_vram",
).setup_jobs()

  10. How SkyPilot managed jobs work in TabArena

    main

    When using SkyPilotSetup, tabflow_slurm manages jobs on GCP VMs that are isolated from the head node.

    The Process:

    1. Setup: setup_jobs() prints sky commands (e.g., sky jobs launch).
    2. Environment: The venv is rebuilt on the worker nodes from a frozen env.json manifest.
    3. Execution: Workers run python -P -m tabflow_slurm.sky_worker, which claims job bundles from a queue in a cloud bucket.
    4. Data Flow: Workers download datasets, fit models, and upload results back to the bucket.
    5. Evaluation: The eval subcommand syncs results from the bucket back to the local workspace to generate the leaderboard.

  11. Manage TabArena Artifacts and Cache

    main

    TabArena caches predictions, results, and leaderboards to allow reproduction and extension of analyses without re-running benchmarks.

    By default, artifacts download to ~/.cache/tabarena/. You can override this location using the TABARENA_CACHE environment variable.

    Warning: Raw data is approximately 100 GB per method type. Ensure TABARENA_CACHE points to a disk with sufficient capacity.

    Artifact Tiers

    TierContentsSize / method
    Raw dataPer-child test predictions, full metadata, system info~100 GB
    Processed dataMinimal data for HPO simulation, portfolios, leaderboards~10 GB
    ResultsPer-config / HPO DataFrames (test error, val error, train time, inference time)<1 MB
    LeaderboardsAggregated ELO, win-rate, average rank, improvability<1 MB
    Figures & PlotsGenerated from results and leaderboards—
    export TABARENA_CACHE="/path/to/large/disk/tabarena_cache"

  12. Configure GPU resource estimates with fake_memory_for_estimates

    main

    When defining a ModelJob for GPU tasks, use the fake_memory_for_estimates key within the resources dictionary. This value should represent the VRAM available on the GPU (in GB).

    Why use this? AutoGluon uses this figure to budget parallel bagging folds against the GPU memory rather than the node's system RAM. This ensures that tensor-heavy operations are scheduled to avoid Out-of-Memory (OOM) errors on the card.

    ModelJob(
    models=("Xiaomi-TabLDM", 0),
    name="gpu",
    resources={
        "num_gpus": 1,
        "fake_memory_for_estimates": 96,
    },
)