DOCSEARCHbyAlgolia
Submit Docs

OpenML-Python

repository·main·Indexed 43 days ago

https://github.com/openml/openml-python

A Python interface for the OpenML platform that enables users to download datasets, upload machine learning experiment results, and interact with machine learning tasks and benchmarking suites for open science collaboration. It supports Python 3.10 through 3.14 and provides tools for managing tasks, flows, setups, and runs.

Tokens
8.5K
Snippets
37
Records
52
Agent score
56%

What's inside OpenML

  1. Understand the OpenML Python Docker image structure

    main

    The image is built on python:3 and includes specific directories for development:

    • /openml: Contains the openml-python repository. If a branch is specified during execution, this directory is set to the HEAD of that branch.
    • /openml/venv/: A virtual environment containing openml-python dependencies. When running test or doc, these dependencies are automatically updated based on the setup.py of the target branch or the mounted /code directory.
    • /scripts/startup.sh: The container entrypoint that handles the logic for test, doc, and standard Python execution.

    To view the README included in a specific image tag:

    docker run --entrypoint=/bin/cat openml/openml-python:TAG readme.md

  2. Understand OpenML core concepts

    main

    To use openml-python effectively, you must understand the hierarchy of machine learning research entities it manages:

    • Task: A combination of a dataset, a specific data split, and an evaluation metric.
    • Setup: The hyperparameter settings used for an algorithm.
    • Flow: The specific algorithm being used.
    • Run: A single machine learning experiment that describes the performance of a flow (algorithm) with a specific setup (hyperparameters) on a task.

  4. Run unit tests using the Docker image

    main

    To run unit tests, pass test as the first argument to the container. You must specify a target repository, such as a branch name from the official repository.

    Example: testing the develop branch:

    docker run openml/openml-python test develop

    To test a local version of the repository, mount your local directory to /code:

    Linux:

    docker run --mount type=bind,source="/path/to/local/openml-python",destination="/code" openml/openml-python test

    Windows:

    docker run --mount type=bind,source="E:\path\to\local\openml-python",destination="/code" openml/openml-python test

  5. Run tests in the OpenML Python SDK

    main

    The SDK uses pytest markers to categorize tests. You can use these markers to filter your test execution.

    Available Markers:

    • sklearn: Tests requiring scikit-learn. Skipped if the library is missing.
    • production_server: Tests that interact with the live OpenML server (real API calls).
    • test_server: Tests requiring the OpenML test server environment.

    Execution Commands:

    • Run all tests: pytest
    • Run only scikit-learn tests: pytest -m sklearn
    • Run all tests except production server tests: pytest -m "not production_server"

    Admin Privilege Tests: Some tests require administrative privileges on the test server. To enable them, provide an admin API key via the OPENML_TEST_SERVER_ADMIN_KEY environment variable.

    • Linux/macOS:
      export OPENML_TEST_SERVER_ADMIN_KEY="admin-key"
    • Windows (PowerShell):
      $env:OPENML_TEST_SERVER_ADMIN_KEY = "admin-key"
    # Run all tests
pytest

# Run specific subset
pytest -m sklearn

# Exclude production tests
pytest -m "not production_server"

  6. Configure openml-python settings

    main

    The openml-python configuration file is located in the user's home directory under .config/openml/config (following the XDG Base Directory Specification). The file uses a key = value format with one pair per line.

    To inspect your current configuration and locate the file path, use the following CLI command:

    openml configure none
    openml configure none

  7. Set up the OpenML Python SDK for development

    main

    To contribute to the openml-python client, follow these steps to create a development environment:

    1. Clone the repository:
      git clone https://github.com/openml/openml-python
cd openml-python
    2. Initialize a virtual environment (e.g., using Conda):
      conda create -n openml-python-dev python=3.12
conda activate openml-python-dev
    3. Install dependencies in editable mode (including dev and docs):
      python -m pip install -e ".[dev,docs]"
    4. Install pre-commit hooks:
      pre-commit install
pre-commit run --all-files
    git clone https://github.com/openml/openml-python
cd openml-python
conda create -n openml-python-dev python=3.12
conda activate openml-python-dev
python -m pip install -e ".[dev,docs]"
pre-commit install
pre-commit run --all-files

  9. Run Python with pre-installed OpenML-Python via Docker

    main

    You can use the openml/openml-python Docker image to run a Python environment that already has openml-python installed. This is useful for quick testing or running scripts in an isolated environment.

    To start an interactive session:

    docker run -it openml/openml-python

    To run a specific Python command:

    docker run openml/openml-python -c "import openml; print(openml.__version__)"

    To run a local script, you must mount the file into the /openml directory inside the container:

    docker run -v PATH/TO/FILE:/openml/MY_SCRIPT.py openml/openml-python MY_SCRIPT.py

  10. Set up the API v2 (Python/FastAPI) Backend

    main

    To set up the FastAPI-based API v2 locally, use Docker and the server-api repository.

    Prerequisites:

    • Docker Desktop (running)
    • Git

    Steps:

    1. Clone the repository:
      git clone https://github.com/openml/server-api
cd server-api
    2. Launch the services:
      docker compose --profile all up

    Verification:

    • Task Endpoint: http://localhost:8001/tasks/31
    • Swagger UI: http://localhost:8001/docs
    git clone https://github.com/openml/server-api
cd server-api
docker compose --profile all up

  11. Structure a standalone OpenML extension repository

    main

    When creating a new extension, it should be hosted in a standalone repository that is compatible with an existing openml-python installation. The recommended directory structure for your public GitHub repository is:

    [repo name]
|-- [extension name]
    |-- __init__.py
    |-- extension.py
    |-- config.py (optional)