DOCSEARCHbyAlgolia
Submit Docs

repo2docker Documentation

repository·main·Indexed 23 days ago

https://github.com/jupyterhub/repo2docker

repo2docker turns code repositories into Jupyter-enabled Docker images by automatically detecting configuration files (such as requirements.txt, environment.yml, or pyproject.toml) to build the environment. It features a Buildpack system for environment assembly, ContentProviders for repository retrieval, and support for custom lifecycle scripts via postBuild and start.

Tokens
15.7K
Snippets
30
Records
104
Agent score
82%

What's inside repo2docker

  1. What is repo2docker and how does it work?

    main

    repo2docker is a tool used to reproducibly build and run user environment container images for interactive computing and data workflows from source code repositories. It is the underlying tool used by JupyterHub and BinderHub to build images.

    Core Workflow: When you run repo2docker <source-repository>, the tool:

    1. Inspects the repository for configuration files to determine the required environment.
    2. Builds a container image based on those configuration files.
    3. Runs the image (optional) to allow interactive exploration via interfaces like Jupyter notebooks or RStudio.
    4. Pushes the image to a container registry (optional) for remote access.

    Key Capabilities:

    • Build reproducible data science environments using community-standard configuration files.
    • Deploy environments to services like JupyterHub or Binder.
    • Support for various repository hosts (GitHub, GitLab, Zenodo, etc.).
    • Compatibility with container technologies other than Docker, such as Podman.

  2. Choose a pathway for using repo2docker images in JupyterHub

    main

    There are four primary ways to define and use environments for Binder or JupyterHub, ranging from using pre-built community images to writing custom Dockerfiles. Use the following decision logic to choose your approach:

    1. Use a community maintained image: The simplest method. Re-use an existing image maintained by a community. Best if an image already exists that meets your needs.
    2. Inherit from a community maintained image and add extras: Use a community image as a base (via a FROM instruction in a Dockerfile) and install additional packages. Best if a community image is almost what you need.
    3. Use repo2docker to build your environment image: Use repo2docker's automated build process by providing supported configuration files (like requirements.txt or environment.yml). Best if you want to avoid Dockerfile syntax and only include specific packages.
    4. Use a full fledged custom Dockerfile: Write a complete Dockerfile from scratch. Best for advanced users requiring absolute control over the environment, optimization, and specific system-level configurations.

  3. What is the Reproducible Execution Environment Specification (REES)?

    main

    The Reproducible Execution Environment Specification (REES) is the collection of configuration files found in a repository, their contents, and the resulting actions taken by repo2docker to build an environment.

    REES aims to automate community best practices for reproducible environments. A repository is considered REES-compliant if it uses standard specification files (like requirements.txt for pip or Project.toml for Julia) so that a human could reproduce the environment manually using common tools without needing repo2docker software.

  4. Specify runtimes with `runtime.txt`

    main

    Use a runtime.txt file to specify versions for runtimes like Python or R when your primary environment specification (like requirements.txt or install.R) does not support version pinning.

    Warning: runtime.txt is ignored if you use environment.yml (Conda) or Project.toml (Julia), as those formats already support runtime specification.

  5. Implement the ContentProvider interface

    main

    When creating a custom content provider, you must implement two core methods defined by the ContentProvider base class:

    • detect(input_string: str) -> dict: Analyzes the input to see if it belongs to this provider. If it does, it returns a spec dictionary.
    • fetch(spec: dict) -> str: Uses the spec dictionary to download or locate the repository and returns the local filesystem path to the repository.

  6. How repo2docker buildpacks work

    main

    repo2docker uses a buildpack system inspired by Heroku. It allows you to create complex environments by combining multiple configuration files. The system looks for configuration files in the repository's root or within a binder or .binder directory.

    Key features include:

    • Common configuration files: Uses familiar installation and packaging tools.
    • Composability: Buildpacks are designed to work together, allowing you to create multi-language environments (e.g., Python + Julia + R) within a single container.
    • Standardized locations: It automatically searches for configuration in the root, binder/, or .binder/ directories.

  7. Understanding repo2docker's deterministic output

    main

    repo2docker acts as a deterministic algorithm. When provided with an input directory containing a specific repository state, it will always produce the exact same Dockerfile.

    This determinism provides two main benefits:

    1. Efficient Caching: Tools built on top of repo2docker (like BinderHub) can reuse cached artifacts by identifying the repository's state (e.g., via a git commit hash).
    2. Docker Build Cache Optimization: Because repositories often share similar structures, repo2docker produces Dockerfiles with high overlap, maximizing the effectiveness of the Docker build cache.

  8. How repo2docker builds images

    main

    repo2docker automates the creation of Open Container Initiative (OCI) compliant images. Instead of requiring users to write complex Dockerfiles, it uses configuration files found within a repository to define the programming languages, software tools, and datasets required for the environment.

    While Dockerfiles are the standard for defining images, repo2docker provides a more accessible workflow for researchers and data scientists by leveraging specialized configuration files. However, you can still use a Dockerfile with repo2docker if required.

  9. How Buildpack detection works

    main

    When repo2docker is executed, it determines which Buildpack to use by following these steps:

    1. It iterates through an ordered list of BuildPack objects in Repo2Docker.buildpacks (from most-specific to least-specific).
    2. It calls the detect method of each BuildPack. This method checks the current working directory for specific files (e.g., requirements.txt, environment.yml, install.R) and returns True if it can handle the repository.
    3. If no buildpack returns True, repo2docker falls back to the default buildpack defined in Repo2Docker.default_buildpack.

  10. Extend existing buildpacks with libraries or UI

    main
    If you want to add support for a new library or UI component to a language that is already supported (e.g., adding RStudio or Shiny support to an existing R buildpack), do not create a new buildpack. Instead, extend the existing buildpack by adding the necessary library support. This is preferred over creating entirely new buildpacks for incremental feature additions.

  11. Categorize roadmap suggestions using 'next step' labels

    main

    When submitting issues related to the project roadmap, use the following categories to describe the nature of the task:

    • now: Concrete and actionable steps ready for immediate work (e.g., fixing specific documentation typos or linked issues).
    • soon: Less concrete steps that are currently under discussion; once a plan of action is finalized, they move to the 'now' category.
    • later: Abstract ideas or tasks requiring significant discussion or experimentation, or concrete steps that have been intentionally postponed.

  12. How repo2docker determines image configuration

    main
    Inspired by Heroku Build Packs, repo2docker automatically detects configuration files within a source repository to determine how the Docker image should be built. It looks for specific files to decide which dependencies to install and how to configure the environment.