DOCSEARCHbyAlgolia
Submit Docs

pyproject.nix

repository·master·Indexed 43 days ago

https://github.com/pyproject-nix/pyproject.nix

A set of Nix utilities for managing and parsing Python project metadata. It supports standards such as PEP-621, PEP-440, PEP-508, PEP-599, Poetry, and requirements.txt. pyproject.nix decouples runtime dependency graphs from build-time graphs to prevent $PYTHONPATH leaking, support cyclic dependencies, and avoid binary wrapping issues by using standard Python virtual environments.

Tokens
6.1K
Snippets
21
Records
32
Agent score
56%

What's inside pyproject.nix

  1. Overview of Pyproject.nix capabilities

    master

    Pyproject.nix provides a collection of Nix utilities designed to work with Python project metadata within the Nix ecosystem. It acts as a 'swiss army knife' for parsing and handling various Python packaging standards, allowing developers to bridge the gap between Python's metadata formats and Nix expressions.

    Supported standards include:

    • PEP-621: pyproject.toml files
    • PEP-440: Versioning standards
    • PEP-508: Environment markers
    • PEP-599: Script inline metadata
    • Poetry: Unified project loading for both Poetry-specific and standard PEP-621 projects
    • Requirements.txt: Standard pip requirements files

  2. What is pyproject.nix

    master

    pyproject.nix is a toolkit of Nix utilities designed to work with Python project metadata within Nix. It is primarily built to handle PEP-621 compliant pyproject.toml files, but it also provides support for legacy or alternative formats like Poetry and requirements.txt.

    It is designed to be flexible and can be used in two main ways:

    1. Alongside the standard nixpkgs Python infrastructure.
    2. With the pyproject.nix build infrastructure.

  3. Understand build-system packages

    master

    For Python package managers that cannot resolve build-system dependencies (those defined in PEP 0518 but missing from lock files), pyproject.nix utilizes a specialized base package set.

    This set is intentionally smaller and more narrowly scoped than nixpkgs. Its sole purpose is to provide the specific dependencies required for the build-system process that must be supplemented from outside the standard Python lock file.

  4. How Renderers work with Python infrastructure

    master

    Renderers are used to convert a project abstraction into a concrete Nix derivation that Python tools understand.

    A common pattern is using a renderer to generate an attribute set for nixpkgs functions like buildPythonPackage or buildPythonApplication.

    Note on manual configuration: If a renderer does not provide all the attributes required for your specific build needs (due to limitations in what can be computed from the metadata), you should manually merge your own attribute set with the one returned by the renderer.

  5. How pyproject.nix builders solve nixpkgs Python issues

    master

    Unlike standard nixpkgs buildPythonPackage which relies on dependency propagation via $PYTHONPATH, pyproject.nix decouples the runtime dependency graph from the build-time graph.

    This approach solves several common Nix/Python problems:

    • Prevents $PYTHONPATH leaking: Dependencies from one environment (like a dev shell) won't leak into unrelated Python programs, which otherwise can override virtualenv dependencies.
    • Supports Cyclic Dependencies: While Nix requires a DAG, Python dependencies can be cyclic. pyproject.nix uses a resolver that returns a flat list of required packages, allowing cycles to exist without breaking the Nix graph.
    • Avoids Binary Wrapping issues: Instead of using wrappers that set NIX_PYTHONPATH, pyproject.nix uses standard Python virtual environments. This ensures that programs calling sys.executable correctly find their modules.
    • Reduces Extraneous Rebuilds: By not requiring runtime dependencies to be present at build-time, derivation hashes change less frequently.
    • Improves Build Scheduling: A flatter dependency graph allows for more efficient parallel scheduling of derivations.

  6. Understand the core concepts of pyproject.nix

    master

    To use pyproject.nix effectively, you should understand its three primary abstractions and how they compose:

    1. Project: The central data structure. A project attribute set represents your Python project by aggregating the parsed pyproject.toml (or other metadata sources), the parsed dependencies, and the project root directory.
    2. Validators: Tools that operate on the dependency constraints defined within a project. Use these to ensure your package set complies with specific requirements or specifications.
    3. Renderers: The bridge to Python infrastructure. A renderer takes a project and a Python interpreter derivation, then transforms them into a format usable by Python build tools. For example, a renderer can produce an attribute set compatible with the nixpkgs buildPythonPackage or buildPythonApplication functions.

  7. Understand the two pyproject.nix resolvers

    master

    The project implements two different resolvers to balance performance and capability:

    1. Fast Resolver (No cyclic support): A highly performant resolver used by resolveBuildSystem. It uses memoization for all known build-systems but cannot handle cyclic dependencies.
    2. Cyclic Resolver (Full support): A resolver that supports cyclic dependencies. This is used when resolving full virtual environments.

    Note: While cyclic build-systems are not supported by default in the fast resolver, you can override the resolver entirely via Nix overrides to use the cyclic-capable version.

  10. Create a Nix environment from requirements.txt

    master

    If your Python project uses a requirements.txt file instead of a standard pyproject.toml, you can use pyproject.nix to load these dependencies and create a Nix environment. This is typically achieved by using python.withPackages from Nixpkgs, where the packages are derived from the contents of your requirements.txt file.

    # Example logic (conceptual based on documentation):
# Use python.withPackages to create an environment containing 
# the dependencies listed in requirements.txt

  11. Build Cargo (Rust) packages from source using importCargoLock

    master

    To handle Python packages that include Rust extensions (like cryptography), use hacks.importCargoLock. This mechanism helps manage Cargo dependencies by creating a Rust vendor directory.

    Note that you may need to provide a cargoRoot if the Rust source is not in the default location, and you might need to manually supplement nativeBuildInputs (e.g., maturin, setuptools, cffi) or buildInputs (e.g., openssl) using overrideAttrs to ensure the build succeeds.

    final: prev: {
  cryptography = 
    (hacks.importCargoLock {
      prev = prev.cryptography;
      # Cryptography uses a non-standard location for it's Rust packaging
      cargoRoot = "src/rust";
    }).overrideAttrs
      (old: {
        nativeBuildInputs = 
          old.nativeBuildInputs 
          ++ final.resolveBuildSystem {
            maturin = [ ];
            setuptools = [ ];
            cffi = [ ];
            pycparser = [ ];
          };
        buildInputs = old.buildInputs or [ ] ++ [ pkgs.openssl ];
      });
}

  12. Generate a Nixpkgs overlay from pyproject-nix packages

    master

    If you want to use pyproject-nix generated packages within a standard Nixpkgs environment, use hacks.toNixpkgs. This creates an overlay that allows you to treat your generated Python packages as if they were part of python3Packages.

    This is useful for overriding a standard Python installation with one that includes specific packages generated by pyproject-nix (or tools like uv2nix).

    {
  callPackage, pyproject-nix, python3 
}:
let
  python = python3;
  hacks = callPackage pyproject-nix.build.hacks {};

  # Note: pythonSet would normally be generated by uv2nix
  pythonSet = callPackage pyproject-nix.build.packages {
    inherit python;
  };

  overlay = hacks.toNixpkgs {
    inherit pythonSet;
    packages = [
      "urllib3" # Use generated urllib3
    ];
  };

  python = python3.override {
    self = python;
    packageOverrides = overlay;
  };
in
  python.withPackages(ps: [ ps.urllib3 ])