poudriere

repository·master·Indexed 19 days ago

https://github.com/freebsd/poudriere

A FreeBSD tool for testing package production and bulk building ports into binary packages using jails and ZFS. It provides an isolated build environment with minimal dependencies on the FreeBSD base system. Features include an incremental build mode via PKG_NO_VERSION_FOR_DEPS to align rebuild behavior with pkg upgrade and shared library tracking.

Tokens
27.8K
Snippets
85
Records
117
Agent score
60%

What's inside poudriere

  1. What is poudriere?

    master
    poudriere is a tool designed for testing package production on FreeBSD and for bulk building ports to create binary packages. It leverages modern FreeBSD features like ZFS and jails to provide an efficient and isolated build environment, while maintaining a minimal dependency footprint by depending only on the FreeBSD base system.
  2. Use the `ucl` Lua module to parse and format data

    master

    The ucl module provides a Lua interface to the libucl C library, allowing you to parse UCL/JSON/YAML strings and files into Lua tables, and convert Lua tables back into formatted strings (like JSON or UCL).

    local ucl = require("ucl")
    
    local parser = ucl.parser()
    local res, err = parser:parse_string('{key=value}')
    
    if not res then
    	print('parser error: ' .. err)
    else
    	local obj = parser:get_object()
    	local got = ucl.to_format(obj, 'json')
    end
  3. What is cpdup and how does it mirror filesystems?

    master

    cpdup is a filesystem mirroring utility that creates an exact mirror of a source in a destination.

    Key behaviors:

    • Synchronization: It creates and deletes files and directories as necessary to ensure the destination matches the source.
    • Metadata Preservation: It mirrors UTimes, hardlinks, softlinks, devices, permissions, and flags.
    • Efficiency: By default, it does not copy files that appear already synchronized (where source and destination sizes and mtimes match).
    • Safety Measures:
      • It asks for confirmation before removing any file or directory from the destination.
      • It refuses to replace a destination directory with a file.
    • Limitations: It does not cross mount points in either the source or the destination.
  4. How the shared library tracking algorithm works

    master

    When PKG_NO_VERSION_FOR_DEPS=yes is active, Poudriere uses an inspection algorithm to decide if a package needs to be rebuilt due to shared library changes. The goal is to rebuild only if the existing package's required libraries are no longer satisfied by the current repository.

    The Inspection Process

    1. Scheduling: During build planning, if a package uses shared libraries and isn't already scheduled for rebuild, it is marked for "inspection" after its dependencies are built.
    2. Stash Requirements: Poudriere stashes the list of shared libraries (shlibs) required by the existing (old/stale) package, excluding base libraries provided by the jail's clean snapshot.
    3. Dependency Graph Construction: For the package being inspected, Poudriere builds a full recursive runtime dependency graph from the existing package (using package_recursive_deps).
    4. Effective Graph Comparison: It then constructs a "current effective" graph by looking at the shared libraries provided by the newly built packages in the current build run.
    5. Decision: The algorithm compares the old requirements against the new providers:
      • Success: If an exact library name is found in the new graph, no rebuild is needed.
      • Rebuild: If a library "looks like" what is needed (e.g., the package provides libfoo.so.2 but the old package needs libfoo.so.1), a rebuild is triggered.
      • Rebuild: If a required library is not provided by any dependency in the new graph, a rebuild is triggered.
  5. Understand the UCL (Universal Configuration Language) syntax

    master

    UCL is a configuration language designed for convenience, heavily inspired by nginx configuration but fully compatible with JSON. It allows for more flexible syntax than strict JSON, such as omitting braces for top-level objects, omitting quotes for strings and keys, and using = instead of : for assignments.

    Key syntax features include:

    • No comma requirement: You can safely use trailing commas or semicolons in arrays and objects.
    • Automatic array creation: Defining the same key multiple times in an object automatically converts that key into an array.
    • Named keys hierarchy: Using section "name" { ... } or section name "subname" { ... } allows you to build nested object hierarchies easily.
    • Flexible booleans: Supports true/false, yes/no, and on/off.
    • Multiline strings: Uses <<TERMINATOR syntax (e.g., <<EOD) for shell-like multiline strings.
    # Nginx-like UCL syntax
    param = value;
    section {
        param1 = value1;
        flag = true;
        number = 10k;
        time = 0.2s;
        string = "something";
        subsection {
            host = { host = "hostname"; port = 900; }
            host = { host = "hostname"; port = 901; }
        }
    }
  6. Behavioral changes in PKG_NO_VERSION_FOR_DEPS mode

    master

    Enabling PKG_NO_VERSION_FOR_DEPS=yes introduces several changes to how Poudriere operates:

    • Dry-run limitations: Dry-run mode can no longer predict exactly what will be done because shared library satisfaction is determined by inspecting packages after they are built.
    • Inspection Category: A new "inspected" category is used. Poudriere inspects a port to see if its shared library dependencies are satisfied. If they are, the port is considered done; if not, it is rebuilt.
    • Importance of PORTREVISION: In the default mode, recursive deletion often hid the importance of PORTREVISION bumps. In this new mode, PORTREVISION bumps become critical because rebuilds only occur if the version or a required shared library version actually changes.
  7. Understand when packages are forced to rebuild in PKG_NO_VERSION_FOR_DEPS mode

    master

    When using PKG_NO_VERSION_FOR_DEPS=yes, Poudriere uses a more surgical approach to rebuilding. It will only force a rebuild of a package in the following specific cases:

    • Version/Metadata Changes:
      • PORTVERSION, PORTREVISION, or PORTEPOCH bump.
      • Changed PKGNAME.
      • Changed FLAVOR for a PKGNAME.
      • Changed ABI, ARCH, or NOARCH.
    • Dependency/Option Changes:
      • New list of dependencies (not including versions) (requires CHECK_CHANGED_DEPS).
      • Changed options (requires CHECK_CHANGED_OPTIONS).
    • Origin/Location Changes:
      • MOVED: origin moved to a new location.
      • MOVED: origin expired.
      • Nonexistent origin.
      • A package with the wrong origin for its PKGNAME.
    • System/Integrity Issues:
      • pkg bootstrap is not available.
      • FORBIDDEN is set for the port.
      • Corrupted package file.
      • bulk -a: A package which the tree no longer creates (e.g., a removed FLAVOR).
    • Shared Library Requirements: If a package requires a shared library that no dependency provides (detected via inspection after builds).
  8. Install and set up poudriere

    master

    To install poudriere from source and perform the initial configuration, follow these steps:

    1. Build and install the tool from the top-level directory using ./configure, make, and make install.
    2. Initialize the configuration file by copying the sample configuration to the active location: /usr/local/etc/poudriere.conf.sample to /usr/local/etc/poudriere.conf.
    3. Edit /usr/local/etc/poudriere.conf to match your environment and requirements.
    4. For specific usage patterns, consult the EXAMPLES section in the man poudriere manual page or follow the "bulk build of binary packages" guide in the official wiki.
    ./configure
    make
    make install
    cp /usr/local/etc/poudriere.conf.sample /usr/local/etc/poudriere.conf
  9. Enable the new incremental build mode with PKG_NO_VERSION_FOR_DEPS

    master

    Poudriere offers two incremental build modes. By default, Poudriere rebuilds everything downstream if a dependency changes, which often results in rebuilding packages that pkg upgrade would not actually install.

    To align Poudriere's behavior with pkg upgrade (only rebuilding what would actually be upgraded), enable the new incremental mode by setting the environment variable PKG_NO_VERSION_FOR_DEPS=yes.

    In this mode, Poudriere does not store specific versions for dependencies. For example, if foo-1.2 is bumped to foo-1.3, it will no longer force a rebuild of downstream packages because the dependency is registered as foo rather than a specific version.

    export PKG_NO_VERSION_FOR_DEPS=yes
    # Then run your poudriere commands
  10. Create Arch Linux or CentOS packages for cpdup

    master

    If you need to distribute cpdup as a package rather than installing it directly from source, use the provided Makefile targets:

    Arch Linux

    Generate a package using make archpkg and then install it using pacman -U.

    CentOS

    Generate an RPM using make rpm and then install it using rpm -ivh.

    Note: Use wildcards to match the generated filename.

    # Arch Linux
    make archpkg
    sudo pacman -U cpdup-*.pkg.*
    
    # CentOS
    make rpm
    sudo rpm -ivh cpdup-*.rpm