poudriere
repository·master·Indexed 19 days ago
https://github.com/freebsd/poudriereA FreeBSD tool for testing package production and bulk building ports into binary packages using jails and ZFS. It provides an isolated build environment with minimal dependencies on the FreeBSD base system. Features include an incremental build mode via PKG_NO_VERSION_FOR_DEPS to align rebuild behavior with pkg upgrade and shared library tracking.
What's inside poudriere
- poudriere is a tool designed for testing package production on FreeBSD and for bulk building ports to create binary packages. It leverages modern FreeBSD features like ZFS and jails to provide an efficient and isolated build environment, while maintaining a minimal dependency footprint by depending only on the FreeBSD base system.
Use the `ucl` Lua module to parse and format data
masterThe
uclmodule provides a Lua interface to thelibuclC library, allowing you to parse UCL/JSON/YAML strings and files into Lua tables, and convert Lua tables back into formatted strings (like JSON or UCL).local ucl = require("ucl") local parser = ucl.parser() local res, err = parser:parse_string('{key=value}') if not res then print('parser error: ' .. err) else local obj = parser:get_object() local got = ucl.to_format(obj, 'json') endWhat is cpdup and how does it mirror filesystems?
mastercpdupis a filesystem mirroring utility that creates an exact mirror of a source in a destination.Key behaviors:
- Synchronization: It creates and deletes files and directories as necessary to ensure the destination matches the source.
- Metadata Preservation: It mirrors UTimes, hardlinks, softlinks, devices, permissions, and flags.
- Efficiency: By default, it does not copy files that appear already synchronized (where source and destination sizes and
mtimesmatch). - Safety Measures:
- It asks for confirmation before removing any file or directory from the destination.
- It refuses to replace a destination directory with a file.
- Limitations: It does not cross mount points in either the source or the destination.
How the shared library tracking algorithm works
masterWhen
PKG_NO_VERSION_FOR_DEPS=yesis active, Poudriere uses an inspection algorithm to decide if a package needs to be rebuilt due to shared library changes. The goal is to rebuild only if the existing package's required libraries are no longer satisfied by the current repository.The Inspection Process
- Scheduling: During build planning, if a package uses shared libraries and isn't already scheduled for rebuild, it is marked for "inspection" after its dependencies are built.
- Stash Requirements: Poudriere stashes the list of shared libraries (
shlibs) required by the existing (old/stale) package, excluding base libraries provided by the jail's clean snapshot. - Dependency Graph Construction: For the package being inspected, Poudriere builds a full recursive runtime dependency graph from the existing package (using
package_recursive_deps). - Effective Graph Comparison: It then constructs a "current effective" graph by looking at the shared libraries provided by the newly built packages in the current build run.
- Decision: The algorithm compares the old requirements against the new providers:
- Success: If an exact library name is found in the new graph, no rebuild is needed.
- Rebuild: If a library "looks like" what is needed (e.g., the package provides
libfoo.so.2but the old package needslibfoo.so.1), a rebuild is triggered. - Rebuild: If a required library is not provided by any dependency in the new graph, a rebuild is triggered.
Understand the UCL (Universal Configuration Language) syntax
masterUCL is a configuration language designed for convenience, heavily inspired by
nginxconfiguration but fully compatible withJSON. It allows for more flexible syntax than strict JSON, such as omitting braces for top-level objects, omitting quotes for strings and keys, and using=instead of:for assignments.Key syntax features include:
- No comma requirement: You can safely use trailing commas or semicolons in arrays and objects.
- Automatic array creation: Defining the same key multiple times in an object automatically converts that key into an array.
- Named keys hierarchy: Using
section "name" { ... }orsection name "subname" { ... }allows you to build nested object hierarchies easily. - Flexible booleans: Supports
true/false,yes/no, andon/off. - Multiline strings: Uses
<<TERMINATORsyntax (e.g.,<<EOD) for shell-like multiline strings.
# Nginx-like UCL syntax param = value; section { param1 = value1; flag = true; number = 10k; time = 0.2s; string = "something"; subsection { host = { host = "hostname"; port = 900; } host = { host = "hostname"; port = 901; } } }Behavioral changes in PKG_NO_VERSION_FOR_DEPS mode
masterEnabling
PKG_NO_VERSION_FOR_DEPS=yesintroduces several changes to how Poudriere operates:- Dry-run limitations: Dry-run mode can no longer predict exactly what will be done because shared library satisfaction is determined by inspecting packages after they are built.
- Inspection Category: A new "inspected" category is used. Poudriere inspects a port to see if its shared library dependencies are satisfied. If they are, the port is considered done; if not, it is rebuilt.
- Importance of PORTREVISION: In the default mode, recursive deletion often hid the importance of
PORTREVISIONbumps. In this new mode,PORTREVISIONbumps become critical because rebuilds only occur if the version or a required shared library version actually changes.
Understand when packages are forced to rebuild in PKG_NO_VERSION_FOR_DEPS mode
masterWhen using
PKG_NO_VERSION_FOR_DEPS=yes, Poudriere uses a more surgical approach to rebuilding. It will only force a rebuild of a package in the following specific cases:- Version/Metadata Changes:
PORTVERSION,PORTREVISION, orPORTEPOCHbump.- Changed
PKGNAME. - Changed
FLAVORfor aPKGNAME. - Changed
ABI,ARCH, orNOARCH.
- Dependency/Option Changes:
- New list of dependencies (not including versions) (requires
CHECK_CHANGED_DEPS). - Changed options (requires
CHECK_CHANGED_OPTIONS).
- New list of dependencies (not including versions) (requires
- Origin/Location Changes:
MOVED: origin moved to a new location.MOVED: origin expired.- Nonexistent origin.
- A package with the wrong origin for its
PKGNAME.
- System/Integrity Issues:
pkg bootstrapis not available.FORBIDDENis set for the port.- Corrupted package file.
bulk -a: A package which the tree no longer creates (e.g., a removedFLAVOR).
- Shared Library Requirements: If a package requires a shared library that no dependency provides (detected via inspection after builds).
- Version/Metadata Changes:
Validate UCL objects with JSON Schema
masterUCL supports validation using the JSON Schema v4 standard. You can define your schema in either JSON or UCL format.
Because UCL supports multiple values for keys (implicit arrays), you can use generic integer constraints like
maxValuesandminValuesto define the limits of how many values a single key can contain.Install and set up poudriere
masterTo install poudriere from source and perform the initial configuration, follow these steps:
- Build and install the tool from the top-level directory using
./configure,make, andmake install. - Initialize the configuration file by copying the sample configuration to the active location:
/usr/local/etc/poudriere.conf.sampleto/usr/local/etc/poudriere.conf. - Edit
/usr/local/etc/poudriere.confto match your environment and requirements. - For specific usage patterns, consult the
EXAMPLESsection in theman poudrieremanual page or follow the "bulk build of binary packages" guide in the official wiki.
./configure make make install cp /usr/local/etc/poudriere.conf.sample /usr/local/etc/poudriere.conf- Build and install the tool from the top-level directory using
Enable the new incremental build mode with PKG_NO_VERSION_FOR_DEPS
masterPoudriere offers two incremental build modes. By default, Poudriere rebuilds everything downstream if a dependency changes, which often results in rebuilding packages that
pkg upgradewould not actually install.To align Poudriere's behavior with
pkg upgrade(only rebuilding what would actually be upgraded), enable the new incremental mode by setting the environment variablePKG_NO_VERSION_FOR_DEPS=yes.In this mode, Poudriere does not store specific versions for dependencies. For example, if
foo-1.2is bumped tofoo-1.3, it will no longer force a rebuild of downstream packages because the dependency is registered asfoorather than a specific version.export PKG_NO_VERSION_FOR_DEPS=yes # Then run your poudriere commandsBenchmark libucl performance
masterTo verify the performance of the UCL parser and emitter on your own system, navigate to thelibucltop directory and run themake checkcommand. This will execute the internal benchmarks.make checkCreate Arch Linux or CentOS packages for cpdup
masterIf you need to distribute
cpdupas a package rather than installing it directly from source, use the provided Makefile targets:Arch Linux
Generate a package using
make archpkgand then install it usingpacman -U.CentOS
Generate an RPM using
make rpmand then install it usingrpm -ivh.Note: Use wildcards to match the generated filename.
# Arch Linux make archpkg sudo pacman -U cpdup-*.pkg.* # CentOS make rpm sudo rpm -ivh cpdup-*.rpm