Pumba Documentation

repository·master·Indexed 25 days ago

https://github.com/alexei-led/pumba

A chaos testing and network emulation tool for Docker, containerd, and Podman containers. Pumba allows developers to improve system resilience by injecting failures such as container termination (kill, stop, pause, restart), resource stress (CPU, memory, I/O), and network chaos using netem (latency, packet loss, duplication, corruption) and iptables.

Tokens
27.5K
Snippets
69
Records
171
Agent score
84%

What's inside Pumba

  1. Review Pumba's modularity and architecture

    master

    The Pumba codebase has undergone a modularity review to assess its maintainability for AI agents and developers. The project currently scores 7.4 / 10 on AI-agent maintainability.

    Key architectural findings include:

    • Healthy Core: Chaos action packages use narrow consumer-side sub-interfaces (e.g., Lister + Netem, Lister + IPTables) which allow for clean mocking.
    • Systemic Weaknesses:
      • A mutable package-level global chaos.DockerClient creates implicit coupling.
      • Runtime-agnostic interfaces (like NetemContainer and IPTablesContainer) leak Docker-specific concepts (such as tcimg and pull) into the general contract.
      • pkg/runtime/docker/docker.go is a large monolith (~1.3k LOC) with mixed responsibilities.
      • Significant boilerplate duplication exists in per-action CLI builders.
  2. Understand Pumba Network Chaos Testing

    master

    Pumba provides two complementary tools for network chaos testing by injecting a helper container into the target container's network namespace with NET_ADMIN capabilities:

    • netem: Manipulates outgoing traffic using Linux tc (traffic control). Supports delay, packet loss, corruption, duplication, and rate limiting.
    • iptables: Manipulates incoming traffic using Linux iptables. Supports packet loss with random or nth-packet matching.

    By combining both, you can create realistic asymmetric network conditions.

  3. Use Podman via Docker-compatible API

    master

    Pumba supports Podman by interacting with its Docker-compatible API socket using the Docker SDK.

    When using the Podman runtime, the podmanClient embeds a Docker client and overrides specific methods to handle Podman-specific behaviors (such as rootless guards, cgroup leaf naming, and sidecar configuration). Most other ctr.Client methods are inherited directly from the Docker delegate and work via the Docker-compat socket.

  4. Combine bandwidth limits and packet loss

    master

    To test combined network degradation, run pumba netem to limit outgoing bandwidth and pumba iptables to introduce incoming packet loss.

    # Limit outgoing bandwidth to 1Mbit/s
    pumba netem --tc-image ghcr.io/alexei-led/pumba-alpine-nettools:latest \
        --duration 10m rate --rate 1mbit myapp &
    
    # 5% loss on incoming traffic
    pumba iptables --iptables-image ghcr.io/alexei-led/pumba-alpine-nettools:latest \
        --duration 10m loss --probability 0.05 myapp &
  5. Perform resource stress testing with `pumba stress`

    master

    Use the pumba stress command to stress-test container resources (CPU, memory, I/O) by running a stress-ng sidecar container. This allows you to simulate resource contention and test how your application handles high load or OOM (Out of Memory) conditions.

    Basic Syntax:

    pumba stress [options] CONTAINERS

    Key Options:

    • --duration, -d (required): How long to run the stress test. Supports unit suffixes like ms, s, m, h.
    • --stressors: Pass specific stress-ng stressors. Note: You must use an = sign when passing values, e.g., --stressors="--cpu 4 --timeout 60s".
    • --stress-image: The Docker image containing stress-ng. Defaults to ghcr.io/alexei-led/stress-ng:latest.
    • --inject-cgroup: Enables same-cgroup injection mode for more realistic resource contention and shared OOM scope (see Same-Cgroup Injection Mode).
    pumba stress --duration 60s \
        --stressors="--cpu 4 --timeout 60s" \
        myapp
  6. Inject Network Chaos

    master

    Pumba provides two main ways to inject network chaos:

    netem

    Uses Linux tc (traffic control) to add latency, packet loss, duplication, corruption, or rate-limiting.

    • delay: Add latency (use --time to specify delay in ms).
    • loss: Drop packets (use --probability for percentage).
    • duplicate: Duplicate packets.
    • corrupt: Corrupt packets.
    • rate: Rate-limit packets.

    iptables

    Uses iptables to drop packets (ingress and egress).

    Tip: If your target container lacks tc or iptables, use the --tc-image flag to spawn a sidecar container that shares the target's network namespace.

    # Add 3 seconds network delay to mydb for 5 minutes
    pumba netem --duration 5m delay --time 3000 mydb
    
    # Drop 10% of incoming packets to myapp for 2 minutes
    pumba iptables --duration 2m loss --probability 0.1 myapp
    
    # Use a sidecar image for containers without nettools
    pumba --runtime containerd netem --tc-image ghcr.io/alexei-led/pumba-alpine-nettools:latest \
      --duration 5m delay --time 3000 <container-id>
  7. Use Podman as the Pumba runtime

    master

    To use Podman instead of Docker or Containerd, use the --runtime podman flag. If your Podman socket is in a non-standard location, you can specify it using the --podman-socket flag.

    Note on Rootless Mode: Many Pumba features (like netem, iptables, and stress) require rootful Podman. If running in rootless mode, these commands will fail with a specific error message suggesting you use podman machine set --rootful or run as root on Linux.

  8. Regenerate mocks using make mocks

    master

    Whenever an interface signature changes (such as updating Stressor or Lifecycle), you must regenerate the project mocks. Do not hand-edit mock_*.go or mocks/*.go files. Use the following command:

    make mocks

    When writing tests with the new request types, use the following mockery matcher style:

    • mock.AnythingOfType("*container.StressRequest")
    • mock.AnythingOfType("*container.RemoveOpts")
  9. Target specific Kubernetes nodes

    master

    Use nodeSelector or nodeAffinity in your DaemonSet specification to target specific node groups (e.g., EKS or GKE pools).

    spec:
      template:
        spec:
          # EKS node group
          nodeSelector:
            alpha.eksctl.io/nodegroup-name: my-node-group
          # Or GKE node pool
          # nodeSelector:
          #   cloud.google.com/gke-nodepool: node-pool
  10. Refactor chaos command dependency injection

    master

    The project currently uses a mutable package-level global chaos.DockerClient to manage the runtime client. This creates implicit dependencies and makes testing in isolation difficult.

    To improve maintainability and allow for multi-runtime use cases, it is recommended to replace the global with constructor injection.

    Recommended Pattern (Option B): Pass a factory closure into the CLI command constructors to make dependencies explicit for both developers and AI agents.

    type Runtime func() container.Client
    
    func NewStopCLICommand(ctx context.Context, runtime Runtime) *cli.Command { ... }