bpftrace Documentation

bpftrace is a general-purpose tracing tool and language for Linux that leverages eBPF for efficient tracing with minimal overhead. It uses LLVM as a compiler backend and libbpf to interact with the Linux BPF subsystem.

Supported tracing mechanisms include:

• Kernel dynamic tracing: kprobes, hardware and software perf events.
• User-level dynamic tracing: USDT, uprobes.
• Tracepoints: Regular tracepoints and raw tracepoints.
• Other: General tracing capabilities via the BPF subsystem.

bpftrace uses a multi-pass system to perform type inference and resolution, ensuring static types are available at runtime. The process involves four primary stages:

1. TypeRuleCollector: Walks the AST to collect type rules. It seeds known types (like integer literals) and registers rules for compound nodes (like binary operations).
2. AstTransformer: Uses resolved types to transform introspection functions (e.g., sizeof, typeinfo) and expand memcmp calls for tuples/records.
3. TypeApplicator: Adds the resolved SizedType to the AST nodes.
4. CastCreator: Injects explicit casts (e.g., (uint32)$b) to ensure operands in binary operations have matching types and sizes.

This system allows bpftrace to handle complex scenarios like comptime branches and variable type propagation.

bpftrace is designed to provide a quick and easy way to write observability-based BPF programs, specifically abstracting away the complexities of eBPF such as the verifier, kernel/userspace interaction, attachment, program loading, memory access, and BPF maps.

When writing or evaluating bpftrace scripts, keep the following language goals in mind (in priority order):

1. Conciseness (one-liners)
2. Readability
3. Clean abstraction from eBPF
4. Ability to quickly iterate
5. Composability
6. Good performance (kernel and userspace)
7. Speed of program initialization/start-up

The bpftrace Standard Library consists of builtins, functions, macros, and map value functions.

Builtins, functions, and macros are often grouped as Helpers. Because their boundaries are blurred by design, many helpers can be invoked either with or without call syntax (e.g., pid and pid() are equivalent).

The bpftrace project follows a meritocratic, consensus-based governance model with the following roles:

• Users: Community members who use the project and provide feedback, report strengths/weaknesses, and evangelize the project.
• Contributors: Members who actively participate by reporting bugs, writing documentation, coding, fixing bugs, or assisting with infrastructure. Contributions are primarily made via GitHub Pull Requests.
• Committers: Experienced contributors with write access who can merge approved PRs using a 'commit-then-review' process. They are nominated by maintainers.
• Maintainers: Identified as 'code owners'. They are responsible for reviewing PRs (at least one maintainer approval is required for all PRs), mentoring developers, triaging issues, and making decisions when community consensus cannot be reached.

To manage expectations when using bpftrace, be aware that the language does not support or prioritize the following:

• Testability or debuggability (e.g., no gdb or self-tracing)
• Dynamic typing
• Classes or Inheritance
• Metaprogramming
• Exception handling
• BPF security, LSM, XDP, or Scheduling
• BPF concepts that do not pertain to observability or cannot be abstracted cleanly

To track data specific to a thread (e.g., capturing context between an entry and exit probe), use a map keyed by the thread ID (tid).

Common pattern:

1. In the entry probe, store the value in a map using tid as the key: @map[tid] = value;
2. In the exit probe, retrieve the value: $val = @map[tid];
3. Clean up the map entry using delete(@map, tid); to prevent memory leaks.

kprobe:do_nanosleep {
  @start[tid] = nsecs;
}

kretprobe:do_nanosleep /has_key(@start, tid)/ {
  printf("slept for %d ms\n", (nsecs - @start[tid]) / 1000000);
  delete(@start, tid);
}

Use kprobe to trace the entry of kernel functions and kretprobe to trace their return values.

To access function arguments, use the built-in arg0, arg1, ..., argN variables. For a kprobe, arg0 represents the first argument of the function being traced.

If your kernel does not provide BTF (BPF Type Format) data, you may need to #include the relevant Linux kernel headers to access specific structure definitions (e.g., <linux/path.h>).

#ifndef BPFTRACE_HAVE_BTF
#include <linux/path.h>
#include <linux/dcache.h>
#endif

kprobe:vfs_open
{
	printf("open path: %s\n", str(((struct path *)arg0)->dentry->d_name.name));
}

Before running the fuzzer, optimize your system settings by setting the core pattern to echo and the CPU scaling governor to performance.

When running afl-fuzz, it is recommended to use --test=codegen mode. You must provide an initial input file in the input directory (e.g., echo a > input/a) to seed the process. The fuzzer will replace the @@ token with its generated input files. Crashes found by the fuzzer will be stored in the output/crashes directory.

echo core | sudo tee -a /proc/sys/kernel/core_pattern
cd /sys/devices/system/cpu
echo performance | sudo tee cpu*/cpufreq/scaling_governor

AFL_NO_AFFINITY=1 \
ASAN_OPTIONS=abort_on_error=1,symbolize=0 \
BPFTRACE_BTF= \
afl-fuzz -a text -M 0 -m none -i ./input -o ./output -t 3000 -- \
     src/bpftrace --test=codegen @@ 2>/dev/null

To fuzz bpftrace with AFL, you must use nix for the environment setup. The process involves entering a development shell and compiling the project using the AFL compilers (afl-clang-fast and afl-clang-fast++) with Address Sanitizer (ASAN) enabled for better bug detection.

Note: Using ASAN can consume significant memory. To disable it, remove AFL_USE_ASAN=1 from the make command and -DBUILD_ASAN=1 from the cmake command.

nix develop #.bpftrace-fuzz
CC=afl-clang-fast CXX=afl-clang-fast++ cmake -B build-fuzz -DCMAKE_BUILD_TYPE=Debug -DBUILD_ASAN=1

cd build-fuzz && AFL_USE_ASAN=1 make -j$(nproc)

When you require precise event statistics or need to capture intermediate states of a map during a tight loop, avoid asynchronous functions like printf(). Instead, use synchronous functions (such as count() and hist()) to ensure more reliable and accurate results that are not subject to user-space processing delays.

BEGIN {
    @=0;
    unroll(10) {
      print(@);
      @++;
    }
    exit()
}