BPF Compiler Collection (BCC)

BCC is a toolkit designed for creating efficient kernel tracing and manipulation programs using eBPF (extended Berkeley Packet Filters). It allows you to attach user-defined, sandboxed bytecode to kernel probes (kprobes) to perform instrumentation on a live kernel without the risk of crashing or hanging the system.

Key features include:

• Kernel Instrumentation in C: Write the core logic in C, utilizing a C wrapper around LLVM.
• High-level Front-ends: Use Python or Lua to manage the lifecycle of BPF programs and process data.
• Use Cases: Performance analysis, network traffic control, and system observability.

Requirements:

• Linux kernel 4.1 or above is generally required for most BCC functionality.

BPF maps allow you for storing state in the kernel. BPF_HASH(name) creates an associative array.

Common operations:

• map.update(&key, &value): Inserts or updates a value.
• map.lookup(&key): Returns a pointer to the value if found, or NULL if not. Note: The BPF verifier requires you to check if the returned pointer is NULL before dereferencing it.
• map.delete(&key): Removes the entry.

Example: Using a hash map to store timestamps to calculate the duration between two events.

from bcc import BPF

b = BPF(text="""
#include <uapi/linux/ptrace.h>

BPF_HASH(last);

int do_trace(struct pt_regs *ctx) {
    u64 ts, *tsp, delta, key = 0;
    tsp = last.lookup(&key);
    if (tsp != NULL) {
        delta = bpf_ktime_get_ns() - *tsp;
        // ... logic ...
        last.delete(&key);
    }
    ts = bpf_ktime_get_ns();
    last.update(&key, &ts);
    return 0;
}
""")

BCC provides two ways to look at block device I/O:

1. biolatency: Traces the time from device issue to completion and prints a histogram. This is ideal for spotting I/O latency outliers and multi-mode distributions.
2. biosnoop: Prints a line for every disk I/O with detailed timing. This is useful for examining time-ordered patterns, such as reads queuing behind writes.

# biolatency example output
     usecs           : count     distribution
       0 -> 1        : 0        |                                      |
     128 -> 255      : 12       |********                              |
     256 -> 511      : 15       |**********                            |

# biosnoop example output
TIME(s)        COMM           PID    DISK    T  SECTOR    BYTES   LAT(ms)
0.000004001    supervise      1950   xvda1   W  13092560  4096       0.74

You can define probes in your application using two different methods:

1. StaticTracepoint.h: Use the FOLLY_SDT macro provided in the BCC headers (located at tests/python/include/folly/tracing/StaticTracepoint.h).
2. SystemTap dtrace: Use systemtap-sdt-dev to generate header and object files from a .d file.

To use dtrace, install the package (e.g., sudo dnf install systemtap-sdt-dev on some distros) and run:

$ dtrace -h -s usdt_sample_lib1/src/lib1_sdt.d -o usdt_sample_lib1/include/usdt_sample_lib1/lib1_sdt.h
$ dtrace -G -s usdt_sample_lib1/src/lib1_sdt.d -o lib1_sdt.o

BPF CO-RE (Compile Once – Run Everywhere) applications use vmlinux.h to access kernel types (both exported and internal) without depending on the local kernel headers package.

• Versioned Headers: The repository includes pre-generated vmlinux.h files for various kernel versions (e.g., vmlinux_505.h for kernel v5.5) to ensure reproducible builds and compatibility.
• Symbolic Link: A symbolic link named vmlinux.h is provided, which typically points to the latest or default version.
• Generation Requirement: Generating vmlinux.h requires a kernel with BTF (BPF Type Format) information enabled via CONFIG_DEBUG_INFO_BTF=y.

Tracepoints are more stable than kprobes because they have a fixed API. Use TRACEPOINT_PROBE(category, name) to instrument them.

When using tracepoints, the args structure is automatically populated with the tracepoint's arguments. You can inspect the available fields by reading the tracepoint's format file in debugfs: /sys/kernel/debug/tracing/events/<category>/<name>/format.

Note: When using perf_submit within a tracepoint, pass args as the first argument instead of pt_regs *ctx.

TRACEPOINT_PROBE(syscalls, sys_enter_setuid) {
    struct data_t data = {};
    data.uid = args->uid; // 'args' is auto-populated
    // ...
    events.perf_submit(args, &data, sizeof(data));
    return 0;
}

Use these tools to monitor network activity and identify unexpected connections:

1. tcpconnect: Prints output for every active TCP connection initiated via connect() (e.g., client-side connections).
2. tcpaccept: Prints output for every passive TCP connection accepted via accept() (e.g., server-side connections).

# tcpconnect output
PID    COMM         IP SADDR            DADDR            DPORT
1479   telnet       4  127.0.0.1        127.0.0.1        23

# tcpaccept output
PID    COMM         IP RADDR            LADDR            LPORT
907    sshd         4  192.168.56.1     192.168.56.102   22

The Tunnel Monitor uses a BPF program to parse packets across encapsulation boundaries. It specifically targets VXLAN environments to record both inner and outer IP addresses, as well as the VXLAN ID, into a hash table. The hash table tracks the number of bytes and packets received/transmitted.

A key technical feature of this implementation is the use of bpf_tail_call, which allows the program to use the same state machine logic to parse two different IP headers (the inner and outer headers) efficiently.

This tutorial provides guidance on using bcc tools to resolve performance, troubleshooting, and networking issues.

Prerequisites:

• Installation: bcc must be already installed on your system. Refer to INSTALL.md for installation instructions.
• Kernel Requirements: This tutorial utilizes enhancements found in the Linux 4.x series and later.

Note for Developers: If your goal is to learn how to develop new bcc tools rather than using existing ones, refer to the tutorial_bcc_python_developer.md guide instead.