DOCSEARCHbyAlgolia
Submit Docs

ClusterFuzz Documentation

repository·master·Indexed 26 days ago

https://github.com/google/clusterfuzz

A highly scalable fuzzing infrastructure used by Google to find security and stability issues in software. It supports multiple coverage-guided engines and automates bug triage and management. Documentation includes guides for the Casp CLI, bot log diagnostics, Docker image builds, local development environment setup using asdf, and CI/CD integration via ClusterFuzzLite.

Tokens
24.1K
Snippets
50
Records
162
Agent score
90%

What's inside ClusterFuzz

  1. Overview of ClusterFuzz features

    master

    ClusterFuzz is a scalable fuzzing infrastructure designed to find security and stability issues in software. It is used by Google for its products and serves as the backend for OSS-Fuzz.

    Key capabilities include:

    • Scalability: Capable of running on large-scale clusters (e.g., 30,000 VMs).
    • Crash Management: Accurate deduplication of crashes and testcase minimization.
    • Automation: Automatic bug filing, triage, and closing for issue trackers like Monorail and Jira.
    • Fuzzing Engines: Supports coverage-guided engines including libFuzzer, AFL++, and Honggfuzz, as well as blackbox fuzzing.
    • Analysis: Provides regression finding through bisection and statistics for analyzing fuzzer performance and crash rates.
    • Management: Includes a web interface for managing and viewing crashes and supports Firebase for authentication.

  2. Overview of Casp CLI

    master
    Casp is a modern Command-Line Interface (CLI) for ClusterFuzz designed for local interaction. It is built using the click library and follows a modular structure where commands are organized into a dedicated directory and shared logic resides in utilities.

  3. Understand ClusterFuzz structured logging contexts

    master
    ClusterFuzz uses structured logging to provide consistent metadata across different execution contexts. This allows for better filtering and faster indexed queries during troubleshooting. Logs are instrumented with specific contexts that propagate through the call stack. A single log entry may contain multiple contexts (e.g., Common, Task, and Testcase) depending on where it was triggered.

  4. System Requirements and Supported Platforms

    master

    ClusterFuzz is written in Python and supports the following operating systems:

    • Linux
    • macOS
    • Windows

    When running on Google Cloud Platform, ClusterFuzz depends on these services:

    • Compute Engine (optional; fuzzing bots can run anywhere)
    • App Engine
    • Cloud Storage
    • Cloud Datastore
    • Cloud Pub/Sub
    • BigQuery
    • Stackdriver Logging and Monitoring

    Note: The only currently supported bug tracker is the Chromium-hosted Monorail.

  7. Navigate the ClusterFuzz Web Interface

    master

    Once the ClusterFuzz server is deployed, you can access several key functional areas through the web interface:

    • Testcases: The default landing page. Use this to view, filter, and search for issues detected by fuzzers.
    • Fuzzer Statistics: View performance metrics for fuzzers. For in-process fuzz targets, this page includes performance reports, improvement recommendations, and links to target metadata.
    • Crash Statistics: Analyze crash data, including crash frequency, affected platforms, and trends over time.
    • Upload Testcase: Use this page to manually upload a single testcase for testing against a specific target. This is useful for verifying bugs reported by external researchers.
    • Jobs: Manage fuzzer workloads by creating new job configurations or modifying existing ones.
    • Configuration: An administrative-only page used to manage ClusterFuzz access control, credentials, and other system settings.

  8. Understand ClusterFuzz terminology and concepts

    master

    This glossary defines the core concepts used within ClusterFuzz to help you navigate the system, configure jobs, and interpret crash reports.

    Core Components

    • Bot: A machine that executes ClusterFuzz tasks.
    • Task: A unit of work performed by a bot (e.g., a fuzzing session or minimizing a testcase).
    • Fuzz target: A function or program that accepts an array of bytes and interacts with the API under test. It is typically driven by a fuzzing engine.
    • Fuzzer: A program that generates or mutates inputs of a specific format for a target program.
    • Fuzzing engine: A tool (like libFuzzer or AFL) used for coverage-guided fuzzing. It mutates inputs, collects coverage data, and manages the corpus.
    • Job type: A specification (defined via environment variables) that determines how to run a target program and where its builds are located.

    Data and Inputs

    • Corpus: A set of inputs for a fuzz target, typically representing minimal test inputs that maximize code coverage.
    • Corpus pruning: A task that removes unnecessary inputs from a corpus while maintaining the same level of code coverage.
    • Testcase: An input that causes a crash or bug. These can be downloaded as "Minimized" or "Unminimized" testcases.
    • Minimization: A task that attempts to reduce a testcase to its smallest possible size while still triggering the same bug.

    Crash and Bug Analysis

    • Crash state: A signature generated from a crash stacktrace used for deduplication.
    • Crash type: A classification used to determine bug severity. Common security-related types include Bad-cast, Heap-buffer-overflow, Heap-double-free, Heap-use-after-free, Stack-buffer-overflow, Stack-use-after-return, and Use-after-poison. Other types include Null-dereference, Timeout, Out-of-memory, Stack-overflow, and ASSERT.
    • Reliability of reproduction: A measure of whether a target program consistently crashes with the same crash state given the same input.
    • Regression range: A range of commits where a bug was introduced, expressed as x:y (where x is the inclusive start revision and y is the exclusive end revision).
    • Revision: A numeric identifier (not a git hash) for a build. For Git, you must map these numbers to hashes (e.g., using an incrementing ID or a date format like 20190110).

    Testing Tools

    • Sanitizer: A dynamic testing tool using compile-time instrumentation to detect bugs. Common examples include ASan (AddressSanitizer), LSan (LeakSanitizer), MSan (MemorySanitizer), UBSan (UndefinedBehaviorSanitizer), and TSan (ThreadSanitizer).

  10. Understand ClusterFuzz components and bot tasks

    master

    ClusterFuzz operates using two main components:

    App Engine

    Provides the web interface for accessing crashes and statistics, and manages regular cron job scheduling.

    Fuzzing Bots

    Machines that lease tasks from platform-specific queues. Bots are categorized into two types:

    • Preemptible: Can shut down at any time. They are recommended for scaling and can only run the fuzz task.
    • Non-preemptible: Expected to run uninterrupted. They can run all tasks, including critical ones like progression.

    Supported Bot Tasks:

    • fuzz: Run a fuzzing session.
    • progression: Check if a testcase still reproduces or is fixed.
    • regression: Calculate the revision range where a crash was introduced.
    • minimize: Perform testcase minimization.
    • corpus_pruning: Minimize a corpus to the smallest size based on coverage (libFuzzer only).
    • analyze: Run a manually uploaded testcase against a job to check for crashes.

  11. Run a ClusterFuzz bot image locally

    master

    Run a bot image using run_docker.bash.

    Requirements:

    • You must run this command as a non-root user. Ensure your user is in the docker group (sudo adduser $USER docker).

    Usage Options:

    • Latest deployed source: Use the base image.
    • Local checkout: Set LOCAL_SRC=1 and provide your configuration directory via CONFIG_DIR_OVERRIDE.