MongoDB Kingfisher

repository·main·Indexed 22 days ago

https://github.com/mongodb/kingfisher

A high-performance, open-source secret scanning and validation tool built in Rust. Kingfisher detects, verifies, and revokes leaked credentials across platforms including GitHub, GitLab, AWS S3, GCS, and Slack. It features live validation via provider APIs, blast radius mapping for cloud identities, and support for various output formats such as JSON, SARIF, and HTML. The project includes a CLI and library crates (kingfisher-core, kingfisher-rules, and kingfisher-scanner) for embedding the scanning engine into Rust applications.

Tokens
156.8K
Snippets
377
Records
478
Agent score
75%

What's inside kingfisher

  1. Overview of kingfisher-core

    main

    The kingfisher-core crate serves as the foundational layer for the Kingfisher secret scanning ecosystem. It provides the stable base types and utilities used by kingfisher-rules and kingfisher-scanner.

    Key capabilities include:

    • Blob and Content Abstractions: Standardized ways to handle raw data and content during scanning.
    • Source Location and Origin Modeling: Tools for tracking where secrets are found and their origin within a file or stream.
    • Shared Utilities: Common error handling and entropy calculation helpers used across the ecosystem.
  2. What is Kingfisher?

    main

    Kingfisher is a high-performance, open-source secret scanner and live secret validation tool built in Rust. It uses Intel's Hyperscan (SIMD-accelerated regex engine) and language-aware parsing to detect, validate, and triage leaked credentials at scale.

    Core Capabilities:

    • Detection: Scans code, Git history, and various platforms (GitHub, GitLab, AWS S3, Google Cloud Storage, Slack, etc.) using 1,051+ built-in rules.
    • Validation: Verifies discovered secrets against provider APIs to minimize false positives.
    • Revocation: Supports direct revocation of supported secrets via the CLI.
    • Reporting: Generates JSON, SARIF, TOON, and HTML outputs. It also includes a browser-based report viewer that can merge and deduplicate findings from Kingfisher, SARIF, Gitleaks, and TruffleHog.
    • Alerting: Sends scan summaries and finding details to various webhooks (Slack, Teams, Discord, etc.).
  3. Overview of Kingfisher built-in rules

    main

    Kingfisher includes a comprehensive set of pre-configured detection rules designed to identify secrets across various services. The rule set consists of:

    • 1,051 total detection rules
    • 612 providers
    • 914 detectors and 137 dependent rules

    Key capabilities within this rule set include:

    • Live Validation: 516 rules support live validation to verify if a detected secret is still active.
    • Direct Revocation: 52 rules support direct revocation to help mitigate the impact of a leak.
  4. Use kingfisher-rules for rule definitions and compilation

    main

    The kingfisher-rules crate is the central component for managing secret detection logic in Kingfisher. It provides the syntax and model types for defining rules, handles YAML parsing for rule files, and includes a set of embedded builtin rules.

    Developers can use RulesDatabase to compile rules into a format optimized for scanning engines. To build a complete scanning workflow, integrate this crate with kingfisher-core and kingfisher-scanner.

  5. Use kingfisher-scanner for secret detection

    main

    The kingfisher-scanner crate is a high-level library designed for detecting secrets within bytes, files, and blobs. It provides ergonomic APIs for scanning and returns finding models that include metadata about the location and the captured secret.

    To build a complete scanning solution, you should use kingfisher-scanner in conjunction with:

    • kingfisher-rules: To compile the detection rules.
    • kingfisher-core: For shared primitives used across the ecosystem.
  6. Explore the Built-in Rules List

    main

    Kingfisher includes 1,051 built-in secret detection rules. These rules can be filtered and searched based on the following criteria:

    • Provider: The source or type of secret being detected.
    • Confidence Level: The certainty with which the rule identifies a secret.
    • Validation Support: Whether the rule supports live validation to reduce false positives.
  7. Overview of Kingfisher Detection Rules

    main

    Kingfisher includes 1,051 built-in rules covering a wide range of categories, including:

    • Cloud Providers: AWS, GCP, Azure, Alibaba Cloud, DigitalOcean, etc.
    • AI & ML: OpenAI, Anthropic, Google Gemini, Cohere, Mistral, etc.
    • Dev & CI/CD: GitHub, GitLab, Docker Hub, npm, Vercel, etc.
    • Databases: PostgreSQL, MySQL, MongoDB, Redis, etc.
    • Security & Identity: Snyk, Auth0, Okta, 1Password, etc.
    • Crypto Material: Private keys (PEM, PGP/GPG, SSH), JWTs, etc.

    Rules support checksum-aware matching, which allows for offline structural verification of credentials (like modern PATs) to eliminate false positives without making external API calls.

  8. Interpret the Scan Summary and validation counters

    main

    After a scan, Kingfisher provides a summary containing validation statistics. Understanding these counters helps evaluate the accuracy and coverage of your scan:

    • Successful Validations: Credentials confirmed as active by the provider (e.g., the API returned a valid response).
    • Failed Validations: Validations that were attempted but failed due to HTTP errors, connection timeouts, or invalid credentials.
    • Skipped Validations: Validations that could not be attempted. This usually happens because:
      • Missing dependent rules: A rule requires a value from another rule (e.g., an AWS Secret Key rule requires an Access Key ID from a previous match) that was not found.
      • Preconditions not met: The validation endpoint requires additional context not available in the scan.

    When a validation is skipped, the finding will display a message like: Validation skipped - missing dependent rules: <helper-rule-id>.

  9. Use the Access Map to visualize secret blast radius

    main

    The --access-map feature performs active authentication against a provider to determine the identity and permissions associated with a detected secret. This helps identify the 'blast radius' (e.g., which S3 buckets or EC2 instances are exposed).

    Warning: Only use this feature when authorized to inspect the target account, as Kingfisher will issue network requests to the provider to validate access.

    To generate an access map during a scan, use the --access-map flag. You can then view the results using the view command.

    # Generate access map during scan
    kingfisher scan /path/to/code --access-map --view-report
    
    # View access-map reports locally
    kingfisher view kingfisher.json
    kingfisher view kingfisher.sarif
  10. Use Per-finding Fingerprints for Automation

    main

    Every finding line in detail mode (and every record in the Generic JSON payload) includes a stable fingerprint. This unique identifier allows downstream automation (like SIEM/SOAR, Jira, or custom deduplication tools) to:

    • Suppress repeat alerts: Avoid noise when the same secret reappears in subsequent runs.
    • Correlate data: Match a chat alert to a specific finding in a baseline file or SARIF report using the kingfisher.fingerprint key.
    • Manage triage: Build per-finding tickets or threads keyed by the fingerprint.
  11. Use `visible: false` to hide non-secret findings

    main
    Set visible: false in a rule definition to prevent Kingfisher from reporting that specific match in the final scan report. This is ideal for rules that match metadata, usernames, or application IDs that are required for validating a real secret via depends_on_rule, but are not secrets themselves.