DOCSEARCHbyAlgolia
Submit Docs

CredSweeper

repository·main·Indexed 42 days ago

https://github.com/samsung/credsweeper

An advanced credential detection tool that identifies exposed secrets such as passwords, API keys, and tokens in source code, configuration files, documents, and binary assets. It utilizes a combination of pattern matching and machine learning to reduce false positives, featuring a customizable rule system, ML-based validation via MlValidator, and support for multiple content providers.

Tokens
14.4K
Snippets
48
Records
72
Agent score
53%

What's inside CredSweeper

  1. Overview of the credsweeper.ml_model.features package

    main
    The credsweeper.ml_model.features package contains the feature extraction logic used by CredSweeper's machine learning models. It provides various submodules that implement specific feature engineering techniques, such as entropy evaluation, pattern matching in attributes, and structural analysis of credentials (e.g., checking for HTML tags, file extensions, or numeric secrets). These features are used to characterize potential credentials for classification.

  2. Overview of CredSweeper

    main

    CredSweeper is a tool designed to detect credentials within directories or files to prevent unwanted exposure of sensitive information. It scans lines of code/text, applies filters, and optionally uses an AI model to identify potential credentials. For every match, CredSweeper reports:

    • The specific line containing the credential
    • The location of the line
    • The expected type of the credential (e.g., personal information, tokens, passwords, API keys, etc.)

  3. Overview of the credsweeper.deep_scanner package

    main
    The credsweeper.deep_scanner package provides a collection of specialized scanners designed to perform deep inspection of various file formats to identify sensitive credentials. The package is organized into submodules, each dedicated to a specific file type or data encoding, allowing for granular scanning of complex or compressed data structures.

  4. Overview of the credsweeper.filters package

    main
    The credsweeper.filters package provides a comprehensive suite of filtering modules used to identify or exclude specific patterns, values, and data types during credential sweeping. Filters are categorized into line-based checks (inspecting entire lines) and value-based checks (inspecting specific extracted values).

  5. Use CredSweeper group filters for credential detection

    main

    The credsweeper.filters.group package provides specialized filter modules designed to group related detection logic. These filters are used to identify credentials by combining multiple detection strategies (keywords, patterns, or specific token formats) into a single logical unit.

    Available filter submodules include:

    • general_keyword: Filters based on general keyword matching.
    • general_pattern: Filters based on general regex patterns.
    • password_keyword: Filters specifically targeting password-related keywords.
    • token_pattern: Filters targeting specific token structures.
    • url_credentials_group: Groups logic for detecting credentials embedded in URLs.
    • weird_base36_token: Specialized detection for unusual Base36 encoded tokens.
    • weird_base64_token: Specialized detection for unusual Base64 encoded tokens.
    • group: The core module for managing and composing these grouped filters.

  6. Understand the three scan types

    main

    The scanning method used depends on the Rule type assigned during rule generation:

    1. SinglePattern: Used when a rule has exactly 1 pattern. It checks if the single-line pattern exists within the line.
    2. MultiPattern: Used when a rule has 2 patterns. It checks if a line is part of a multi-line credential and verifies if the remaining part of the pattern exists within 10 lines below the current line.
    3. PEMKeyPattern: Used when the rule type is pem_key. It checks if a line has high entropy and contains no substring with 5 identical consecutive characters (e.g., 'AAAAA').

  7. Configure ML Model Sensitivity

    main

    CredSweeper uses an ML model classifier to reduce False Positives. You can adjust the sensitivity using the --ml_threshold argument:

    • Higher threshold: Decreases the number of alerts (fewer False Positives, but potentially more False Negatives).
    • Lower threshold: Increases the number of alerts (more False Positives).
    • Setting --ml_threshold 0: Turns ML off and maximizes the number of alerts.

    Allowed values are floats between 0 and 1, or the strings: lowest, low, medium, high, highest (default is medium).

  8. How CredSweeper's overall architecture works

    main

    CredSweeper operates through a four-stage pipeline to detect credentials in files:

    1. Pre-processing: Files are gathered from provided paths. Files are filtered out based on exclusion rules defined in config.json (patterns, specific extensions, or paths).
    2. Scan: The engine performs scanning based on Rule definitions. Depending on the rule type, it uses one of three scanning methods: SinglePattern, MultiPattern, or PEMKeyPattern.
    3. Filter: Detected candidates are passed through a series of filters. If a candidate matches a filter, it is removed from the results to reduce false positives. There are 21 filters organized into 4 filter groups.
    4. ML Validation: By default, a Bidirectional LSTM model combined with handcrafted features validates candidates. It calculates a probability (0-1) that the line is a real credential, which is then compared against a user-defined threshold.

  12. Run CredSweeper scans

    main

    Execute CredSweeper as a Python module. You can specify a target file or directory using the --path flag and save the results to a JSON file using the --save-json flag.

    python -m credsweeper --path tests/samples/password.gradle --save-json output.json