Nosey Parker Documentation

repository·main·Indexed 25 days ago

https://github.com/praetorian-inc/noseyparker

A high-performance CLI tool for finding secrets and sensitive information in textual data, including files, directories, and Git history. Optimized for offensive security, it features field-tested rules, match deduplication, and support for scanning GitHub users and organizations. The tool includes a data model based on datastores, blobs, and findings, and provides commands for scanning, summarizing, and reporting results.

Tokens
26.2K
Snippets
18
Records
204
Agent score
77%

What's inside Nosey Parker

  1. Third-party integrations for Nosey Parker

    main

    Nosey Parker integrates with several third-party tools and platforms:

  2. How Nosey Parker works: Terminology and Data Model

    main

    Understanding the Nosey Parker data model is essential for using the tool effectively:

    • Datastore: A directory used to record findings and maintain internal state. It is implicitly created by the scan command if it doesn't exist.
    • Blob: An individual scanned input (e.g., a file). Each has a unique SHA-1 blob ID.
    • Provenance: Metadata describing how a blob was discovered (e.g., a filesystem path or Git history entry).
    • Rule: A regular expression with at least one capture group used to isolate secret content.
    • Ruleset: A collection of rules (the default ruleset is optimized for secret detection).
    • Match: A specific instance where a rule's pattern matches an input, defined by the rule, blob ID, and byte offsets.
    • Finding: The top-level unit of reporting. A finding is a group of matches that share the same rule and capture groups (deduplicated matches).
  3. Structure of a Nosey Parker Rule

    main

    Nosey Parker rules are defined in YAML syntax. A rules file must contain a top-level rules field which is a list of rule objects.

    Each rule object contains the following fields:

    • name: A string for human consumption, used in reports.
    • id: A globally unique identifier (max 20 characters, alphanumeric segments separated by hyphens or periods).
    • pattern: The regular expression used for matching. Must include at least one capture group to isolate the secret.
    • description: (Optional) A string explaining what was detected and the potential impact.
    • references: A list of strings (typically URLs) describing the format being matched.
    • examples: A list of strings that are asserted to match the rule. Used for automated testing.
    • negative_examples: (Optional) A list of strings asserted not to match the rule. Used for automated testing.
    • categories: A list of freeform strings used to classify the rule (e.g., secret, api, fuzzy, hashed).
    - name: Jenkins Setup Admin Password
      id: np.jenkins.2
    
      pattern: |
        (?x)(?m)
        Please\ use\ the\ following\ password\ to\ proceed\ to\ installation:
        (?: \n\n | \r\n\r\n )
        ([a-f0-9]{30,36})$
    
      examples:
      - |
          *************************************************************
          *************************************************************
          *************************************************************
    
          Jenkins initial setup is required. An admin user has been created and a password generated.
          Please use the following password to proceed to installation:
    
          bd9627decc6346d780b3b6ab6ea8fe1f
    
          This may also be found at: /root/.jenkins/secrets/initialAdminPassword
    
          *************************************************************
          *************************************************************
          *************************************************************
    
      categories: [fuzzy, secret]
    
      description: |
        A Jenkins setup wizard admin user password was detected.
        This password is used to configure a new Jenkins installation.
        An attacker with possession of this password could control the Jenkins instance.
        This could enable exfiltration of proprietary code, insertion of backdoors, or lateral movement to other resources.
    
      references:
      - https://www.jenkins.io/doc/book/installing/linux/#setup-wizard
  4. Enumerate GitHub repositories

    main

    The github repos list command uses the GitHub REST API to list URLs for repositories belonging to a specific user or organization. This is useful for building target lists for scanning.

    Set the NP_GITHUB_TOKEN environment variable to improve rate limits.

    Example:

    $ noseyparker github repos list --user octocat

    Use the --format=FORMAT option to output in json or jsonl.

    noseyparker github repos list --user octocat
  5. Scan GitHub users or organizations

    main

    You can scan all accessible repositories for a specific GitHub user or organization using the --github-user or --github-org flags.

    To increase API rate limits and access more repositories, set the NP_GITHUB_TOKEN environment variable with a valid GitHub access token.

    Example:

    $ noseyparker scan --datastore np.noseyparker --github-user octocat
    noseyparker scan --datastore np.noseyparker --github-user octocat
  6. Use the noseyparker-github-repos CLI

    main

    The noseyparker-github-repos command is used to interact with GitHub repositories. It supports various global options for API configuration, output formatting, and performance tuning, and uses subcommands to perform specific tasks like listing repositories.

    Basic Usage

    noseyparker github repos [options] <subcommand>

    Available Subcommands

    • noseyparker-github-repos-list: List repositories belonging to a specific user or organization.
  7. Generate usage example GIFs with vhs

    main

    To reproduce the usage GIFs shown in the repository, you must use vhs (a console script runner and recording tool) to execute the fragments located in the examples/ directory. The process involves combining individual vhs fragments with common settings via the record-examples.zsh script.

    Prerequisites

    • Install vhs
    • Install noseyparker

    Steps

    1. Ensure vhs and noseyparker are installed.
    2. Run the recording script from the root of the repository:
      ./record-examples.zsh

    Generated GIFs will be stored in the gifs/ directory.

    ./record-examples.zsh
  8. Scan files, directories, and Git repositories

    main

    Use the scan command to search for secrets in local filesystem content or Git repositories. You can specify a datastore directory with -d or --datastore to save findings.

    Note for Docker users: When running in Docker, you must mount your local directory to /scan to use relative paths and enable tab completion:

    docker run -v "$PWD":/scan ghcr.io/praetorian-inc/noseyparker:latest <ARGS>

    Examples:

    Scan a local Git repository:

    $ noseyparker scan -d cpython.np cpython.git

    Scan a Git repository via HTTPS URL:

    $ noseyparker scan --datastore np.noseyparker --git-url https://github.com/praetorian-inc/noseyparker
    noseyparker scan -d cpython.np cpython.git
  9. Authenticate with GitHub using a Personal Access Token

    main

    By default, noseyparker-github uses unauthenticated access. To increase rate limits and access additional content, provide a personal access token by setting the NP_GITHUB_TOKEN environment variable.

    Note: This documentation is from version 0.17.0. While the core mechanism of using environment variables for authentication is common in CLI tools, users should verify compatibility with the current version (v0.24.0) as Nosey Parker is officially retired and replaced by Titus.

  10. Install Nosey Parker

    main

    Nosey Parker can be installed via several methods depending on your operating system:

    • macOS (Homebrew): brew install noseyparker
    • Linux (Prebuilt Binaries): Download from the latest release page (supports x86_64/aarch64).
    • Linux (Arch Linux): Available via the AUR.
    • Docker:
      • Standard: docker pull ghcr.io/praetorian-inc/noseyparker:latest
      • Alpine base: docker pull ghcr.io/praetorian-inc/noseyparker-alpine:latest
    • Windows: Not natively supported, but can be run via WSL1 using the Linux release.

    Building from source: Requires cargo, cmake, boost (>=1.57), git, patch, pkg-config, sha256sum, and zsh. Use the provided build script:

    $ rm -rf release && ./scripts/create-release.zsh

    Artifacts will be located in the release/bin/noseyparker directory.

    brew install noseyparker
  11. Install pre-commit hooks for Nosey Parker development

    main

    If you are contributing to the Nosey Parker repository, you can install the enabled pre-commit hooks to detect simple errors locally. Ensure you have pre-commit installed on your system before running the command.

    $ pre-commit install