DOCSEARCHbyAlgolia
Submit Docs

hypergrep

repository·master·Indexed 42 days ago

https://github.com/p-ranav/hypergrep

A high-performance, regex-based search tool optimized with Intel Hyperscan, multi-threading, and SIMD. It features specialized git repository searching via libgit2 to iterate through the git index instead of manual file scanning. Designed to compete with ripgrep, ag, and ugrep, it supports UTF-8, Unicode properties, and memory-mapped parallel searching for files exceeding 1 MiB.

Tokens
5.7K
Snippets
17
Records
23
Agent score
43%

What's inside hypergrep

  1. Overview of hypergrep features

    master

    hypergrep is a high-performance search tool designed for recursive regex pattern searching. It is optimized for speed using C++17, multi-threading, and SIMD instructions.

    Key Features:

    • Regex Search: Uses Intel Hyperscan for high-speed pattern matching.
    • Git Integration: When a git repository is detected, it uses libgit2 to search the repository index instead of scanning files manually.
    • Performance: Designed to compete with and outperform tools like ripgrep, ag (The Silver Searcher), and ugrep.
    • Platform Support: Currently tested on Linux; not explicitly cross-platform.

  2. Use Unicode and Unicode properties

    master

    The hypergrep regex engine supports UTF-8. You can use Unicode character properties like \p{L}, \P{Sc}, or \p{Greek} in your patterns.

    By default, character mnemonics like \w and \s use ASCII interpretations. To use Unicode properties for these mnemonics and POSIX character classes, use the --ucp flag.

    hgrep --ucp '\p{Greek}' path

  3. Understand hypergrep search constraints and behaviors

    master

    To manage performance and output clarity, hypergrep follows these internal rules:

    • Chunking: Files are searched in 64 KiB chunks. If a chunk contains no newlines (e.g., a minified JS file), the file is skipped.
    • Large File Threshold: Any file exceeding 1 MiB is treated as a "large" file, triggering memory mapping and multi-threaded parallel searching.
    • Line Length Limit: Lines longer than 4096 bytes are omitted from the output to keep results clean, though the number of matches within those lines will still be reported.
    • Whitespace Trimming: The --trim flag specifically targets ' ' (0x20) and ' ' (0x09) at the start of matching lines.

  4. How hypergrep handles Git repositories

    master

    When hypergrep encounters a directory containing a .git/ subdirectory, it treats it as a Git repository rather than an ordinary directory. Instead of traversing the file tree and checking against .gitignore rules, it uses [libgit2] to iterate through the entries in the .git/index file. This approach is more efficient as it avoids manually checking every file against ignore rules.

    Key Behaviors:

    • Submodules: By default, hypergrep recursively searches any git submodules found. You can disable this using --ignore-submodules.
    • Standard Directory Mode: If you want hypergrep to ignore the Git index and treat the repository as a normal directory (traversing all files/folders), use the --ignore-gitindex flag.
    # Search a git repo normally (using git index)
hgrep 'pattern' .

# Search a git repo as a normal directory (ignoring git index)
hgrep --ignore-gitindex 'pattern' .

# Search a git repo but exclude submodules
hgrep --ignore-submodules 'pattern' .

  5. How hypergrep handles different search targets

    master

    hypergrep is a line-oriented search tool that uses Intel Hyperscan to perform multi-threaded, SIMD-vectorized regex searches. It optimizes its workflow based on the type of input provided:

    1. Git Repositories: If a .git directory is detected, hypergrep uses libgit2 to open the repository and iterate through the git index. This is faster than evaluating .gitignore rules. It also automatically searches submodules unless excluded.
    2. Directories: For non-git directories, hypergrep performs a multi-threaded recursive traversal. If it encounters a nested git repository during this traversal, it switches to the git index method for that subdirectory and stops further manual recursion into it.
    3. Single Large Files: Files are memory-mapped and searched in parallel across multiple threads. Each thread processes a portion of the file and stores results in a local queue, which a consumer thread then processes to determine correct line numbers and output results.

  6. Install dependencies with vcpkg

    master

    Before building hypergrep, you must install its required dependencies using vcpkg. The required libraries are concurrentqueue, fmt, argparse, libgit2, and hyperscan.

    Follow these steps:

    1. Clone the vcpkg repository.
    2. Bootstrap vcpkg.
    3. Install the libraries.
    git clone https://github.com/microsoft/vcpkg
cd vcpkg
./bootstrap-vcpkg.sh
./vcpkg install concurrentqueue fmt argparse libgit2 hyperscan

  7. Build hypergrep using cmake and vcpkg

    master

    To build hypergrep, clone the repository and use cmake with the vcpkg toolchain. The build process differs depending on your cmake version.

    For cmake < 3.19

    Create a build directory and point cmake to your vcpkg toolchain file manually.

    For cmake >= 3.19

    Use the release preset by exporting the VCPKG_ROOT environment variable.

    Build for x86_64 Portability

    To create a more portable binary, use the -DBUILD_PORTABLE=on flag. This enables -march=x86-64 -mtune=generic and links the C++ standard library and GCC runtime statically.

    # For cmake >= 3.19
export VCPKG_ROOT=<path_to_vcpkg>
cmake -B build -S . --preset release
cmake --build build

# For portable x86_64 builds
cmake -DBUILD_PORTABLE=on ...

  8. Perform a simple search with hypergrep

    master

    To search the current working directory for a single pattern, run hypergrep followed by the pattern. If no path is provided, the current directory is searched.

    When piping output to other programs (like wc or cat), the output format changes so that each line represents a single line of output from the search results.

    hgrep mmap

  9. Search using multiple patterns

    master

    You can search for multiple independent patterns using two different methods:

    1. Command line patterns: Use the -e/--regexp option for each pattern you want to include.
    2. Pattern file: Use the -f/--file option to point to a text file containing multiple patterns, with one pattern per line.
    # Using -e for multiple patterns
hgrep -e pattern1 -e pattern2 path

# Using -f with a pattern file
hgrep -f list_of_patterns.txt path

  11. How Hypergrep handles search modes

    master

    Hypergrep switches its input interpretation based on the presence of specific flags:

    1. Pattern + Path Mode (Default): If you do not use -f, --files, or -e, the first argument is the search pattern, and all following arguments are paths.

      • Example: hg "pattern" path/to/dir
      • If only one argument is provided: hg "pattern" (searches current directory).

    2. Path-Only Mode: If you use -f (pattern file), --files, or -e (explicit regex), Hypergrep treats all positional arguments as paths to be searched. If no paths are provided, it defaults to the current directory (.).

      • Example: hg -e "pattern" path/to/dir
      • Example: hg --files path1 path2

    3. Pipe Mode: If stdin is not a TTY (e.g., you pipe data into hypergrep), it enters a specialized mode where it scans the incoming lines using file_search::scan_line instead of searching files on disk.

    # Pattern + Path mode
hg "search_term" ./src

# Path-only mode (all args are paths)
hg -e "search_term" ./src ./docs

# Pipe mode
echo "some text" | hg -e "text"

  12. Hypergrep search command examples

    master

    hypergrep (aliased as hgrep in examples) supports various flags for controlling search behavior. Below are common usage patterns based on performance benchmarks:

    • Count matches: Use -c to count occurrences.
    • Word boundary: Use -w to search for whole words.
    • Case insensitive: Use -i for case-insensitive searches.
    • Line numbers: Use -n to show line numbers.
    • Filtering: Use --filter to restrict searches to specific file extensions.

    Examples:

    # Count occurrences of a pattern
hgrep -c 'pattern' file.txt

# Case-insensitive whole-word search
hgrep -inw 'Sherlock Holmes' en.txt

# Search for specific file types (e.g., C/C++ files)
hgrep --filter "\.(c|cpp|h|hpp)$" test

# Search with regex alternation
hgrep -n 'pattern1|pattern2|pattern3' en.txt