DOCSEARCHbyAlgolia
Submit Docs

pgit

repository·main·Indexed 41 days ago

https://github.com/imgajeed76/pgit

A Git-like CLI that imports Git repositories into a PostgreSQL database using the pg-xpatch extension. It enables deep repository analysis through SQL queries and pre-built analytics such as churn, coupling, and bus-factor across the entire commit history.

Tokens
30K
Snippets
129
Records
159
Agent score
51%

What's inside pgit

  1. What is pgit?

    main

    pgit is a Git-like version control CLI that stores repository data in PostgreSQL instead of on the filesystem. It allows you to import a Git repository and then use SQL, pre-built analyses, or cross-history search to interrogate the codebase's history.

    Key capabilities include:

    • Importing Git repos: Pull full history (merges, renames, author dates) into Postgres.
    • Pre-built analyses: Run commands for churn, coupling, hotspots, authors, activity, and bus-factor without writing SQL.
    • Raw SQL access: Query the underlying tables directly for custom analysis.
    • Cross-history search: Perform grep-like searches across every version of every file in the history, not just the current checkout.

    Note on Workflow: pgit is designed for repository analysis rather than daily development. While it supports standard Git commands like init, add, commit, log, diff, checkout, and blame, the imported history is append-only and immutable. The primary workflow is import followed by analyze, sql, or search.

  2. Use pgit analyze subcommands for repository analysis

    main

    The pgit analyze command provides six pre-built analyses to answer common questions about a codebase's history. Each subcommand is optimized for pgit's storage to avoid manual delta chain processing.

    Available subcommands:

    • churn: Identifies files changed in the most commits (maintenance hotspots).
    • coupling: Finds file pairs that change together (dependency/abstraction hints).
    • hotspots: Aggregates churn by directory levels.
    • authors: Ranks contributors by commit count and activity span.
    • activity: Shows commit volume over time (timeline).
    • bus-factor: Identifies files touched by only a few authors (knowledge silos).
    pgit analyze <subcommand> [flags]

  3. Understand the two configuration scopes in pgit

    main

    pgit uses two distinct configuration layers:

    1. Repository config: Stored in .pgit/config.toml within each repository. It manages identity (name/email) and remotes.
    2. Global config: Stored in your user config directory (e.g., ~/.config/pgit/config.toml on Linux, ~/Library/Application Support/pgit/config.toml on macOS). It manages the PostgreSQL container settings and import defaults shared across all repositories.

    It is recommended to use the pgit config command rather than editing these files manually.

  4. Understand the performance characteristics of pgit analyses

    main

    The performance of pgit analyses depends on whether they touch heap tables or xpatch tables:

    • Instant Analyses: Tools like churn, coupling, and hotspots only touch pgit_file_refs and pgit_paths. Since these are heap tables, they are extremely fast even with millions of versions because no content decompression is required.
    • Streaming Analyses: Tools like authors, activity, and bus-factor must read commit metadata from pgit_commits (an xpatch table). These must stream the table front-to-back (ORDER BY seq ASC) to efficiently reuse decompressed data from the delta chain.
    • Ancestry Queries: Operations like HEAD~50 or checking if X is an ancestor of Y use the pgit_commit_graph (a small heap table) and do not require decoding the heavy commit metadata chain.

  5. Optimize SQL query performance for pgit

    main

    pgit uses two types of tables. Understanding the difference is critical for performance:

    1. Heap Tables (pgit_paths, pgit_file_refs, pgit_commit_graph, pgit_refs, pgit_metadata, pgit_sync_state): Standard PostgreSQL tables. These are fast for filtering, joining, and aggregating. Always try to get as much information as possible from these first.
    2. xpatch Tables (pgit_commits, pgit_text_content, pgit_binary_content): These store delta chains. Reading them involves decompression costs.

    Performance Rules for xpatch Tables

    • Constrain group_id: Always include group_id in your WHERE clause to avoid scanning every group.
    • Use Primary Keys: Lookups using (group_id, version_id) are very efficient.
    • Read Front-to-Back: Use ORDER BY version_id ASC to allow the engine to reuse decompression state. Use LIMIT to prevent pulling entire chains.
    • Avoid OFFSET: It forces decompression and discarding of rows. Use LIMIT and handle skipping in your application.
    • Avoid Range/Set Filters on version_id: Using version_id > x or version_id = ANY(...) can trigger expensive bitmap scans.

    Performance Rules for pgit_commits

    pgit_commits is a single large delta chain ordered by seq.

    • Scan with ORDER BY seq ASC for multi-commit reads.
    • Avoid JOINing onto pgit_commits: Joins can trigger a full sequential scan of the chain. Instead, perform a two-step lookup: fetch the ID from a heap table first, then fetch the specific commit by ID.

    Example of efficient two-step lookup:

    -- Step 1: Get ID from heap table
SELECT commit_id FROM pgit_refs WHERE name = 'HEAD';

-- Step 2: Fetch specific commit by PK
SELECT * FROM pgit_commits WHERE id = <id_from_step_1>;

  6. Understand pgit storage overhead and compression methodology

    main

    The pgit-bench tool compares the storage efficiency of pgit against standard Git storage methods (Normal and Aggressive).

    Storage Metrics Definitions

    • Raw Uncompressed Size: The sum of all Git object sizes, calculated via git cat-file --batch-all-objects --batch-check='%(objecttype) %(objectsize)'. This serves as the baseline for compression ratios.
    • Git Storage: Measured by the size of the .pack file after running git gc (Normal) or git gc --aggressive (Aggressive). Note that .idx, .rev, and .bitmap files are excluded as they are reconstructable.
    • pgit Storage (On-disk): The sum of pg_table_size() across all pgit_* tables, including heap and TOAST, but excluding indexes.
    • pgit Storage (Actual Data): The size of the data after stripping PostgreSQL overhead. It uses xpatch.stats().compressed_size_bytes for xpatch tables and SUM(octet_length(columns)) for normal tables.
    • Compression Ratio: Calculated as raw_uncompressed / stored_size. A higher ratio indicates better compression.

    PostgreSQL Overhead Factors

    When using pgit, the storage footprint includes PostgreSQL-specific overhead such as:

    • Tuple headers (23 bytes per row)
    • TOAST chunk metadata
    • Page headers
    • Alignment padding

  7. How pgit architecture works

    main

    pgit operates using three primary components:

    1. The CLI (pgit): A single Go binary with no runtime dependencies.
    2. A local PostgreSQL container: A container running the pg-xpatch image (requires Docker or Podman). This container hosts multiple pgit repositories, with each repository residing in its own database.
    3. pg-xpatch: A PostgreSQL extension (table access method) that provides delta-compression for versioned content, allowing many versions of a file to be stored efficiently.

    Connectivity: A PostgreSQL connection URL acts as your "remote." Access to a pg-xpatch database allows you to perform push, pull, and clone operations; there is no separate authentication system.

  8. Understand the pgit storage model: Heap vs Xpatch tables

    main

    pgit splits its schema into two types of tables to balance performance and storage efficiency:

    • Heap tables: Standard PostgreSQL tables used for structure and metadata. These are small, fast to scan, and ideal for filtering, joining, or counting. They include tables like pgit_commit_graph, pgit_paths, pgit_file_refs, and pgit_refs.
    • Xpatch tables: Tables using the USING xpatch access method. These hold bulky, repetitive content (like commit messages or file bodies) and use delta-compression to save space. They include pgit_commits, pgit_text_content, and pgit_binary_content.

    Performance Tip: When writing SQL, perform your filtering and joining on heap tables to keep queries fast. Avoid full scans of xpatch tables unless you specifically need the content, as they require decompressing delta chains.

  9. Understand pgit compression performance vs git

    main

    The pgit-bench report compares the storage efficiency of pgit against standard git (normal and aggressive settings).

    Key metrics used in the comparison:

    • Stored size: The total size on disk.
    • Compression ratio: The ratio of uncompressed data to stored size.
    • pgit (on-disk): The total storage footprint of the pgit repository, including PostgreSQL overhead.
    • pgit (actual data): The storage footprint of the actual repository data, excluding PostgreSQL overhead.
    • PostgreSQL overhead: The additional storage required by the PostgreSQL database engine to manage the pgit data.

    In many cases (e.g., serde, ripgrep, fzf), pgit achieves a higher actual data compression ratio than git aggressive, even if the total on-disk size is larger due to PostgreSQL overhead.

  10. Determine when pgit is more efficient than git

    main

    The efficiency of pgit's delta compression compared to git's packfiles depends on the nature of your repository's changes:

    pgit is typically better for:

    • Source code with frequent, incremental, line-level edits.
    • Long histories of the same files.
    • Repositories with many renames and copies (which benefit from shared delta chains).

    git is typically better for:

    • Repositories dominated by large binary assets.
    • Repositories with large vendored dependencies.
    • Repositories with frequent minified or generated output.
    • Repositories with frequent wholesale reformatting (where consecutive versions are not similar).

  11. How delta groups optimize storage for renames and copies

    main

    Instead of giving every file path its own delta chain, pgit uses delta groups to handle renames, copies, and file splits efficiently.

    During import, pgit performs a union-find pass over content hashes. Any two paths that have ever held byte-identical content are merged into the same group_id.

    • Mechanism: The pgit_paths table maps many paths to a single group_id (an N:1 mapping).
    • Storage: Content tables (pgit_text_content, pgit_binary_content) are keyed by (group_id, version_id). This allows a single delta chain to span across renames (e.g., moving a file from drivers/char/ to drivers/gpu/).
    • Hashing: Content is hashed using BLAKE3 (truncated to 16 bytes). Empty files are excluded from grouping.

  12. What a pgit remote is

    main
    A pgit remote is any PostgreSQL database running the pg-xpatch extension. There is no separate account system or transport protocol; the remote is defined by a standard PostgreSQL connection URL (e.g., postgres://user:pass@host:5432/dbname). Authentication is handled via the credentials provided in the URL. If you can reach the database, you can sync with it.