DOCSEARCHbyAlgolia
Submit Docs

Gitingest

repository·main·Indexed 12 days ago

https://github.com/coderamp-labs/gitingest

A CLI tool and Python package (v0.3.1) that converts local or remote Git repositories into prompt-friendly text digests optimized for Large Language Models (LLMs). It provides structured context including file trees, statistics, and file contents. Gitingest can be used via CLI, as a Python library, through browser extensions, or self-hosted using Docker and Docker Compose.

Tokens
6.8K
Snippets
27
Records
36
Agent score
95%

What's inside Gitingest

  1. Self-host Gitingest using Docker Compose

    main

    The project provides a compose.yml file with profiles for development and production.

    Development Mode

    Uses the dev profile. It enables debug mode, mounts source code for live development, and includes a minio service for S3-compatible local storage.

    MinIO Local Credentials (Default):

    • Username: minioadmin
    • Password: minioadmin
    • API Port: 9000
    • Web Console Port: 9001

    Production Mode

    Uses the prod profile. It is configured for stable operation with restart: unless-stopped and sets the Sentry environment to production.

    # Run in development mode
docker compose --profile dev up

# Run in production mode (detached)
docker compose --profile prod up -d

# Build and run in production mode
docker compose --profile prod build
docker compose --profile prod up -d

  2. Self-host Gitingest using Docker

    main

    You can run Gitingest as a standalone container. After building the image, run it to make the application available at http://localhost:8000.

    If hosting on a custom domain, configure the ALLOWED_HOSTS environment variable to permit access from your domain.

    # Build the image
docker build -t gitingest .

# Run the container
docker run -d --name gitingest -p 8000:8000 gitingest

# Example with custom allowed hosts
ALLOWED_HOSTS="example.com, localhost, 127.0.0.1" docker run -d --name gitingest -p 8000:8000 gitingest

  4. Install Gitingest

    main

    Gitingest can be installed via pip or pipx.

    Use pip install gitingest for standard installation. If you plan to self-host, install with server dependencies using pip install gitingest[server].

    For a cleaner environment, it is recommended to use pipx:

    # If using pipx for the first time
pipx ensurepath

# Install gitingest
pipx install gitingest
    pip install gitingest

  5. Configure Gitingest via Environment Variables

    main

    Gitingest is configurable through several environment variables covering host permissions, metrics, error tracking (Sentry), and S3 storage settings.

    Host Configuration

    • ALLOWED_HOSTS: Comma-separated list of allowed hostnames (default: "gitingest.com, *.gitingest.com, localhost, 127.0.0.1").

    Metrics (Prometheus)

    • GITINGEST_METRICS_ENABLED: Set to any value to enable the Prometheus metrics server.
    • GITINGEST_METRICS_HOST: Host for the metrics server (default: 127.0.0.1).
    • GITINGEST_METRICS_PORT: Port for the metrics server (default: 9090).

    Error Tracking (Sentry)

    • GITINGEST_SENTRY_ENABLED: Set to any value to enable Sentry.
    • GITINGEST_SENTRY_DSN: Required if Sentry is enabled.
    • GITINGEST_SENTRY_TRACES_SAMPLE_RATE: Sampling rate for performance data (default: 1.0, range: 0.0-1.0).
    • GITINGEST_SENTRY_PROFILE_SESSION_SAMPLE_RATE: Sampling rate for profile sessions (default: 1.0, range: 0.0-1.0).
    • GITINGEST_SENTRY_PROFILE_LIFECYCLE: Profile lifecycle mode (default: trace).
    • GITINGEST_SENTRY_SEND_DEFAULT_PII: Send default personally identifiable information (default: true).

    S3 Storage

    • S3_ALIAS_HOST: Public URL/CDN for accessing S3 resources (default: 127.0.0.1:9000/gitingest-bucket).
    • S3_DIRECTORY_PREFIX: Optional prefix for S3 file paths.

  6. Run Gitingest using Docker Compose profiles

    main

    The project uses Docker Compose profiles to separate production and development environments.

    • To run the production service, use the prod profile.
    • To run the development service (including MinIO for S3 emulation and hot-reloading), use the dev profile.
    # Example command to run development profile
docker compose --profile dev up

# Example command to run production profile
docker compose --profile prod up

  7. Use Gitingest as a Python package

    main

    You can import gitingest into your Python projects to programmatically generate digests.

    Synchronous Usage

    Use ingest(path_or_url, ...) which returns a tuple of (summary, tree, content).

    Asynchronous Usage

    Use ingest_async(path_or_url, ...) for async environments. In Jupyter notebooks, you can await this function directly.

    Key Parameters

    • path_or_url: A string representing the local path or a Git URL.
    • token: A string for accessing private repositories.
    • include_submodules: Boolean to include submodules.
    • output: Argument to enable file writing (disabled by default in Python usage).

    Authentication

    You can pass a token directly to the function or set the GITHUB_TOKEN environment variable.

    from gitingest import ingest

# Synchronous usage
summary, tree, content = ingest("https://github.com/coderamp-labs/gitingest")

# Asynchronous usage
import asyncio
from gitingest import ingest_async

result = asyncio.run(ingest_async("path/to/directory"))

  8. Configure allowed hosts middleware

    main

    To prevent host header attacks, Gitingest uses TrustedHostMiddleware. You can define the allowed hosts via the ALLOWED_HOSTS environment variable as a comma-separated list. If not provided, the application defaults to:

    gitingest.com, *.gitingest.com, localhost, and 127.0.0.1.

  9. Configure S3 environment variables

    main

    Gitingest uses environment variables to configure S3 connectivity and behavior. Ensure the following variables are set in your environment:

    VariableDescription
    S3_ENABLEDSet to true to enable S3 features. Defaults to false.
    S3_ENDPOINTThe S3 endpoint URL (e.g., for MinIO or custom S3 providers).
    S3_ACCESS_KEYAWS access key ID.
    S3_SECRET_KEYAWS secret access key.
    S3_REGIONAWS region (defaults to us-east-1 if AWS_REGION is not set).
    S3_BUCKET_NAMEThe name of the S3 bucket (defaults to gitingest-bucket).
    S3_ALIAS_HOSTA custom host used to generate public URLs (e.g., a CDN domain).
    S3_DIRECTORY_PREFIXAn optional prefix to prepend to all S3 file paths.

  10. Configure Sentry for error tracking

    main

    Gitingest supports Sentry for error tracking and performance monitoring. You can enable and configure it using the following environment variables:

    • GITINGEST_SENTRY_ENABLED: Set to any non-null value to enable Sentry.
    • GITINGEST_SENTRY_DSN: The Sentry Data Source Name (DSN).
    • GITINGEST_SENTRY_TRACES_SAMPLE_RATE: Float value for transaction tracing (default: 1.0).
    • GITINGEST_SENTRY_PROFILE_SESSION_SAMPLE_RATE: Float value for session profiling (default: 1.0).
    • GITINGEST_SENTRY_PROFILE_LIFECYCLE: Set to manual or trace (default: trace).
    • GITINGEST_SENTRY_SEND_DEFAULT_PII: Boolean string (true or false) to include PII (default: true).
    • GITINGEST_SENTRY_ENVIRONMENT: The name of the deployment environment.

  11. Configure metrics server

    main

    If enabled, Gitingest starts a metrics server in a separate background thread. Use these environment variables to configure it:

    • GITINGEST_METRICS_ENABLED: Set to any non-null value to enable the metrics server.
    • GITINGEST_METRICS_HOST: The host address for the metrics server (default: 127.0.0.1).
    • GITINGEST_METRICS_PORT: The port for the metrics server (default: 9090).

  12. Handle GitHub authentication errors with InvalidGitHubTokenError

    main

    If you encounter an InvalidGitHubTokenError, it means the GitHub Personal Access Token provided is malformed. To resolve this, generate a new token at the following URL and ensure it is correctly configured as an environment variable or passed to the application:

    https://github.com/settings/tokens/new?description=gitingest&scopes=repo

    raise InvalidGitHubTokenError()