ingest

The code compressor is designed to extract structural information from source code while omitting detailed implementation bodies. This process reduces code volume while retaining essential context for developers or AI agents.

Key information extracted includes:

• Imports and package definitions
• Function, method, and class signatures
• Type definitions
• Comments

The compressor uses Tree-sitter to parse source code into an Abstract Syntax Tree (AST) and executes queries to identify and capture these specific structural elements.

The compressor handles several JavaScript constructs by identifying specific patterns via Tree-sitter queries.

Currently Supported:

• Imports and comments
• Method definitions
• Exported named functions, classes, and generators (e.g., export function foo() {})
• Default exported named functions, classes, and generators (e.g., export default function foo() {})
• Standalone named functions, classes, and generators

Known Limitations/Refinements in Progress:

• Arrow Functions: Body stripping for arrow functions assigned to variables (e.g., const myArrow = () => { ... }) is a target for refinement.
• Anonymous Default Exports: Handling of anonymous default exported functions/classes (e.g., export default function() { ... }) is being clarified to ensure bodies are stripped correctly.

The --compress flag enables code compression using Tree-sitter. This reduces the token count of the generated prompt by intelligently condensing code while preserving structural meaning, making it more efficient for LLM context windows.

# Ingest with compression to save tokens
ingest ./project --compress

The ingest CLI tool generates a single markdown file containing a source tree and the contents of files within a directory or specific files. This output is designed to be used as a prompt for Large Language Models (LLMs). It supports local files, directories, web URLs, and git diffs.

# Ingest the current directory
ingest

# Ingest a specific directory and save to a file
ingest ./src -o prompt.md

# Ingest a directory with code compression enabled
ingest ./src --compress

By using the --web flag or providing a URL as an argument, ingest can crawl web pages and include their content in the prompt. You can control the depth and scope of the crawl.

# Crawl a website with a specific depth and domain restriction
ingest https://example.com --web --web-depth 2 --web-domains example.com

# Crawl a single URL
ingest https://example.com/docs

You can include version control information to provide context about changes to the LLM.

• --diff or -d: Include the current git diff for the path.
• --git-diff-branch: Generate a git diff between two specific branches (format: branch1,branch2).
• --git-log-branch: Retrieve the git log between two specific branches (format: branch1,branch2).

# Include current changes in the prompt
ingest . -d

# Include diff between main and feature-branch
ingest . --git-diff-branch "main,feature-branch"

The --llm flag enables sending the generated prompt directly to an OpenAI-compatible API for inference. This requires configuring your LLM settings (BaseURL, Model, AuthToken) via the application's configuration file.

# Send the generated prompt to your configured LLM endpoint
ingest . --llm

The --vram flag allows you to estimate if the generated prompt (and the model itself) will fit in your available GPU memory. This is useful for local LLM deployment planning.

# Estimate vRAM for a specific model and context length
# --model: Model ID
# --context: Desired context length
# --memory: Available vRAM in GB
# --quant: Quantization type (e.g., q4_k_m)

ingest . --vram --model "llama3" --context 8192 --memory 12.0 --quant "q4_k_m"

You can control which files are included in the generated prompt using glob patterns.

• --exclude or -e: Patterns to exclude from the ingestion.
• --include or -i: Patterns to include (takes priority over excludes if --include-priority is set).
• --exclude-from-tree: Exclude files/folders from the source tree representation itself.
• --no-default-excludes: Disable the tool's built-in default exclusion patterns.
• --pattern-exclude: Path to a specific .glob file containing exclude patterns.

# Exclude all .log and node_modules files
ingest . -e "*.log" -e "node_modules"

# Use a custom glob file for exclusions
ingest . --pattern-exclude ./my-excludes.glob

The compressor utilizes smacker/go-tree-sitter and is designed to support a wide range of languages. While development focuses on Go, Python, and JavaScript, the underlying engine supports the following languages:

bash
c
cpp
csharp
css
cue
dockerfile
elixir
elm
golang
groovy
hcl
html
java
javascript
kotlin
lua
markdown
ocaml
php
protobuf
python
ruby
rust
scala
sql
svelte
swift
toml
typescript
yaml

Flags for controlling how the generated prompt is displayed or saved.

#!/bin/bash
# Output Flags
--output, -o <path>       Optional output file path
--json                    Print output as JSON
--save, -s                Automatically save to ~/ingest/<dirname>.md
--no-clipboard, -n         Disable copying to clipboard
--relative-paths          Use relative paths instead of absolute paths
--no-codeblock            Disable wrapping code inside markdown code blocks
--tokens                  Display the token count of the generated prompt
--line-number, -l         Add line numbers to the source code

# Prompt Customization
--prompt, -p <suffix>      Prompt suffix to append to the generated content
--template, -t <path>     Path to a custom Handlebars template

Flags for configuring the web crawler behavior.

#!/bin/bash
# Web Crawler Flags
--web                     Enable web crawling mode
--web-depth <int>         Maximum crawling depth (default: 1)
--web-domains <list>      Allowed domains for web crawling
--web-timeout <int>       Timeout in seconds (default: 120)
--web-concurrent <int>    Number of concurrent crawling jobs (default: 6)