yek

Yek is designed with a modular architecture that separates concerns into distinct layers. The core components include:

• Domain Models: Data structures like ProcessedFile and RepositoryInfo that represent the state of files and repositories.
• Repository Pattern: Abstractions (FileSystem and GitOperations traits) that decouple the logic from the actual file system and Git implementations.
• Processing Pipeline: A middleware-based system where files pass through various ProcessingStage implementations (e.g., discovery, filtering, and formatting).
• Configuration Architecture: A split configuration model consisting of InputConfig, OutputConfig, and ProcessingConfig to manage different aspects of the tool's behavior.

If running Yek in environments with limited RAM, use the following strategies:

1. Enable streaming mode: Avoids loading the entire file set into memory.
2. Use smaller batch sizes: Reduces the amount of data processed in a single step.
3. Enable lazy token counting: Prevents unnecessary computation of token counts until they are actually required.
4. Set memory limits: Explicitly define limits to prevent system instability.

You can install yek using a shell script for Unix-like systems or PowerShell for Windows.

Unix-like Systems (macOS, Linux)

curl -fsSL https://azimi.me/yek.sh | bash

Windows (PowerShell)

irm https://azimi.me/yek.ps1 | iex

Build from Source

If you prefer to build from source using Cargo:

git clone https://github.com/mohsen1/yek
cd yek
cargo install --path .

When processing very large repositories, follow these performance guidelines to ensure efficiency and stability:

1. Use token mode: Provides more accurate size management for LLM contexts.
2. Enable parallel processing: Maximizes CPU utilization.
3. Set memory limits: Use memory_limit_mb to prevent Out-Of-Memory (OOM) errors.
4. Use streaming output: Reduces memory pressure for very large outputs.
5. Optimize ignore patterns: Reduces the overhead of file system traversal.

To use Yek within your own Rust application, you can manually construct a ProcessingContext and run it through a ProcessingPipeline. This gives you programmatic control over the input, output, and processing configurations.

use yek::{models::ProcessedFile, pipeline::ProcessingPipeline, error::YekResult};

// Create processing context
let context = ProcessingContext::new(
    input_config,
    output_config,
    processing_config,
    repository_info,
    Arc::new(RealFileSystem),
    git_operations,
);

// Use the pipeline
let pipeline = ProcessingPipeline::new(context);
let files = pipeline.process()?;

To get detailed information about the file discovery process, processing stage timing, memory usage statistics, cache hit rates, and error details, run Yek with the --debug flag.

yek --debug --input src/ --max-size 10MB

The categorize_file function uses heuristics based on file paths and extensions to assign a FileCategory. The categorization follows a specific order of precedence to ensure accuracy:

1. Test Files: Checked first via directory patterns (e.g., /tests/, /spec/) or naming conventions (e.g., .test.js, test_*.py).
2. Configuration Files: Checked via extensions (e.g., .toml, .yaml, .json) or specific filenames (e.g., package.json, Cargo.toml, .gitignore).
3. Documentation Files: Checked via extensions (e.g., .md, .rst, .txt) or common names (e.g., README, CHANGELOG, LICENSE).
4. Source Files: Checked via common programming language extensions (e.g., .rs, .py, .js, .cpp) or directory patterns (e.g., /src/, /lib/).
5. Other: If no patterns match, the file is categorized as Other.

File priority is determined by combining rule-based priority and Git-based boosts. The FilePriority struct calculates a combined score which is used to order files.

• rule_priority: The base priority assigned by user-defined rules.
• git_boost: A boost value derived from Git history (e.g., recency of commits).
• combined: The sum of rule_priority and git_boost.

yek categorizes files into five distinct types to determine their importance during processing. Each category has a default priority offset used for sorting. Higher offsets indicate higher priority.

Categories and Default Offsets:

• Source: Main application or library source code (Offset: 20)
• Documentation: Documentation files like markdown or rst (Offset: 15)
• Test: Test files and testing related code (Offset: 10)
• Configuration: Configuration files like yaml, toml, or json (Offset: 5)
• Other: Files that do not fit into the above categories (Offset: 1)

The yek CLI tool serializes a repository into a single output file (either .txt or .json). It can operate in two modes:

1. Streaming Mode: Skips checksum calculation and processes files sequentially. Use this when stream is enabled in your configuration.
2. Standard Mode: Runs repository serialization and checksum calculation in parallel. This mode requires an output_dir to be specified if an output_name is not provided, as it generates a filename based on the content checksum (e.g., yek-output-<checksum>.json).

When an output_name is provided, the tool writes the output to that specific name (optionally within an output_dir). If no output_name is provided in standard mode, the tool automatically generates a unique filename using a checksum of the input paths.