DOCSEARCHbyAlgolia
Submit Docs

repo2txt

repository·master·Indexed 23 days ago

https://github.com/abinthomasonline/repo2txt

A browser-based tool that converts GitHub repositories, local directories, or zip files into a single plain text format optimized for LLM prompts. Version 2.0.0-beta.1 features 100% client-side processing, .gitignore support, and token counting via Web Workers. It supports multiple source providers including GitHub, Local, GitLab (Beta), and Azure DevOps (Beta).

Tokens
13.4K
Snippets
26
Records
84
Agent score
75%

What's inside repo2txt

  1. Overview of repo2txt features

    master

    repo2txt is a fast, browser-based tool designed to convert code repositories into plain text for use in LLM prompts.

    Key Capabilities:

    • Multiple Sources: Supports GitHub (public/private), Local Directories, Zip uploads, GitLab (Beta), and Azure DevOps (Beta).
    • Smart Filtering: Includes extension filters, .gitignore support, custom ignore patterns, and directory selection.
    • Privacy: 100% browser-based processing. No code is uploaded to a server; all processing happens locally on your device. GitHub tokens are stored only in sessionStorage.
    • Performance: Uses Web Workers for background tokenization, virtual scrolling for large repos (10,000+ files), and progressive loading.

  2. Understand the repo2txt v2.0 technology stack

    master

    The repo2txt v2.0 redesign is built on a modern web stack designed for performance, accessibility, and type safety.

    Core Frameworks & Tools:

    • React 18: Utilizes Concurrent features, Suspense for lazy loading, and automatic batching.
    • TypeScript 5.x: Enforces strict mode and full type coverage.
    • Vite: Used for fast Hot Module Replacement (HMR) and optimized builds.
    • TailwindCSS 3.x: Provides utility-first styling and built-in dark mode support.

    Key Libraries:

    • State Management: Zustand (lightweight, ~1KB).
    • UI Components: Radix UI (accessible, unstyled components).
    • Forms: React Hook Form (performant, uncontrolled components).
    • File Processing: JSZip for archives and gpt-tokenizer for token counting.
    • Testing: Vitest, React Testing Library, Playwright, and MSW (Mock Service Worker).

  3. Core Functionality of repo2txt

    master

    repo2txt is a browser-only tool designed to convert repository contents into text format, primarily for use with LLMs. Key features include:

    • GitHub Integration: Fetches repositories via API with smart branch and path parsing.
    • Local Uploads: Supports local directory uploads using the webkitdirectory API.
    • Archive Support: Handles .zip, .rar, and .7z files.
    • Smart Filtering: Parses .gitignore files to filter out unnecessary files.
    • File Visualization: Provides a directory tree view with checkboxes for file selection.
    • Token Management: Uses the cl100k_base tokenizer to count tokens.
    • Output Options: Supports clipboard copying and file downloading.

  4. How Gitignore patterns are parsed

    master

    The project includes a GitIgnoreParser class designed to provide comprehensive support for the .gitignore specification. It handles complex patterns including:

    • Negation patterns (e.g., !file)
    • Double-asterisk globbing (e.g., **/file)
    • Character classes (e.g., [abc])
    • Directory-only rules (e.g., dir/)
    • Comments and escaped characters. Matching is optimized via a PatternMatcher utility that converts patterns to regex and caches compiled patterns for performance.

  5. How the Provider Pattern works in repo2txt

    master

    To support multiple data sources (like GitHub, GitLab, or Azure DevOps) without coupling them to the UI, repo2txt uses a Provider Pattern.

    Each data source must implement a common IProvider interface. This abstraction allows the application to swap providers (e.g., switching from a GitHub repo to a local folder) without changing the UI logic. This pattern also makes it easy to mock providers for testing and to add new services like GitLab or ADO by simply implementing the interface.

    interface IProvider {
  // Metadata
  getType(): ProviderType;
  getName(): string;

  // Authentication
  requiresAuth(): boolean;
  setCredentials(credentials: ProviderCredentials): void;

  // Data fetching
  fetchTree(url: string, options?: FetchOptions): Promise<FileNode[]>;
  fetchFile(node: FileNode): Promise<string>;
  fetchMultiple(nodes: FileNode[]): AsyncGenerator<FileContent>;

  // Repository info
  getRepoInfo(): RepoMetadata;
}

  6. System Architecture of repo2txt

    master

    The architecture is organized into four distinct layers to ensure separation of concerns:

    1. UI Layer: Built with React, TailwindCSS, and Radix UI primitives.
    2. State Layer: Managed via Zustand, handling app state like the current provider, file tree, theme, and user selections.
    3. Business Logic Layer: Contains the core logic, including the FileTreeManager, Formatter (for tokenization), and the IProvider interface definitions.
    4. Data Providers Layer: Concrete implementations of the provider interface (e.g., GitHubProvider, LocalProvider, GitLabProvider, AzureProvider).

  7. Format repository contents into text

    master

    The Formatter class converts file contents into a structured text format suitable for LLM prompts. It generates an ASCII directory tree and calculates statistics.

    Key features:

    • Async Formatting: Uses formatAsync() which runs in a Web Worker (tokenizer.worker.ts) to prevent UI freezing during large file processing.
    • Tokenization: Uses gpt-tokenizer (specifically cl100k_base) to provide accurate token counts.
    • Statistics: Provides line counts and per-file token counts via the FileStats component.

  8. How the Provider System works

    master
    The project uses a provider pattern to abstract different source types (GitHub, GitLab, Azure DevOps, Local directories). A BaseProvider abstract class defines the interface, which is then implemented by specific providers like GitHubProvider, LocalProvider, GitLabProvider, and AzureProvider. This allows the core application to interact with any source using a unified set of methods for URL parsing, tree fetching, and file content retrieval, regardless of whether the source is a remote API or a local file system.

  9. How file tokenization and formatting are handled

    master
    To maintain a responsive UI, text formatting and token counting are handled by a TextFormatter and a Tokenizer class. The Tokenizer integrates gpt-tokenizer to provide per-file and total token counts (supporting cl100k_base and o200k_base). Crucially, tokenization is offloaded to a Web Worker to ensure that processing large repositories does not block the main UI thread, providing non-blocking progress reporting and cancellable tasks.

  10. Supported repository sources

    master

    repo2txt allows you to convert code repositories into text format using several different providers:

    • GitHub: Supports both public and private repositories.
    • GitLab: Supports GitLab.com and self-hosted instances (requires Private-Token).
    • Azure DevOps: Supports dev.azure.com and visualstudio.com (requires PAT via Basic Auth).
    • Local: Upload a local directory.
    • Zip: Upload a .zip file containing your code.

  11. Explore the repo2txt v2.0 project structure

    master

    The project follows a feature-based modular architecture. Key directories include:

    • src/features/: Contains domain-specific modules like github/, local/, gitlab/, and azure/. Each feature contains its own components, providers, and tests.
    • src/components/: Houses shared UI components such as FileTree/, ExtensionFilter/, OutputPanel/, and base ui/ elements.
    • src/lib/: Contains core business logic, including providers/ (via BaseProvider.ts and ProviderFactory.ts), file-tree/ logic, gitignore/ parsing, and formatter/ logic.
    • src/hooks/: Custom React hooks for shared logic like useTheme, useFileSelection, useProvider, and useTokenCount.
    • src/store/: Zustand state management organized into slices (e.g., providerSlice, fileTreeSlice).
    • src/workers/: Web Workers for heavy lifting like tokenizer.worker.ts and parser.worker.ts.
    • src/types/: Global TypeScript definitions.
    • tests/: Separated into unit/ (often colocated), integration/, and e2e/ (Playwright).
    repo2txt-v2/
├── src/
│   ├── main.tsx
│   ├── App.tsx
│   ├── features/
│   │   ├── github/
│   │   ├── local/
│   │   └── ...
│   ├── components/
│   │   ├── FileTree/
│   │   ├── ExtensionFilter/
│   │   └── ui/
│   ├── lib/
│   │   ├── providers/
│   │   ├── file-tree/
│   │   ├── gitignore/
│   │   └── formatter/
│   ├── hooks/
│   ├── store/
│   ├── workers/
│   └── types/
├── tests/
└── ...

  12. Use Azure DevOps as a source

    master

    repo2txt supports Azure DevOps repositories, including both modern (dev.azure.com) and legacy (visualstudio.com) formats. Authentication requires a Personal Access Token (PAT) using Basic Auth.

    Key components for Azure DevOps integration:

    • AzureDevOpsProvider: The core logic for fetching items via the Items API with recursion.
    • AzureAuth: Manages the required Personal Access Token (PAT).
    • AzureUrlInput: Provides URL input with hints for correct formatting.