OneFileLLM

The Conduit web app provides several interactive features for managing LLM-ready file generation:

• Live SSE progress: Uses a text/event-stream endpoint to stream per-stage events (detect, fetch, crawl, extract, tokenize). Web crawls include a progress bar showing "N / M pages".
• Structured results: Displays estimated model tokens, uncompressed/compressed token counts, compression delta, output size, source count, and elapsed time.
• Output tree: A virtualized list of files/pages in the output. The Raw tab provides a bounded preview (first 256 KB on a UTF-8 boundary).
• Client-side copy: Copy full uncompressed or compressed output to the clipboard via /result/raw.
• Download: Download uncompressed_output.txt or compressed_output.txt.
• Theme & Navigation: Light/dark theme toggle (persisted in localStorage); use Alt+T to switch themes and / to focus the input.
• No-JS fallback: If JavaScript is disabled, the form submits synchronously, and the server renders the results (bounded <pre>, token counts, and download links) using Jinja autoescaping.

Once processing is complete, the Result Panel provides three main views:

1. Metrics Panel

Displays key performance and size indicators:

• Estimated model tokens: (Hero card) round(uncompressed × 1.37).
• Uncompressed tokens: From uncompressed_token_count.
• Compressed tokens: From compressed_token_count (shows reduction percentage).
• Output size: final_output_kb.
• Sources processed: sources_processed.
• Elapsed: Total time taken.

2. Structured Overview (Tree View)

Provides a navigable tree of the processed files.

• Source rows: Show source type and URL.
• File rows: Show path, size, and line count.
• Navigation: Clicking a file row scrolls the Raw Preview to that specific file's offset.

3. Bounded Raw Preview

A code-like view of the output.

• Safety Cap: To protect DOM performance, only the first 256 KB (PREVIEW_CAP_BYTES) is rendered in the browser.
• Truncation Notice: If the file is larger than 256 KB, a notice bar appears: Showing first 256 KB of [Total Size] — download or copy for the complete output.

Aliases allow you to group multiple URLs or sources under a single name for quick access.

Creating Aliases

Simple Alias:

python onefilellm.py --alias-add mcp "https://github.com/anthropics/mcp"

Complex/Ecosystem Alias (Multiple Sources):

python onefilellm.py --alias-add modern-web \
  "https://github.com/facebook/react https://reactjs.org/docs/ https://github.com/vercel/next.js"

Dynamic Placeholders

You can create aliases with {} placeholders to inject values at runtime.

1. Create the alias:

python onefilellm.py --alias-add gh-user "https://github.com/{}"

2. Use the alias:

python onefilellm.py gh-user "microsoft"

python onefilellm.py --alias-add mcp "https://github.com/anthropics/mcp"
python onefilellm.py --alias-add modern-web \
  "https://github.com/facebook/react https://reactjs.org/docs/ https://github.com/vercel/next.js"

python onefilellm.py --alias-add gh-search "https://github.com/search?q={}"
python onefilellm.py gh-search "machine learning transformers"

The Web App follows a specific state machine for processing requests. The UI state is driven by the data-view attribute on the .app__transcript element, which can be one of four values: empty, processing, result, or error.

The Full Flow:

1. empty: The initial state where the user provides input.
2. submit: User triggers a request. The client validates the input. If valid, the state transitions to processing.
3. processing: The app opens an EventSource connection to /stream?input=…. During this phase, the app displays progress via SSE (Server-Sent Events) such as stage events, page events (for crawling), and an elapsed timer.
4. result: Triggered by a done SSE event. The timer freezes, and the transcript crossfades to show the final result, including metrics and an overview tree.
5. error: Triggered by an error SSE event or an <error> tag, transitioning the view to an error state.

Transitions:

• Cancel: If a process is cancelled, the state returns to empty and the input is restored.
• Retry: Users can edit or retry from the result or error states to return to the empty state.

The OneFileLLM Web App uses specific data-view and data-error attributes on the container to manage UI states. Developers or contributors implementing the frontend should use these attributes to trigger the correct screen layouts:

• [data-view="empty"]: The initial state with the input prompt and suggestion cards.
• [data-view="processing"]: The state shown while a task is running (supports indeterminate or determinate progress).
• [data-view="result"]: The state shown after a successful extraction, displaying metrics and file previews.
• [data-view="error"]: The state shown when a process fails, used in conjunction with specific error types.

Error Types (data-error):

• [data-error="security"]: Triggered when an input is blocked (e.g., local paths, private/internal URLs) to prevent SSRF.
• [data-error="processing"]: Triggered when the backend fails to process a valid source (e.g., no video transcript found).
• [data-error="empty"]: Triggered when a source is reached but contains no extractable content.

Conduit implements specific UX patterns to manage large data processing and long execution times:

Long-running Jobs (G1)

To prevent the UI from appearing frozen, the app uses:

• An SSE-driven (Server-Sent Events) stage log with append-only lines.
• An always-moving elapsed timer.
• An indeterminate accent shimmer for indeterminate tasks, or a determinate N/max progress bar for crawls.
• A Cancel button to abort the process.

Managing Large Data (G2 & G4)

To avoid overwhelming the user with massive text blobs, the result is presented in layers:

1. Metrics Panel: High-level statistics.
2. Structured Overview Tree: A lightweight tree parsed from tags.
3. Bounded Raw Preview: A preview capped at 256 KB with a label indicating "showing first X of Y."

Full payloads are accessed via Copy or Download actions. For large payloads, the Copy action uses a "preparing..." state that fetches /result/raw before writing to the clipboard via navigator.clipboard.writeText.

On success, the backend returns a single string of XML-like tagged text. This is not strict XML and should be treated as text containing recognizable tags.

Output Structure:

• The primary block is <source type="..." url="...">...</source>.
• Inside <source>, individual files are represented as <file path="...">...contents...</file>.
• The CLI may wrap multiple sources in a <onefilellm_output> root; the UI should handle both bare <source> blocks and wrapped roots.
• Error Handling: If the backend encounters an error, it returns <source type="..." ...><error>message</error></source>. The UI must detect the <error> tag and render it as a failure state rather than displaying it as content.

Handling Large Payloads: Outputs can be massive (hundreds of thousands of tokens/multiple MBs). Do not dump the entire payload into the DOM. Instead, implement:

1. A structured overview parsed from the tags (a list/tree of files and paths).
2. A bounded raw preview (e.g., the first 100–200 KB) with a notice indicating the total size.

<source type="github_repository" url="https://github.com/owner/repo">
<file path="README.md">
...file contents, unescaped...
</file>
<file path="src/main.py">
...
</file>
</source>