DOCSEARCHbyAlgolia
Submit Docs

screenshot-to-code

repository·main·Indexed 11 days ago

https://github.com/abi/screenshot-to-code

An AI-powered tool that converts screenshots, mockups, Figma designs, and screen recordings into functional code. It supports HTML/Tailwind, React, Vue, and Bootstrap using models from Gemini, GPT, and Claude. Features include a FastAPI backend, React/Vite frontend, and a screenshot preview tool powered by headless Chromium.

Tokens
12.6K
Snippets
38
Records
62
Agent score
97%

What's inside screenshot-to-code

  1. New asset handling capabilities

    main

    The image-tools branch introduces improved asset management:

    • Smart Extraction: Extracts specific assets (logos, hero images, feature icons) from screenshots rather than just buttons, text, or whole pages.
    • Pixel Accuracy: Extracted crops are pixel-accurate and integrated into the generated page.
    • Verbatim Usage: Uploaded logos are used exactly as provided instead of being redrawn.
    • Screenshot Preview: The agent renders its own HTML and inspects it.
    • Batch Image Editing: edit_images and remove_backgrounds support local asset URLs for batch processing.

  2. What is Screenshot Preview and how is it enabled?

    main

    Screenshot preview is an optional feature that allows the AI agent to render its own generated page in a headless browser to visually verify its work.

    It is enabled automatically if the Chromium browser is installed via poetry run playwright install chromium (or is present in the Docker image). You can check the availability of this feature in the application's Settings dialog.

  3. Implement Provider-Specific Tool Continuation

    main

    When implementing a new provider, you must satisfy the ProviderSession and ProviderTurn contract defined in backend/agent/providers/base.py. Each provider must implement append_tool_results to handle how tool outputs are fed back into the model's history.

    Implementation Patterns by Provider:

    • OpenAI: Append the original assistant turn (turn.assistant_turn) and then append one function_call_output per tool result: {"type":"function_call_output","call_id":...,"output": json_string}.
    • Anthropic: Append assistant message blocks (optional text and tool_use blocks with id, name, input) followed by a user message containing tool_result blocks (including tool_use_id, serialized result, and is_error).
    • Gemini: Append the exact original model content (turn.assistant_turn) and then append a role="tool" content using Part.from_function_response(...) for each tool.

  4. Proposed non-blocking variant generation flow

    main
    The proposed system moves from a blocking flow to a progressive, non-blocking flow. Instead of waiting for the entire batch, the system signals individual variant completion. This allows the frontend to show the first completed variant immediately, enabling user interaction (like selecting a variant or starting an update) while other variants are still generating or can be cancelled.

  5. Understand the Commit System

    main

    The commit system manages the application's history by representing discrete versions of generated code. Each commit is a unique snapshot that allows for history tracking and version switching.

    Commit Structure

    A commit contains:

    • hash: A unique identifier generated via nanoid().
    • parentHash: A link to the previous commit to reconstruct history.
    • variants: An array of code generation options (typically 2).
    • selectedVariantIndex: The index of the variant currently being viewed.
    • isCommitted: A boolean indicating if the commit is finalized (true) or still being edited (false).
    • type: The origin of the commit (see Commit Types).
    • inputs: Type-specific input data.

    Commit Types

    • ai_create: Initial generation from a screenshot or video.
    • ai_edit: Updates made based on user instructions.
    • code_create: Import from existing code.
    type CommitType = "ai_create" | "ai_edit" | "code_create";

type Commit = {
  hash: CommitHash;
  parentHash: CommitHash | null;
  dateCreated: Date;
  isCommitted: boolean;
  variants: Variant[];
  selectedVariantIndex: number;
  type: CommitType;
  inputs: any;
}

type Variant = {
  code: string;
  status: VariantStatus;
}

type VariantStatus = "generating" | "complete" | "cancelled";

  6. Verify asset usage via content-addressing

    main

    Assets in the system are content-addressed using a SHA256 hash. To verify that a specific uploaded file (like a logo) was used verbatim in the output rather than being redrawn by the AI, hash the original file and look for the corresponding filename in the served assets and the generated HTML.

    Asset filename format: asset_<sha256[:24]>.png

  7. Compare Claude 3 and GPT-4 Vision performance

    main

    Based on evaluations using the HTML + Tailwind stack, here is how the models compare in replication accuracy:

    • Claude 3 Sonnet: ~70.31% (Highest accuracy and faster speed).
    • GPT-4 Vision: ~65.10% (The baseline).
    • Claude 3 Opus: ~61.46% (Lower accuracy in this specific test, potentially due to prompting).

    Key Observations:

    • Laziness: Claude 3 is generally less "lazy" than GPT-4 Vision. GPT-4 Vision often uses comments like <!-- Repeat for each news item --> instead of generating full content, whereas Claude 3 tends to complete the requested task more fully.
    • Colors: Claude 3 may struggle with background and text color accuracy.
    • Layouts: All models currently struggle with side-by-side "flex" layouts.
    • Prompting: Current prompts are optimized for GPT-4 Vision; adjusting them for Claude can yield small improvements.

  8. How model selection and cycling works

    main

    The system selects models based on the API keys currently available in the environment. If the number of requested variants (NUM_VARIANTS) exceeds the number of available models, the system cycles through the available models in a loop.

    Example Cycling Logic: If you have 2 models available ([A, B]) and set NUM_VARIANTS = 5, the resulting models used for generation will be [A, B, A, B, A].

    Generation Types:

    • Create: Uses Claude 3.7 Sonnet as the primary model.
    • Update: Uses Claude Sonnet 4.5 as the primary model.
    # Both API keys present
models = [claude_model, Llm.GPT_4_1_NANO_2025_04_14]

# Claude only  
models = [claude_model, Llm.CLAUDE_4_5_SONNET_2025_09_29]

# OpenAI only
models = [Llm.GPT_4O_2024_11_20]

  9. Understand the Agent Tool-Calling Flow

    main

    The backend agent operates through a core loop managed by AgentEngine._run_with_session(...). The process follows these steps:

    1. Turn Initialization: Creates event IDs for assistant and thinking streams and initializes tracking for tool IDs and streamed lengths.
    2. Provider Turn Streaming: Calls session.stream_turn(on_event) to stream deltas. The on_event handler routes assistant_delta to the assistant websocket, thinking_delta to the thinking websocket, and tool_call_delta to the tool delta handler.
    3. Tool Branching: If turn.tool_calls is present, the engine executes each tool, emits lifecycle messages, and collects results.
    4. Conversation Continuation: Tool results are appended via session.append_tool_results(turn, executed_tool_calls), and the loop repeats with the updated history.
    5. Guardrail: The loop is limited to a maximum of 20 tool turns to prevent infinite loops; exceeding this raises an error.

  10. How Non-Blocking Variant Generation works

    main

    Unlike traditional generation where the user must wait for all variants to finish, the non-blocking system allows users to interact with the UI as soon as the first variant completes. This is achieved through a hybrid state approach combining global AppState and individual VariantStatus.

    Key Behaviors

    • Immediate Interaction: If the currently selected variant completes, the UpdateInterface becomes available even if other variants are still generating.
    • Parallel Processing: Multiple models generate code simultaneously.
    • Automatic Cancellation: When a user starts a new update (e.g., an ai_edit), the system automatically sends a cancellation signal to all other variants that are still in the generating status to save resources.
    • Error Isolation: If one variant fails (status: cancelled), it does not block the user from using other successful variants.

  12. How the current variant generation system works

    main
    The current system uses a blocking, all-or-nothing approach for generating code variants. When a user requests code generation, the backend creates parallel tasks for multiple variants (e.g., NUM_VARIANTS = 2) using different AI models. The WebSocket connection remains open and the UI is blocked until all variants have completed generation. Users cannot select a variant, make updates, or start new generations until the entire batch is finished.