DOCSEARCHbyAlgolia
Submit Docs

Open Generative AI

repository·main·Indexed 12 days ago

https://github.com/anil-matcha/open-generative-ai

An open-source alternative to commercial AI video platforms providing access to over 400 models for image, video, audio, and lip sync generation. Version 2.0.0 includes specialized studios for Cinema, Workflows, and Lip Sync, as well as support for local model inference via sd.cpp and Wan2GP. It features a node-based visual editor for AI pipelines and professional camera controls for cinematic shot generation.

Tokens
19.6K
Snippets
52
Records
82
Agent score
99%

What's inside Open Generative AI

  1. Follow the Control Contract for prompt controls

    main

    Every control for parameters such as model, aspect-ratio, duration, resolution, quality, or preset inside a floating prompt panel must adhere to the following contract:

    • Sizing: Must remain 38 px high.
    • Styling: Use promptControlClassName() to preserve shared background, border, radius, hover state, and spacing.
    • Active State: Use promptControlClassName({ active: true }) for an open or selected control.
    • Labels: Use a 12 px semibold label (unless accessibility requirements dictate otherwise).
    • Icons: Use the shared 16 px parameter icon for aspect ratio, duration, or quality.
    • Configuration: Use the compact or iconOnly options instead of overriding horizontal padding or width with conflicting utility classes.

    Approved Icons

    • PromptAspectRatioIcon: For aspect ratio parameters.
    • PromptDurationIcon: For duration parameters.
    • PromptQualityIcon: The single approved resolution/quality symbol (an unfilled, faceted gemstone outline).
    • PromptChevronIcon: The shared affordance for controls that open a list.

  2. Use Video Studio in Text-to-Video or Image-to-Video mode

    main

    Video Studio operates in two modes based on input:

    • Text-to-Video (t2v): Default mode (no image). Requires a text prompt. Supports 40+ models like Kling, Sora, and Runway.
    • Image-to-Video (i2v): Triggered when a start frame image is uploaded. The prompt is optional. Supports 60+ models like Kling I2V, Runway I2V, and Midjourney I2V.

  3. How the ImageStudio component manages generation state

    main

    The ImageStudio.js component acts as the central controller for the image generation interface. It manages several key pieces of state:

    • selectedModel: The currently active model chosen from the model picker.
    • selectedAr: The selected aspect ratio.
    • Generation Status: Tracks whether a request is currently in progress.

    Generation Lifecycle:

    1. Auth Check: It verifies if an API key exists in localStorage. If not, it triggers the AuthModal.
    2. Submission: It calls muapi.generateImage() to initiate the request.
    3. Polling: It enters a polling loop to wait for the generation result.
    4. Completion: Upon success, it updates the generationHistory and displays the result.

    History Management: Generation history is persisted in localStorage using the key muapi_history. Users can click thumbnails to re-view images or hover to download them.

  4. Build AI pipelines with Workflow Studio

    main

    Workflow Studio is a node-based visual editor for building multi-step AI pipelines.

    Key Features:

    • Templates: Use pre-built image or video chains.
    • Node-based Builder: Drag-and-drop to connect models and route outputs.
    • Playground: Run workflows via an interactive form UI.
    • API execution: Workflows can be called via the Muapi API.

    Developers can integrate the underlying engine into their own applications using Vibe Workflow.

  5. Use Image Studio in Text-to-Image or Image-to-Image mode

    main

    Image Studio automatically switches modes based on whether you provide a reference image:

    • Text-to-Image (t2i): The default mode when no image is uploaded. Requires a text prompt. Supports 50+ models including Flux, Nano Banana 2, and Midjourney.
    • Image-to-Image (i2i): Triggered automatically when a reference image is uploaded. The prompt becomes optional. Supports 55+ models including Kontext, Nano Banana 2 Edit, and various Upscalers.

  6. How models are defined in models.js

    main

    The models.js file serves as the single source of truth for all available generative models. It exports a t2iModels array where each object defines the capabilities and routing for a specific model.

    Each model object contains:

    • id: The internal identifier.
    • name: The display name.
    • inputs: A schema defining supported features like resolution and aspect ratio.
    • endpoint: The specific API path used for requests. This maps the internal ID to the actual API route (e.g., the ID flux-schnell maps to the endpoint flux-schnell-image).

  7. Understand the Open Generative AI architecture

    main

    The project is a Next.js monorepo that utilizes npm workspaces. The core logic and UI components reside in packages/studio, which is consumed by both the local Next.js application and the hosted version at muapi.ai.

    Key Directory Structure

    • app/: Next.js App Router implementation.
    • components/: UI components like StandaloneShell.js (handles navigation and API key management via localStorage) and ApiKeyModal.js.
    • packages/studio/: The shared React component library.
      • src/index.js: Exports all major Studio interfaces (e.g., ImageStudio, VideoStudio, WorkflowStudio, AgentStudio).
      • src/models.js: The single source of truth for 400+ model definitions.
      • src/muapi.js: The API client for interacting with Muapi.ai.
      • src/components/: Specific studio implementations (e.g., CinemaStudio.jsx, LipSyncStudio.jsx).

    Model updates made in packages/studio/src/models.js automatically apply to both the self-hosted and hosted versions.

  8. How the muapi.js engine handles API communication

    main

    The muapi.js client encapsulates all communication with api.muapi.ai. It follows a specific authentication and request pattern:

    Authentication Requests must include the API key in the x-api-key header. Note that it does not use the standard Authorization: Bearer format.

    The Submit-and-Poll Pattern Because image generation is asynchronous, the client uses a two-step process:

    1. Submit: A POST request is sent to the model's endpoint (e.g., /api/v1/nano-banana-pro). The API returns a request_id.
    2. Poll: The client enters a POST or GET loop on the /api/v1/predictions/{id}/result endpoint. This continues until the status reaches completed, succeeded, or failed.

    Data Normalization Since different models may return slightly different response structures, muapi.js normalizes the output to ensure a consistent url property is always available (often by extracting it from outputs[0]).

  9. Manage uploaded images with Upload History

    main

    To avoid re-uploading files, the application stores reference images locally (URL + thumbnail) in localStorage.

    • Accessing History: Click the upload button to open the reference image picker.
    • Single-image selection: Click a thumbnail to instantly select it.
    • Multi-image selection: Toggle multiple thumbnails (using order numbers) and click Use Selected.
    • Management: Use the Upload files button for batch uploads or the ✕ button to remove individual images from history.

  10. Install the Open Generative AI Desktop App

    main

    The desktop application is available as one-click installers for macOS, Windows, and Linux. No Node.js or terminal knowledge is required for standard installation.

    macOS

    • Apple Silicon (M1/M2/M3/M4): Use the arm64.dmg installer.
    • Intel (x64): Use the standard .dmg installer.

    Windows

    • Use the .exe setup installer.

    Linux

    • Ubuntu x64: Download the .AppImage or .deb from the releases page.
    • Build from source: Use npm run electron:build:linux to generate local installers.
    npm run electron:build:linux

  11. Use Multi-Image Input for compatible models

    main

    Certain models support multiple reference images. When such a model is selected, the upload trigger switches to a multi-select mode with the following features:

    • Order-sensitive selection: Images are sent to the model in the exact order you select them (indicated by order badges).
    • Batch upload: You can select multiple files at once from the file dialog.
    • Selection confirmation: Use the Use Selected button to confirm your choices.

    Model Capacity Reference:

    ModelMax Images
    Nano Banana 2 Edit14
    Nano Banana Edit10
    Flux Kontext Dev I2I10
    Kling O1 Edit Image10
    GPT-4o Edit / GPT Image 1.5 Edit10
    Bytedance Seedream Edit v4 / v4.510
    Vidu Q2 Reference to Image7
    Flux 2 Flex/Pro Edit8
    Nano Banana Pro Edit8
    Flux Kontext Pro/Max I2I2
    Wan 2.5/2.6 Image Edit2–3
    Qwen Image Edit Plus / 25113
    GPT-4o Image to Image5

  12. Use Prompt Composer UI primitives in Studio

    main

    When building floating generation prompt panels in Studio, do not recreate the panel shell, textarea resizing, parameter-control sizing, footer layout, or primary action buttons. Instead, use the shared prompt composer primitives to ensure visual and functional consistency across the application.

    Required Primitives

    Layout & Shell

    • PromptComposer: Provides the floating panel position, background, border, radius, padding, shadow, and animation.
    • PromptTextarea: Provides shared typography, placeholder treatment, scrolling, and responsive auto-resize behavior.
    • PromptFooter: Provides the responsive divider and action-row layout.
    • PromptAction: Provides the shared primary generation button.
    • PromptPopover: Provides the shared dropdown surface, placement, border, radius, shadow, scrolling, and responsive height limit.
    • PromptPopoverHeader: Provides the shared uppercase section heading.
    • PromptMenuList and PromptMenuItem: Provide consistent option spacing, typography, hover feedback, and selected-state checkmarks.
    • PromptSegmentedControl and PromptSegmentOption: Provide the shared two-state or multi-state mode switch.

    Controls & Media

    • PromptControls: Keeps parameter controls aligned and spaced consistently.
    • promptControlClassName(): Provides the 38 px parameter-control contract.
    • promptMediaButtonClassName(): Provides the 40 px circular media-attachment contract.
    • PROMPT_MEDIA_PREVIEW_CLASS: Keeps uploaded media previews on the same 40 px circular contract.

    Implementation Pattern

    Shared primitives own presentation and textarea resizing only. All business logic (API calls, validation, uploads, persistence, and generation handlers) must remain inside the owning Studio component. Pass behavior into the primitives via standard React props like value, onChange, onClick, and disabled.