DOCSEARCHbyAlgolia
Submit Docs

OpenScreen

repository·main·Indexed 12 days ago

https://github.com/siddharthvaddem/openscreen

An open-source alternative to Screen Studio for creating polished product demos and walkthroughs. It features auto-zooming, webcam overlays, custom cursors, and automatic on-device captions. The tool utilizes a Native Bridge architecture with platform-specific capture pipelines using ScreenCaptureKit on macOS and Windows Graphics Capture (WGC) on Windows to provide high-quality window-level recording and cursor telemetry.

Tokens
20.4K
Snippets
66
Records
91
Agent score
99%

What's inside OpenScreen

  1. Understand the macOS Native Recorder Architecture

    main

    The macOS native recorder uses a split architecture between Electron and a platform-native Swift helper to ensure high-performance capture and timing alignment.

    • Electron (Orchestrator): Responsible for session orchestration, persistence, resolving output paths, resolving display/window/device sources, managing the helper process lifecycle (start, pause, resume, stop, cancel), and writing RecordingSession manifests.
    • Swift Helper (Media Engine): Responsible for the actual media capture using ScreenCaptureKit and AVFoundation. It handles display/window frames, system audio, microphone, webcam composition, encoding (H.264/AAC), muxing into MP4, and timestamp normalization.

    This separation ensures that screen video, audio, webcam, and cursor are all aligned to a single native timing origin, avoiding the jitter associated with browser-side stream assembly.

  2. Understand the Windows Native Recorder Architecture

    main

    On Windows, OpenScreen uses a native backend to handle high-quality recording instead of relying on Electron's getDisplayMedia or MediaRecorder.

    The Architecture consists of two main parts:

    1. Electron (The Orchestrator): Manages the recording session. It resolves the selected source (display or window), determines output paths, starts cursor sampling, manages the helper process, and sends control commands (pause, resume, stop, cancel). It also writes RecordingSession manifests.

    2. The Native Helper (The Media Engine): Handles the actual media capture and muxing. It captures Windows Graphics Capture (WGC) frames, WASAPI system loopback audio, WASAPI microphone input, and Media Foundation webcam video. It is responsible for stream timestamp normalization and encoding/muxing everything into a single H.264/AAC MP4 file.

    Key Constraint: On Windows, if a native capability is unavailable, the system should return an explicit error rather than silently falling back to Electron capture.

  3. How the Native Bridge architecture works

    main

    OpenScreen uses a layered Native Bridge architecture to provide platform-native capabilities (like cursor telemetry or system asset discovery) to the Electron renderer.

    Instead of binding directly to ad hoc Electron APIs, the system follows these layers:

    1. Native adapters: Platform-specific providers implementing stable domain interfaces.
    2. Main-process services: Orchestrate adapters and own the runtime state.
    3. Unified IPC transport: A single native-bridge:invoke channel using versioned contracts.
    4. Renderer client: The interface used by React code to interact with the bridge.

    Key principles include a Single source of truth (native state lives in the main process), Capability-first design (the renderer can query support before acting), and Resilience (consistent result envelopes with stable error codes).

  4. Understand platform differences in OpenScreen

    main

    While the editor and export features are identical across all platforms, the capture capabilities vary based on the operating system's native pipeline.

    Capture Pipeline

    • macOS & Windows: Use native pipelines (ScreenCaptureKit and Windows Graphics Capture respectively) for high-quality, clean window-level capture. This allows for capturing the real cursor (shape, type, clicks) to power cursor themes and click effects.
    • Linux: Uses a browser pipeline. Only the cursor position is captured (used for auto-zoom), so advanced cursor themes and click effects are unavailable.

    Webcam & Audio

    • Webcam: Captured natively on macOS and Windows. On Linux, it is recorded through the browser (supports picture-in-picture).
    • System Audio:
      • macOS: Requires macOS 13+. On macOS 14.2+, you must grant explicit audio capture permission. macOS 12 and below do not support system audio capture.
      • Windows: Works out of the box.
      • Linux: Requires PipeWire (default on Ubuntu 22.04+, Fedora 34+). Older PulseAudio-only setups may not capture system audio.

  5. Build and Test the macOS Native Helper

    main

    Use the following npm commands to build the Swift helper binaries or run specific smoke tests for different capture capabilities on macOS.

    # Build Swift helper binaries
npm run build:native:mac

# Run smoke tests
npm run test:sck-helper:mac          # Display-only test
npm run test:sck-window:mac          # Window capture test
npm run test:sck-audio:mac           # System audio test
npm run test:sck-mic:mac             # Microphone test
npm run test:sck-webcam:mac          # Webcam test

  6. Install OpenScreen on Windows

    main

    Install OpenScreen on Windows using winget.

    Using winget:

    winget install SiddharthVaddem.OpenScreen

    Manual Installation: You can also download the .exe installer directly from the GitHub Releases page.

    Management Commands:

    • Update: winget upgrade SiddharthVaddem.OpenScreen
    • Uninstall: winget uninstall SiddharthVaddem.OpenScreen

  7. Install OpenScreen on macOS

    main

    You can install OpenScreen on macOS using Homebrew, which handles architecture detection (Apple Silicon vs Intel) and notarization.

    Using Homebrew:

    brew install --cask siddharthvaddem/openscreen/openscreen

    Manual Installation & Gatekeeper Bypass: If you download the .dmg directly from the Releases page and Gatekeeper blocks the app, run the following command in your terminal to remove the quarantine attribute:

    xattr -rd com.apple.quarantine /Applications/Openscreen.app

    Note: You must grant your terminal Full Disk Access in System Settings > Privacy & Security to execute this command.

    Permissions: After installation, you must manually grant permissions for "screen recording" and "accessibility" in System Preferences > Security & Privacy.

    Troubleshooting Permission Issues: If an upgrade causes recording or accessibility to fail, perform a clean reinstall:

    1. Uninstall the old version.
    2. Remove existing OpenScreen entries under System Settings > Privacy & Security (both Screen Recording and Accessibility).
    3. Reinstall and grant permissions when prompted.

  8. Build and use Windows WGC capture helpers

    main

    Windows native recording uses a Windows Graphics Capture (WGC) helper. The helper operates on a process-based contract: the app starts the process with a single JSON argument and sends commands via stdin. Sending stop\n finalizes the recording.

    Helper Resolution Order

    1. OPENSCREEN_WGC_CAPTURE_EXE: Used for local development and diagnostics.
    2. electron/native/wgc-capture/build/wgc-capture.exe: Locally built Ninja helper.
    3. electron/native/wgc-capture/build/Release/wgc-capture.exe: Locally built multi-config helper.
    4. electron/native/bin/win32-x64/wgc-capture.exe or electron/native/bin/win32-arm64/wgc-capture.exe: Packaged prebuilt helpers.

    Capabilities

    • Display/window video capture.
    • System audio loopback.
    • Selected-microphone capture.
    • Media Foundation webcam capture (with DirectShow fallback for virtual cameras).
    • Webcam frames are composed as a bottom-right picture-in-picture overlay in the MP4.

    Device Resolution

    • Microphones: Tries the requested WASAPI endpoint ID first, then resolves by microphoneDeviceName, then falls back to the default endpoint.
    • Webcams: Uses Media Foundation first; if the requested camera is absent, it uses the DirectShow filter CLSID resolved by Electron.
    npm run build:native:win

  10. Run Windows Native Recorder Tests

    main

    Use the following npm commands to validate different aspects of the Windows native recording implementation. Note that some tests require specific hardware (like a webcam) to be present.

    # Display-only helper smoke test
npm run test:wgc-helper:win

# Validate AAC audio track presence and duration
npm run test:wgc-audio:win

# Capture a fixture window by HWND
npm run test:wgc-window:win

# Validate webcam output (skips if no webcam is found)
npm run test:wgc-webcam:win

# Validate default-microphone capture
npm run test:wgc-mic:win

# Validate mixed audio (system loopback + microphone)
npm run test:wgc-mixed-audio:win

  11. Install OpenScreen on Linux

    main

    OpenScreen is available in several formats for Linux. Choose the one matching your distribution.

    Debian / Ubuntu / Pop!_OS (.deb):

    sudo apt install ./Openscreen-Linux-latest.deb

    Arch / Manjaro (.pacman):

    sudo pacman -U Openscreen-Linux-latest.pacman

    AppImage (Universal):

    chmod +x Openscreen-Linux-*.AppImage
./Openscreen-Linux-*.AppImage

    Note: If the AppImage fails with a "sandbox" error, run it with the --no-sandbox flag.

    Nix / NixOS:

    • Run without installing: nix run github:siddharthvaddem/openscreen
    • Install to user profile: nix profile install github:siddharthvaddem/openscreen
    • For NixOS flake configuration, use the openscreen.nixosModules.default module and enable it via programs.openscreen.enable = true;.

  12. Build and use macOS native capture helpers

    main

    macOS native recording uses a ScreenCaptureKit helper. The helper manages ScreenCaptureKit/AVFoundation capture, timing, encoding, and muxing.

    Helper Resolution Order

    1. OPENSCREEN_SCK_CAPTURE_EXE: Used for local development and diagnostics.
    2. electron/native/screencapturekit/build/openscreen-screencapturekit-helper: Locally built Swift output.
    3. electron/native/bin/darwin-arm64/openscreen-screencapturekit-helper or electron/native/bin/darwin-x64/openscreen-screencapturekit-helper: Packaged prebuilt helpers.

    The macOS cursor-shape helper is resolved from OPENSCREEN_MAC_CURSOR_HELPER_EXE first, then the matching openscreen-macos-cursor-helper binary in the local build or packaged directories.

    Capabilities

    • Display/window ScreenCaptureKit video capture.
    • Cursor exclusion via SCStreamConfiguration.showsCursor.
    • H.264 encoding and MP4 muxing.
    • ScreenCaptureKit system audio.
    • Native ScreenCaptureKit microphone capture (on supported macOS versions).
    • Note: Webcam recording is currently an Electron sidecar attached after native screen capture stops.

    Probing Availability

    Use the is-native-mac-capture-available capability in Electron to probe for the helper. It returns missing-helper if no Swift binary is found.

    npm run build:native:mac