streamlit-webrtc

repository·main·Indexed 23 days ago

https://github.com/whitphx/streamlit-webrtc

A library for real-time video and audio streaming and processing within Streamlit applications using WebRTC. It provides the `webrtc_streamer` component and supports function-based callbacks for manipulating media frames via PyAV. The library includes tools for managing media track lifecycles, configuring STUN/TURN servers for remote deployment, and ensuring HTTPS connectivity for media device access.

Tokens
17.2K
Snippets
21
Records
98
Agent score
82%

What's inside streamlit-webrtc

  1. Manage the lifecycle of source and sink tracks

    main

    Factory helpers like create_video_source_track(), create_audio_source_track(), create_video_sink_track(), create_audio_sink_track(), and create_pcm_audio_source_track() cache their objects in st.session_state using the provided key. This ensures tracks survive Streamlit reruns.

    By default, these tracks are scoped to the active WebRTC session. When the session ends (e.g., user clicks STOP or closes the page), the object is stopped and removed from st.session_state.

    To keep a track alive across multiple WebRTC sessions within the same Streamlit session, set lifecycle_scope="streamlit-session".

    from streamlit_webrtc import create_video_source_track
    
    # Default: scoped to the WebRTC session
    video_track = create_video_source_track(
        callback=video_source_callback,
        key="video-source",
    )
    
    # Opt-out: survives multiple WebRTC sessions in the same Streamlit session
    video_track = create_video_source_track(
        callback=video_source_callback,
        key="video-source",
        lifecycle_scope="streamlit-session",
    )
  2. Understand the AI Issue Triage and Implementation Workflow

    main

    The workflow operates in two distinct phases based on GitHub events:

    1. Triage Phase (claude-issue-triage.yml)

    Triggered by issues.opened. Claude analyzes the issue and performs one of the following:

    • needs-info: Automatically applies this label if information (reproduction steps, environment, etc.) is missing. Claude posts a comment requesting the missing details.
    • triaged: Automatically applies this label if the report is actionable. Claude posts a 3-5 line summary.
    • out-of-scope: Automatically applies this label if the issue belongs to another project (e.g., Streamlit core, aiortc) or is a support question. Claude posts a redirection comment.

    Note: Bot-authored issues are automatically skipped.

    2. Implementation Phase (claude-issue-implement.yml)

    Triggered by issues.labeled. This phase is driven by manual maintainer actions:

    • ai-implement: When a maintainer applies the ai-implement label, Claude creates a branch (claude/issue-<number>), runs tests/linters, adds a changelog fragment, and opens a PR.
    • wontfix: When a maintainer applies the wontfix label, Claude posts an explanation and closes the issue.
  3. Pull Values from Callbacks to the Main Script

    main

    To read data generated inside a callback (e.g., analysis results) in your main Streamlit script, you must account for the fact that the callback runs in a forked thread.

    Requirements for implementation:

    1. Thread-safety: Use threading.Lock or similar primitives when accessing shared objects (like a dictionary or queue) between the callback and the main script.
    2. Polling Loop: Because the main script execution normally reaches the end and stops, you must use a loop in the main script to continuously poll the shared container for updates while the media is streaming.

    Warning: Do not attempt to use st.* methods (like st.write()) inside the callback, as they will not work in the forked thread.

  4. Pull values from the callback to the main thread

    main

    To read data generated inside a callback (e.g., processed images or analysis results) in the main Streamlit thread, you must handle thread-safety and polling.

    Key requirements:

    1. Thread-safety: Use threading.Lock to protect shared mutable objects (like a dictionary or queue) accessed by both the callback thread and the main thread.
    2. Polling: Because the main script execution stops at the bottom while streaming, use a while ctx.state.playing: loop to continuously poll for updates from the callback.
    import threading
    import cv2
    import streamlit as st
    from matplotlib import pyplot as plt
    
    from streamlit_webrtc import webrtc_streamer
    
    
    lock = threading.Lock()
    img_container = {"img": None}
    
    
    def video_frame_callback(frame):
        img = frame.to_ndarray(format="bgr24")
        with lock:
            img_container["img"] = img
    
        return frame
    
    
    ctx = webrtc_streamer(key="example", video_frame_callback=video_frame_callback)
    
    fig_place = st.empty()
    fig, ax = plt.subplots(1, 1)
    
    
    while ctx.state.playing:
        with lock:
            img = img_container["img"]
        if img is None:
            continue
        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
        ax.cla()
        ax.hist(gray.ravel(), 256, [0, 256])
        fig_place.pyplot(fig)
  5. Manage AI API Spend and Security

    main

    Security Mitigations

    Because issue bodies are untrusted, always review generated PRs to prevent prompt-injection attacks. The workflow uses restricted toolsets:

    • Triage/Wontfix: Limited to gh issue:*, gh label:*, and gh search:*.
    • Implementation: Limited to Bash, Edit, Write, Read, Grep, Glob.
    • Permissions: The GITHUB_TOKEN is scoped to the minimum required permissions.

    Controlling Costs

    To prevent high API spend during noisy periods:

    • The workflow already skips bot-authored issues.
    • Consider adding filters to skip issues that do not follow the official issue template.
    • Consider reacting only to issues with specific labels.
    • Route non-collaborator issues to human review instead of full automation.
  6. Understand the internal orchestration of webrtc_streamer()

    main

    The webrtc_streamer() function manages the lifecycle of a WebRTC connection by coordinating five distinct concerns. Understanding this flow is helpful for debugging connection issues or lifecycle events:

    1. Context Management: Retrieves or creates a WebRtcStreamerContext in st.session_state using the provided key.
    2. Frontend Rendering: Renders the Streamlit component and wires up the on_change plumbing.
    3. State Restoration: Restores the component value snapshot to ensure survivability across Streamlit rerun() calls.
    4. Worker Lifecycle: Manages the WebRtcWorker (stopping on idle, creating on incoming offers, and flushing SDP answers).
    5. Callback Updates: Forwards updated user-provided callbacks (like video_frame_callback) to the running worker.

    If you encounter issues where the connection drops or state is lost during a rerun, the issue likely resides in the interaction between the State Restoration and Worker Lifecycle steps.

  7. How frame callbacks work for video and audio processing

    main

    Real-time media processing is achieved through callback functions that are executed for every frame of media as it flows through the application. You can implement custom logic for either video, audio, or both.

    • video_frame_callback(frame): Receives an av.VideoFrame, allows for processing (e.g., computer vision, filtering), and expects a modified av.VideoFrame to be returned.
    • audio_frame_callback(frame): Receives an av.AudioFrame, allows for processing (e.g., audio effects, sound analysis), and expects a modified av.AudioFrame to be returned.

    These callbacks run in real-time, making them suitable for applying machine learning models or signal processing to live streams.

  8. Accessing WebRtcStreamerContext properties

    main

    The WebRtcStreamerContext object provides access to various media stream components. Depending on the version and implementation, these properties are either forwarded directly from the internal worker or accessed via the .worker attribute.

    Available properties include:

    • video_processor
    • audio_processor
    • video_receiver
    • audio_receiver
    • source_video_track
    • source_audio_track
    • input_video_track
    • input_audio_track
    • output_video_track
    • output_audio_track
    • video_transformer (Deprecated since v0.20)

    Note on Lifecycle: If you encounter issues accessing these properties, it may be because the underlying worker is None. In newer implementations or when using the .worker pattern, you should check if the worker exists before accessing its attributes.

  9. Callback limitations and threading caveats

    main

    Callbacks (both frame callbacks and lifecycle hooks) are executed in forked threads (or the aiortc asyncio loop) independent of the main Streamlit thread. This leads to several critical limitations:

    • No Streamlit commands: You cannot call st.* methods (like st.write()) inside a callback.
    • Scope issues: Variables inside callbacks cannot be directly referred to from the outside, and the global keyword does not behave as expected.
    • Thread-safety required: Any shared state between the callback and the main script must be managed using thread-safe primitives like threading.Lock, queue.Queue, or threading.Event.
  10. Serve your app via HTTPS for remote deployment

    main

    Because streamlit-webrtc uses the getUserMedia() API, your app must be served over HTTPS to access local media devices (webcam/microphone) when hosted on a remote server.

    • Streamlit Community Cloud: Automatically serves apps via HTTPS by default.
    • Local Development: Use streamlit-remote to create an HTTPS tunnel via cloudflare or ngrok.

    To use streamlit-remote with a provider, install the provider command (like cloudflared or ngrok) separately.

    $ pip install streamlit-remote
    $ st-remote your_app.py --provider cloudflare  # Remote HTTPS URL via cloudflared
    $ st-remote your_app.py --provider ngrok      # Remote HTTPS URL via ngrok
    $ st-remote your_app.py --https self-signed --no-remote  # Local HTTPS only
  11. Set up the AI Issue Automation Workflow

    main

    This workflow uses GitHub Actions and anthropics/claude-code-action to automate issue triage and implementation. To enable it, you must register an Anthropic API key, provision specific labels, and configure GitHub Actions permissions.

    1. Register the Anthropic API key

    Store your key as a repository secret:

    1. Navigate to Settings → Secrets and variables → Actions.
    2. Click New repository secret.
    3. Name: ANTHROPIC_API_KEY.
    4. Value: Your Anthropic console API key.

    2. Provision Labels

    Run the setup script to create the required labels idempotently:

    bash scripts/setup-issue-labels.sh

    3. Configure Action Permissions

    To allow the ai-implement job to open Pull Requests, ensure the following settings are enabled in Settings → Actions → General → Workflow permissions:

    • Read and write permissions must be selected.
    • Allow GitHub Actions to create and approve pull requests must be enabled.
  12. Quick start with webrtc_streamer

    main

    To create a basic WebRTC stream, use the webrtc_streamer function. Note that unlike most Streamlit components, webrtc_streamer() requires a unique key argument.

    from streamlit_webrtc import webrtc_streamer
    
    webrtc_streamer(key="sample")