DOCSEARCHbyAlgolia
Submit Docs

fastrtc

repository·main·Indexed 26 days ago

https://github.com/gradio-app/fastrtc

A real-time communication library for Python that allows developers to turn any Python function into a real-time audio or video stream over WebRTC or WebSockets. It integrates with the Gradio SDK to support various multimodal AI demos, including voice chat, object detection, and live transcription.

Tokens
29.5K
Snippets
75
Records
148
Agent score
89%

What's inside fastrtc

  4. Core Concepts of the Stream object

    main

    The Stream object is the central component of FastRTC, used to stream audio, video, or both. It can be configured with different modes, modalities, and handlers to define how data flows between the client and server.

    Stream Modes

    • send-receive: Bidirectional streaming (default).
    • send: Client-to-server only.
    • receive: Server-to-client only.

    Modalities

    • video: Video streaming.
    • audio: Audio streaming.
    • audio-video: Combined audio and video streaming.

    Handlers

    The handler is the main argument for Stream. The required type depends on the modality and mode:

    Modalitysend-receivesendreceive
    videoFunction (frame $\rightarrow$ frame)Function (frame $\rightarrow$ frame)Function (frame $\rightarrow$ frame)
    audioStreamHandler or AsyncStreamHandler subclassStreamHandler or AsyncStreamHandler subclassGenerator yielding audio frames
    audio-videoAudioVideoStreamHandler or AsyncAudioVideoStreamHandler subclassNot Supported YetNot Supported Yet
    from fastrtc import Stream
import gradio as gr
import numpy as np

def detection(image, slider):
    return np.flip(image, axis=0)

stream = Stream(
    handler=detection, # (1)
    modality="video", # (2)
    mode="send-receive", # (3)
    additional_inputs=[
        gr.Slider(minimum=0, maximum=1, step=0.01, value=0.3) # (4)
    ],
    additional_outputs=None, # (5)
    additional_outputs_handler=None # (6)
)

  7. Explore FastRTC Application Examples

    main

    The FastRTC cookbook provides a collection of real-world applications demonstrating various capabilities of the library. You can explore implementations for:

    • Audio & Voice Chat: Real-time conversations with LLMs (Llama, Claude, GPT-4o), speech-to-speech models (Moshi, Ultravox), and integration with providers like ElevenLabs, Groq, and OpenAI.
    • Video & Computer Vision: Real-time object detection using YOLOv10 and RT-DETR on webcam streams or uploaded videos.
    • Real-time APIs: Direct integration with Google Gemini, OpenAI, and Azure Realtime APIs for low-latency voice/video interactions.
    • Transcription: Real-time speech-to-text using Whisper (both cloud-based via Groq and local via Transformers).
    • Specialized Tools: Voice-activated code editors, stop-word detection (e.g., 'Hello Llama'), and agentic voice assistants.

  10. Configure Cloudflare Calls API with Cloudflare API Token

    main

    If you have exhausted your free Hugging Face quota, you can use your own Cloudflare account.

    1. Create a TURN App in the Cloudflare Calls dashboard.
    2. Obtain your TURN_KEY_ID and TURN_KEY_API_TOKEN.
    3. Set these as environment variables: TURN_KEY_ID and TURN_KEY_API_TOKEN.
    4. Pass get_cloudflare_turn_credentials_async to the rtc_configuration parameter of the Stream class.
    from fastrtc import Stream, get_cloudflare_turn_credentials_async

# Make sure the TURN_KEY_ID and TURN_KEY_API_TOKEN environment variables are set
stream = Stream(
    handler=...,
    rtc_configuration=get_cloudflare_turn_credentials_async,
    modality="audio",
    mode="send-receive",
)

  12. Run Whisper Realtime Transcription demo

    main

    This demo provides realtime audio transcription using Whisper. It requires a Gradio SDK version of 5.16.0 or higher and utilizes WebRTC and WebSockets for low-latency streaming. To run this application, you must provide the following environment variables/secrets:

    • HF_TOKEN: Hugging Face token.
    • GROQ_API_KEY: API key for Groq services.

    The application entry point is app.py.