Vision Agents by Stream

Open Vision Agents by Stream is a framework for building multi-modal AI agents capable of watching, listening, and understanding video in real-time. It is designed for low-latency applications, supporting join times of ~500ms and audio/video latency as low as 30ms.

Key capabilities include:

• Real-time Video AI: Integration with models like YOLO and Roboflow alongside Gemini/OpenAI realtime.
• Model Agnostic: Uses native SDK methods from OpenAI (create response), Gemini (generate), and Claude (create message), ensuring compatibility with the latest model features.
• Flexible Infrastructure: While optimized for Stream's edge network, it is designed to work with any video edge network.
• Multi-platform SDKs: Support for React, Android, iOS, Flutter, React Native, and Unity.

The Fast Whisper STT plugin offers the following capabilities:

• Fast Inference: Uses the faster-whisper CTranslate2-based implementation.
• Model Sizes: Supports tiny, base, small, medium, large, large-v2, and large-v3.
• Language Support: Supports automatic language detection or manual language specification.
• Hardware Acceleration: Supports both CPU and GPU.
• Quantization: Supports int8, float16, and float32 quantization.

The Sarvam AI plugin provides three main capabilities:

• STT (Speech-to-Text): WebSocket streaming speech-to-text (Saarika / Saaras) with Voice Activity Detection for turn events.
• TTS (Text-to-Speech): WebSocket streaming text-to-speech (Bulbul) with configurable speaker, pace, and language.
• LLM (Large Language Model): OpenAI-compatible chat completions (Sarvam-30B / Sarvam-105B / Sarvam-M) using the ChatCompletionsLLM interface.

Authentication: All services require the SARVAM_API_KEY environment variable. The plugin sends this key via the api-subscription-key header.

To quickly test the agent with a browser-based participant:

1. Open the Tencent TRTC Web SDK quick demo.
2. Enter your TENCENT_SDK_APP_ID and TENCENT_SDK_SECRET_KEY into the demo form to generate a UserSig.
3. Click Enter Room and Start Local Video.
4. Copy the RoomID(String) from the demo.
5. Run the agent using Docker, passing the room ID:

cd plugins/tencent
TENCENT_TEST_ROOM_ID=<paste-room-id-here> docker compose run --rm tencent-agent

This example builds a voice bot that joins a Stream video call, transcribes participants using Cartesia STT (Ink), generates LLM responses via OpenAI, and speaks back using Cartesia TTS (Sonic).

Prerequisites

• Stream Account: API credentials from the Stream Dashboard.
• Cartesia Account: API key from Cartesia.
• OpenAI Account: An OPENAI_API_KEY for the LLM.
• Python 3.10+

To maintain consistency, all plugin README.md files should follow this standardized format:

1. # Plugin Name: The name of the plugin.
2. ## Features: A bulleted list of main functionalities.
3. ## Installation: Installation instructions using uv add (e.g., uv add vision-agents[PLUGIN-NAME]).
4. ## Usage: A basic code snippet demonstrating how to initialize and use the plugin.
5. ## Configuration: A table containing parameter names, descriptions, and accepted values/defaults.
6. ## Dependencies: A bulleted list of required dependencies.

To use Qwen Realtime features, install the vision-agents package with the qwen extra using uv:

uv add "vision-agents[qwen]"

To use the Qdrant RAG plugin for hybrid search (dense vector + BM25 sparse embeddings), install it using uv with the qdrant extra or by installing the plugin package directly.

uv add "vision-agents[qdrant]"
# or directly
uv add vision-agents-plugins-qdrant

To use TwelveLabs Pegasus video understanding in your Vision Agents project, install the plugin using uv:

uv add "vision-agents[twelvelabs]"
# or directly
uv add vision-agents-plugins-twelvelabs

You can obtain an API key at https://twelvelabs.io.

To add Speechify Text-to-Speech (TTS) functionality to your project, install the plugin using uv with the speechify extra or by installing the plugin package directly.

uv add "vision-agents[speechify]"
# or directly
uv add vision-agents-plugins-speechify