DOCSEARCHbyAlgolia
Submit Docs

Micdrop

website·Indexed 31 days ago

https://micdrop.dev

A framework for building real-time voice AI conversations on the web and mobile. Micdrop provides a Node.js server to orchestrate AI agents and client SDKs for the browser and React Native to handle microphone capture, audio playback, Voice Activity Detection (VAD), and turn detection.

Tokens
109.3K
Snippets
457
Records
716
Agent score
53%

What's inside Micdrop

  1. Overview of Micdrop AI Integration Architecture

    Micdrop uses a modular AI architecture that allows developers to mix and match providers for STT, TTS, and AI Agents to optimize for cost and quality. It supports three primary integration paths: using ready-made provided integrations, running models locally (no API keys, data stays on machine), or building custom integrations via abstract base classes.

    There are two primary architectural patterns supported:

    1. Pipeline Architecture: Splitting transcription (STT), the agent (LLM), and the voice (TTS) into three separate services.
    2. Realtime Architecture: Using a single connection to a realtime model (e.g., Gemini Live or OpenAI Realtime API) that handles hearing and speaking in one stream for lower latency.

  2. Overview of Micdrop local model components

    Micdrop allows running the entire conversation pipeline locally to ensure data privacy and eliminate the need for API keys. The pipeline is split into three main parts, each with local implementation options:

    • Agent (LLM): Uses @micdrop/ai-sdk to connect to any local server that speaks the OpenAI protocol (e.g., Ollama).
    • Speech-to-Text (STT): Uses @micdrop/whisper, which runs Whisper directly within the Node process.
    • Text-to-Speech (TTS): Multiple options available:
      • @micdrop/kokoro: Runs in Node process (English only).
      • @micdrop/piper: Runs as a subprocess (supports ~40 languages).
      • @micdrop/pocket-tts: Runs in Node process (English only, supports voice cloning).
      • @micdrop/qwen-tts: Requires an mlx-audio server (supports 10 languages).

  3. Overview of MicRecorder

    The MicRecorder class handles audio recording from a microphone using the browser's MediaRecorder API. It integrates with a Voice Activity Detection (VAD) system to identify when a user is speaking. It can be used as a standalone component for custom audio handling or as part of the MicdropClient for full voice conversation management.

  4. Overview of Micdrop for TypeScript web applications

    Micdrop is an open-source, MIT-licensed library for TypeScript web applications that allows developers to run the speech-to-text, LLM, and text-to-speech pipeline directly within their own Node.js server. Unlike managed call-center platforms, Micdrop handles web-based voice conversations within the application's own process, enabling direct payment to AI providers (tokens for models, characters for voice) rather than per-minute platform fees. It consists of two main packages: @micdrop/client for the browser and @micdrop/server for Node.js.

  5. Overview of Micdrop for Voice AI

    Micdrop is an MIT-licensed TypeScript framework for building voice AI agents. Unlike LiveKit Agents, which uses a WebRTC-based room/participant model and requires a media server (SFU), Micdrop uses a plain WebSocket connection between the browser and a Node.js server. It handles the voice pipeline (microphone capture, Voice Activity Detection (VAD), and playback) in the browser and orchestrates STT, LLM, and TTS on the server.

  6. Overview of Jev System One Model

    Jev is a System One model by TypeSafe AI designed for fast, intuitive judgment calls over text or JSON objects. Unlike LLMs that generate text token-by-token, Jev returns typed answers (labels, scores, or probabilities) in a single query. It is optimized for low latency (70ms to 500ms) and uses Reinforcement Learning for Calibrated Decisions (RLCD) to provide probabilities that accurately reflect the likelihood of a correct answer.

  7. Overview of Micdrop voice agent architecture

    Micdrop is an MIT-licensed TypeScript library that allows developers to run the voice AI loop (Speech-to-Text, LLM, and Text-to-Speech) directly within their own Node.js server instead of using a hosted orchestration platform. This approach eliminates per-minute platform fees, keeps audio data within the developer's own infrastructure, and allows for local debugging of the conversation logic.

    Key components include:

    • @micdrop/client: Handles browser-side microphone, speaker, voice activity detection, and connection.
    • @micdrop/server: Orchestrates the agent, STT, and TTS within the Node process.
    • @micdrop/react: Provides hooks for UI state management.

  8. Overview of Voice Activity Detection (VAD) in Micdrop

    Micdrop uses Voice Activity Detection (VAD) to detect speech and silence, sending audio chunks to the server only when speech is detected. To prevent losing the start of a sentence, Micdrop keeps the last few moments of audio in reserve and sends them once speech is triggered.

    Regarding interruptions:

    • If interruptions are enabled, user speech is recorded even while the assistant speaks (using echo cancellation).
    • If interruptions are disabled, audio captured while the assistant speaks (and for 300ms after) is ignored.
    • Micdrop automatically disables interruptions if the browser reports that echo cancellation is unavailable.

  9. Overview of the Micdrop WebSocket Protocol

    Micdrop uses a custom protocol over WebSocket for real-time bidirectional communication between the client and server. It is designed for efficiency by using client-side Voice Activity Detection (VAD) to send audio only when the user is speaking, reducing bandwidth and processing load compared to continuous streaming or WebRTC.

  10. Overview of the Micdrop STT abstraction

    The STT class is the core abstract base class in Micdrop used to integrate speech-to-text providers into real-time voice conversations. It extends EventEmitter and handles real-time audio stream processing, audio format detection, event emission for transcriptions, and resource cleanup.
    export abstract class STT extends EventEmitter<STTEvents> {
  public logger?: Logger
  // Transcribe audio stream to text (emits Transcript event)
  abstract transcribe(audioStream: Readable): void
  // Cleanup
  destroy(): void
}

  11. Overview of Jev System One Model

    Jev is a System One model by TypeSafe AI designed for software decision-making rather than text generation. It processes text or JSON state and returns calibrated probabilities, scores on a rubric, or a choice among labels. It is significantly faster than traditional LLMs (typically 70ms to 500ms per request) and is intended for tasks like routing, triage, guardrails, and voice commands.

  12. Estimate local model latency for TTS

    Local call response times vary by the Text-to-Speech (TTS) model used. Generally, a local call answers in about 1 second with Piper, Pocket TTS, or Qwen3-TTS, and approximately 2 seconds with Kokoro. Note that automatic prompts require a second pass, which increases the total turn time beyond the combined cost of transcription and voice generation.