DOCSEARCHbyAlgolia
Submit Docs

ricky0123/vad

repository·master·Indexed 24 days ago

https://github.com/ricky0123/vad

A JavaScript library for accurate Voice Activity Detection (VAD) optimized for browser environments. It utilizes Silero VAD and ONNX Runtime Web to detect speech via microphone or pre-recorded audio. The project includes @ricky0123/vad-web for core browser functionality and @ricky0123/vad-react for React integration via the useMicVAD hook. It supports real-time detection with MicVAD and offline processing with NonRealTimeVAD.

Tokens
10.4K
Snippets
23
Records
51
Agent score
83%

What's inside ricky0123/vad

  1. Core Concepts of ricky0123/vad

    master

    The ricky0123/vad project provides Voice Activity Detection (VAD) for JavaScript, primarily targeting browser-based applications. It allows developers to detect user speech via microphone, capture audio segments, and trigger callbacks for UI updates or server-side processing.

    Key components include:

    • @ricky0123/vad-web: The core package for browser-based detection.
    • @ricky0123/vad-react: A React-specific wrapper for easier integration into React applications.
    • Silero VAD: The underlying machine learning model used for detection.
    • ONNX Runtime Web: The engine used to run the Silero VAD model in the browser.

    Note on Node.js support: Support for ricky0123/vad-node has been discontinued as of October 2024 to focus on browser-based use cases.

  2. How the VAD algorithm works

    master

    The Voice Activity Detection (VAD) algorithm processes audio through the following lifecycle:

    1. Resampling: Input audio is converted to a 16000 Hz sample rate.
    2. Framing: Audio is batched into frames (512 samples for v5 models, 1536 samples for legacy models).
    3. Probability Scoring: The Silero VAD model assigns a probability (0 to 1) to each frame indicating the likelihood of speech.
    4. State Management:
      • not speaking → speaking: Occurs when a frame's probability exceeds positiveSpeechThreshold.
      • speaking → not speaking: Occurs when frames for redemptionMs milliseconds have a probability below negativeSpeechThreshold (without hitting the positive threshold again).
      • Neutral Zone: Probabilities between negativeSpeechThreshold and positiveSpeechThreshold are ignored.
    5. Validation and Padding:
      • When a segment ends, the algorithm checks if the duration of high-probability frames meets minSpeechMs. If not, it is discarded as a false positive.
      • If valid, preSpeechPadMs of audio is prepended to the segment before it is returned via the API.

  3. Configure frame size for Silero v5

    master
    When using the Silero v5 model, the frameSamples parameter is automatically managed. The library sets the frame size to 512 samples for the "v5" model and 1536 samples for the "legacy" model. You do not need to configure frameSamples manually when using v5.

  4. Self-host VAD assets (WASM, ONNX, and Worklet files)

    master

    By default, VAD loads assets from a CDN. To serve them yourself, use the baseAssetPath and onnxWASMBasePath options in MicVAD.new() and ensure the following files are available at those paths:

    1. Under baseAssetPath:
      • vad.worklet.bundle.min.js
      • silero_vad_legacy.onnx
      • silero_vad_v5.onnx
    2. Under onnxWASMBasePath:
      • All .wasm files from onnxruntime-web.
      • All .mjs files from onnxruntime-web (required for newer versions).
    import { MicVAD } from "@ricky0123/vad-web"
const myvad = await MicVAD.new({
  baseAssetPath: "/", // path to worklet and onnx files
  onnxWASMBasePath: "/", // path to onnxruntime-web wasm/mjs files
  onSpeechEnd: (audio) => {
    // do something with `audio` (Float32Array of audio samples at sample rate 16000)...
  },
})
myvad.start()

  6. Use VAD via script tags

    master
    To use the Voice Activity Detector (VAD) directly in an HTML file using script tags, you must ensure that onnxruntime-web is included in a script tag and loaded prior to loading the VAD script. This is because the VAD depends on the ONNX Runtime Web environment to function.

  7. Set up a local development environment

    master

    To contribute to the project or develop locally, clone the repository and run the following commands from the top level of the repository to install dependencies and build all packages:

    1. Install dependencies: npm install
    2. Build all packages: npm run build
    npm install
npm run build

  8. Quick start with @ricky0123/vad-web via CDN

    master

    You can implement Voice Activity Detection in the browser using script tags. This requires onnxruntime-web to be loaded before the @ricky0123/vad-web bundle. The vad.MicVAD.new() method initializes the detector, which prompts the user for microphone permissions. You can provide an onSpeechEnd callback that receives a Float32Array containing the audio samples (at a 16000Hz sample rate) for the detected speech segment.

    <script src="https://cdn.jsdelivr.net/npm/onnxruntime-web/dist/ort.js"></script>
<script src="https://cdn.jsdelivr.net/npm/@ricky0123/vad-web@latest/dist/bundle.min.js"></script>
<script>
  async function main() {
    const myvad = await vad.MicVAD.new({
      onSpeechEnd: (audio) => {
        // do something with `audio` (Float32Array of audio samples at sample rate 16000)...
      }
    })
    myvad.start()
  }
  main()
</script>

  9. Quick Start: Use VAD via script tags in the browser

    master

    To use the Voice Activity Detector directly in a browser environment using script tags, you must include both the onnxruntime-web WASM files and the @ricky0123/vad-web bundle.

    When initializing vad.MicVAD.new, you must provide:

    • onSpeechStart: A callback triggered when speech is detected.
    • onSpeechEnd: A callback triggered when speech ends, receiving an audio parameter (a Float32Array of audio samples at a 16000 sample rate).
    • onnxWASMBasePath: The URL path to the onnxruntime-web WASM assets.
    • baseAssetPath: The URL path to the @ricky0123/vad-web assets.

    After initialization, call .start() on the VAD instance to begin detection.

    <script src="https://cdn.jsdelivr.net/npm/onnxruntime-web@1.22.0/dist/ort.wasm.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.29/dist/bundle.min.js"></script>
<script>
  async function main() {
    const myvad = await vad.MicVAD.new({
      onSpeechStart: () => {
        console.log("Speech start detected")
      },
      onSpeechEnd: (audio) => {
        // do something with `audio` (Float32Array of audio samples at sample rate 16000)...
      },
      onnxWASMBasePath:
        "https://cdn.jsdelivr.net/npm/onnxruntime-web@1.22.0/dist/",
      baseAssetPath:
        "https://cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.29/dist/",
    })
    myvad.start()
  }
  main()
</script>

  11. Quick Start with Script Tags

    master

    You can use the VAD directly in HTML via script tags without a build step. You must include the onnxruntime-web WASM file and the @ricky0123/vad-web bundle. When initializing vad.MicVAD.new(), you must provide the onnxWASMBasePath and baseAssetPath pointing to the respective CDN locations.

    <script src="https://cdn.jsdelivr.net/npm/onnxruntime-web@1.22.0/dist/ort.wasm.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.29/dist/bundle.min.js"></script>
<script>
  async function main() {
    const myvad = await vad.MicVAD.new({
      onSpeechEnd: (audio) => {
        // do something with `audio` (Float32Array of audio samples at sample rate 16000)...
      },
      onnxWASMBasePath:
        "https://cdn.jsdelivr.net/npm/onnxruntime-web@1.22.0/dist/",
      baseAssetPath:
        "https://cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.29/dist/",
    })
    myvad.start()
  }
  main()
</script>

  12. Integrate @ricky0123/vad-react into a Next.js project

    master
    To use @ricky0123/vad-react in a Next.js application, you must configure your next.config.js to handle the necessary assets (typically WebAssembly and worker files used by the underlying VAD engine). The implementation involves using the VAD React hook within your components, as demonstrated in the example project's src/pages/index.tsx.