TEN VAD Documentation

What is TEN VAD?

TEN VAD is a real-time voice activity detection (VAD) system designed for enterprise-grade conversational voice AI agents. It provides accurate frame-level speech activity detection.

Key Advantages

• High Precision: Offers superior precision compared to industry standards like WebRTC VAD and Silero VAD.
• Efficiency: Features lower computational complexity and reduced memory usage compared to Silero VAD.
• Low Latency: The architecture is optimized for temporal efficiency, which significantly reduces end-to-end response and turn detection latency in conversational AI systems.
• Cross-Platform: Supports multiple programming languages (Python, JS, Java, Go, C) and platforms (Linux, macOS, Windows, Android, iOS, Web/WASM).

TEN VAD is part of a larger ecosystem designed for conversational AI Agents. Developers building full-duplex dialogue systems may need to integrate other components from the TEN ecosystem:

• TEN Framework: The core open-source framework for conversational AI Agents.
• TEN Turn Detection: Enables full-duplex dialogue communication (working alongside VAD).
• TEN Agent Examples: A collection of use cases powered by the TEN framework.
• TEN Portal: The official site for documentation and news.

The TEN VAD WebAssembly build is composed of two distinct layers:

1. Static Library (libInferEngine.a): Contains the private or third-party inference engine and the neural network implementation.
2. WebAssembly Module (ten_vad.js + ten_vad.wasm): The complete VAD module that includes pre-processing and post-processing logic. It links against the static library to form the final runtime.

This separation allows the inference engine to be optimized independently before being integrated into the final WebAssembly module.

TEN VAD is part of a larger ecosystem designed for building conversational voice AI agents. The ecosystem includes:

• TEN Framework: The core framework.
• Agent Examples: Reference implementations for AI agents.
• VAD (TEN VAD): Voice Activity Detection.
• Turn Detection: Logic for detecting when a user has finished speaking.
• Portal: Interface/entry point for the ecosystem.

Build a Python extension module with pybind11 bindings. Navigate to the python directory within examples_onnx before running the build script.

Linux:

cd ten-vad/examples_onnx/python
./build-and-deploy-linux.sh --ort-path ~/onnxruntime-linux-$ARCH-$ONNX_VER

Output (ARM64): python/build-linux/lib/ten_vad_python.cpython-312-aarch64-linux-gnu.so Output (X86_64): python/build-linux/lib/ten_vad_python.cpython-312-x86_64-linux-gnu.so

macOS:

cd ten-vad/examples_onnx/python
./build-and-deploy-macos.sh --ort-path ~/onnxruntime-osx-$ARCH-$ONNX_VER

Output: python/build-macos/lib/ten_vad_python.*.so

cd python
./build-and-deploy-linux.sh --ort-path ~/onnxruntime-linux-$ARCH-$ONNX_VER

To use TEN VAD in an iOS application:

1. Generate Project: Run ./build-and-deploy-ios.sh in the examples/ directory to create Xcode project files.
2. Open Project: Open the generated .xcodeproj in Xcode (located in ./build-ios/ten_vad_demo.xcodeproj).
3. Target Configuration:
   • Select the ten_vad_demo target.
   • Set Edit Scheme $\rightarrow$ Run $\rightarrow$ Release.
   • Select an iOS Device (Note: Does not support Simulator or iPad).
4. Embed Framework: Drag ten_vad.framework (from ten_vad/lib/iOS/) into the "Frameworks, Libraries, and Embedded Content" section of your target. Set Embed to "Embed & Sign".
5. Signing: In Signing & Capabilities, modify the Bundle Identifier and specify a Provisioning Profile and Code Signing Identity.
6. Build & Run: Build the project in Xcode and run it on your device.

To use the ONNX model, you must download onnxruntime packages (version >= 1.17.1) from the official Microsoft repository.

1. Extract the onnxruntime package to get the include/ and lib/ directories.
2. Run the build script providing the absolute path to the ONNX Runtime root directory.

Note: The ONNX model is located in src/onnx_model. If running the demo from a different directory than the build directory, you must create a symbolic link to src/onnx_model/ to avoid loading failures.

When building the inference engine as a static library (libInferEngine.a), use specific compiler flags to optimize for WebAssembly size and performance. Key flags include -fno-rtti and -fno-exceptions to reduce binary size by 20-30%, and -s ALLOW_MEMORY_GROWTH=1 to support dynamic memory allocation.

# Compiler flags for static library
set(CMAKE_CXX_FLAGS
    "${CMAKE_CXX_FLAGS} \
    -std=c++11 \
    -s ALLOW_MEMORY_GROWTH=1 \
    -s LLD_REPORT_UNDEFINED \
    -fno-rtti \
    -fno-exceptions \
    -fPIE \
    -funroll-loops \
    -fvisibility=hidden \
    -finline-functions"
)

# Output: libInferEngine.a
add_library(Infer STATIC ${INFERENCE_ENGINE_SOURCES})

Before building TEN VAD with ONNX Runtime support, install the required system tools.

Linux (Ubuntu):

sudo apt update
sudo apt install cmake build-essential python3-venv curl

macOS: Install cmake using Homebrew:

brew install cmake

Node.js

Run the test script from the terminal:

cd ./examples
node test_node.js s0724-s0730.wav out.txt

Browser

1. Start a local Python server:

   python3 -m http.server 8000

2. Navigate to http://localhost:8000/examples/test_browser.html in your browser.

# Terminal usage
cd ./examples
node test_node.js s0724-s0730.wav out.txt