DOCSEARCHbyAlgolia
Submit Docs

voxtype

repository·dev·Indexed 20 days ago

https://github.com/peteonrails/voxtype

A high-performance, local-first push-to-talk voice-to-text tool for Linux and macOS. It supports multiple transcription engines, including Whisper and Cohere Transcribe, and integrates with Wayland compositors to type text directly at the cursor. Features include a Quickshell OSD for visual feedback, a native SwiftUI setup application for macOS, and integrations for Hammerspoon, Sway, and Waybar.

Tokens
106.2K
Snippets
419
Records
504
Agent score
70%

What's inside voxtype

  1. What is Parakeet and when to use it

    dev

    Parakeet is an alternative transcription engine for Voxtype based on NVIDIA's FastConformer-based speech recognition models. It uses ONNX Runtime and is optimized for high-speed CPU inference.

    Use Parakeet when:

    • You want excellent CPU performance without a GPU (especially with AVX-512 support).
    • You need proper punctuation and capitalization.
    • You are transcribing English (Parakeet TDT models are English-only).

    Use Whisper instead when:

    • You need multilingual support.
    • You want to avoid potential repetition hallucinations or proper noun substitution issues common in Parakeet.

  2. How Voxtype works on Linux

    dev

    Voxtype is designed to provide a seamless voice-typing experience on Linux by overcoming Wayland's security restrictions (which block global hotkeys and input simulation) and the fragmentation of Linux audio systems. It uses a specific stack of technologies to ensure compatibility and privacy:

    • Global Hotkeys: Uses evdev to read keyboard events directly from the kernel, bypassing the Wayland compositor.
    • Input Simulation: Uses ydotool (via the kernel's uinput interface) to simulate typing, with a clipboard fallback for compatibility.
    • Speech Recognition: Uses whisper.cpp, an optimized C++ port of OpenAI's Whisper model, for fast, local, offline recognition.
    • Audio Capture: Uses cpal to provide cross-platform audio support that works with PipeWire, PulseAudio, and ALSA.

  3. Use Soniox Realtime Streaming vs. Async Batch mode

    dev

    Soniox supports two primary operational modes:

    1. Realtime Streaming (streaming = true)

    Uses a live WebSocket session. Tokens stream back and are typed at the cursor as they arrive.

    • Requirement: Requires [hotkey] mode = "toggle".
    • Note: Push-to-talk is automatically promoted to toggle mode for the session to avoid clobbering compositor (Hyprland/Sway/River) key states.
    • type_partials: If true (default), non-final tokens are typed immediately (lower latency). If false, only finalized segments are typed at the cursor.

    2. Async Batch Mode (async_api = true)

    Uses the Soniox async transcription API (file upload + poll). Audio is buffered while the hotkey is held and sent in one shot upon release.

    • Accuracy: Uses the stt-async-v4 model, which is generally higher accuracy than the realtime model.
    • Latency: Typically 3-6 seconds total wait after releasing the key.
    • Compatibility: Fully compatible with push_to_talk mode.
    # Realtime Example
engine = "soniox"
[hotkey]
mode = "toggle"
[soniox]
streaming = true
type_partials = true

# Async Example
engine = "soniox"
[hotkey]
mode = "push_to_talk"
[soniox]
async_api = true

  4. Choose and configure a transcription engine

    dev

    Voxtype supports multiple engines. The default is whisper. Other engines are provided via ONNX and are optimized for specific use cases.

    EngineLanguagesBest For
    Whisper99General use, multilingual
    ParakeetEnglishFast English transcription
    MoonshineEnglishEdge devices, low memory
    SenseVoicezh, en, ja, ko, yueChinese, Japanese, Korean
    Paraformerzh+en, zh+yue+enChinese-English bilingual
    Dolphin40+ languagesEastern languages (no English)
    Omnilingual1600+Low-resource and rare languages

    To set the engine in config.toml:

    engine = "sensevoice"

    To override via CLI:

    voxtype --engine sensevoice

    Note: If building from source, you must enable the corresponding Cargo feature (e.g., --features parakeet) to use an ONNX engine.

    voxtype --engine sensevoice

  5. Configure Multi-Model setup for speed and accuracy

    dev

    You can set up a fast model for everyday use and a more accurate model that is loaded on-demand using a modifier key.

    Usage:

    • Normal hotkey press: Uses the primary model (fast).
    • Hold model_modifier + hotkey: Uses the secondary_model (accurate).

    Note: You must download the models first using the CLI.

    voxtype setup --download --model base.en
voxtype setup --download --model large-v3-turbo
    [hotkey]
key = "SCROLLLOCK"
model_modifier = "LEFTSHIFT"

[whisper]
model = "base.en"
secondary_model = "large-v3-turbo"
available_models = ["medium.en"]
max_loaded_models = 2
cold_model_timeout_secs = 300

[audio.feedback]
enabled = true

  6. How Voxtype's transcription engines and backends work

    dev

    Voxtype supports two primary runtime backends for transcription:

    1. Whisper (Default): Uses OpenAI's Whisper model via whisper.cpp. It supports three backend modes:

      • local: In-process transcription using whisper-rs.
      • cli: Transcription via a whisper-cli subprocess.
      • remote: Transcription via an HTTP API to a remote server.
      • Supports 99 languages.

    2. ONNX Engines: Uses ONNX Runtime to run various models. You can switch to this engine using the command voxtype setup onnx. Supported models include:

      • Parakeet (English)
      • Moonshine (English)
      • SenseVoice (zh/en/ja/ko/yue)
      • Paraformer (zh+en bilingual)
      • Dolphin (40 languages + Chinese dialects, no English)
      • Omnilingual (1600+ languages)

    Transcription results can optionally be piped through a Post-Process step. Any command that reads stdin and writes stdout can be used (e.g., piping to an LLM like Ollama for grammar correction).

    voxtype setup onnx

  7. How Voxtype's macOS architecture works

    dev

    The macOS implementation uses a thin SwiftUI GUI layer (VoxtypeSetup) that delegates all core functionality to the voxtype Rust binary via CLI calls.

    Key components of the macOS layer include:

    • VoxtypeCLI.swift: A wrapper that executes voxtype commands.
    • PermissionChecker.swift: Handles native macOS permission checks (Microphone, Accessibility, and Input Monitoring).

    This architecture ensures that business logic remains in the cross-platform Rust core while providing a native macOS user experience.

  10. Workarounds for compositors without submap support

    dev

    The automatic voxtype setup compositor fix only works for compositors that support input modes or submaps (Hyprland, Sway, River). It is not supported on Qtile, Niri, GNOME, or KDE.

    If you use an unsupported compositor, use one of these alternatives:

    1. Use a dedicated key without modifiers: Use keys like SCROLLLOCK, Pause, or F13-F24 which do not have modifiers to interfere.
    2. Use a post-processor delay: Add a delay to your configuration to allow time to release all keys before typing begins.
    3. Use evdev hotkeys: Use voxtype's built-in evdev hotkey detection instead of compositor keybindings.
    # Alternative 1: Dedicated key
[hotkey]
key = "SCROLLLOCK"

# Alternative 2: Post-processor delay
[output.post_process]
command = "sleep 0.3 && cat"
timeout_ms = 5000

  11. Understand the difference between Soniox Realtime and Async modes

    dev

    Voxtype supports two distinct Soniox API modes via the [soniox] async_api configuration key.

    Realtime Mode (async_api = false, default)

    Uses WebSockets for low-latency streaming. Tokens appear as you speak.

    • Latency: Partials appear within ~100ms; finals appear at utterance boundaries.
    • Model: stt-rt-v4.
    • Activation: Best used with mode = "toggle". If using push_to_talk, it is recommended to set streaming = false to avoid compositor issues.
    • Live Typing: By default, type_partials = true provides live cursor feedback. To only commit text when tokens are finalized, set type_partials = false.

    Async Mode (async_api = true)

    Uses REST-based file uploads and polling. Higher accuracy but higher latency.

    • Latency: ~3-6s wait after releasing the hotkey.
    • Model: stt-async-v4.
    • Activation: Compatible with push_to_talk mode.
    • Use Case: Better for accuracy and avoids live cursor artifacts.

    Automatic Mode Switching

    Voxtype automatically routes requests based on the context:

    • Dictation (Hotkey): Follows your async_api setting (defaults to Realtime).
    • Meeting (voxtype meeting start): Always uses Async REST mode for better stability and diarization, regardless of your config.

  12. Configure GPU Isolation for VRAM optimization

    dev

    On laptops or systems with hybrid graphics, you can use gpu_isolation = true. This runs transcription in a subprocess that exits after each recording, allowing the discrete GPU to power down. The model loads while you speak, so perceived latency is nearly identical to standard mode.

    [whisper]
model = "large-v3-turbo"
gpu_isolation = true

[audio.feedback]
enabled = true
theme = "default"