DOCSEARCHbyAlgolia
Submit Docs

OmniSenseVoice Documentation

repository·main·Indexed 21 days ago

https://github.com/lifeiteng/omnisensevoice

A high-performance speech recognition solution built on SenseVoice, optimized for fast inference and precise timestamps. It provides a CLI for transcription and benchmarking, the OmniSenseVoiceSmall class for speech-to-text inference, and utilities for audio processing via WavFrontend and text processing via SentencepiecesTokenizer.

Tokens
4.8K
Snippets
19
Records
21
Agent score
25%

What's inside OmniSenseVoice

  1. Prepare LibriTTS dataset for benchmarking

    main

    To benchmark using the LibriTTS dataset, follow these steps using lhotse to download, prepare, and cut the audio files into segments:

    1. Download: lhotse download libritts -p dev-clean benchmark/data/LibriTTS
    2. Prepare: lhotse prepare libritts -p dev-clean benchmark/data/LibriTTS/LibriTTS benchmark/data/manifests/libritts
    3. Cut: Use lhotse cut simple to create the manifest files required for the benchmark command.
    DIR=benchmark/data
lhotse download libritts -p dev-clean benchmark/data/LibriTTS
lhotse prepare libritts -p dev-clean benchmark/data/LibriTTS/LibriTTS benchmark/data/manifests/libritts

lhotse cut simple --force-eager -r benchmark/data/manifests/libritts/libritts_recordings_dev-clean.jsonl.gz \
    -s benchmark/data/manifests/libritts/libritts_supervisions_dev-clean.jsonl.gz \
    benchmark/data/manifests/libritts/libritts_cuts_dev-clean.jsonl

  3. Run performance benchmarks with omnisense

    main

    Use the omnisense benchmark command to evaluate transcription speed and accuracy. You can control concurrency with --num-workers, batching with --batch-size, and hardware targeting with --device-id.

    omnisense benchmark -s -d --num-workers 2 --device-id 0 --batch-size 10 --textnorm woitn --language en benchmark/data/manifests/libritts/libritts_cuts_dev-clean.jsonl

  4. Reference: omnisense transcribe options

    main

    The following options are available for the omnisense transcribe command:

    • --language: Automatically detect the language or specify one of: auto, zh, en, yue, ja, ko.
    • --textnorm: Choose text normalization style: withitn (for inverse normalized) or woitn (for raw).
    • --device-id: Specify a GPU ID to run on. Defaults to -1 for CPU.
    • --quantize: Use a quantized model to achieve faster processing.
    • --help: Display detailed help information.

  5. Avoid pickling errors with SentencepiecesTokenizer in multiprocessing

    main

    When using SentencepiecesTokenizer within a multiprocessing.Process(), you may encounter the following error if the processor is initialized too early:

    TypeError: can't pickle SwigPyObject objects

    To prevent this, the SentencepiecesTokenizer implements lazy loading. The SentencePieceProcessor is not built during __init__, but is instead built on-demand when the first encoding or decoding method is called. This ensures the tokenizer object remains picklable when passed to child processes.

  6. Transcribe audio with the omnisense CLI

    main

    Use the omnisense transcribe command to perform speech recognition on an audio file. You can specify language detection, text normalization preferences, and hardware acceleration via CLI options.

    omnisense transcribe [OPTIONS] AUDIO_PATH

  8. Initialize WavFrontend for audio processing

    main

    The WavFrontend class provides a conventional frontend structure for Automatic Speech Recognition (ASR), handling feature extraction (Fbank) and normalization.

    When initializing, you can configure sampling frequency, window type, mel filterbank dimensions, and frame parameters. You can also provide a cmvn_file to apply Cepstral Mean and Variance Normalization (CMVN).

    from omnisense.utils.frontend import WavFrontend

frontend = WavFrontend(
    cmvn_file="path/to/cmvn.txt",
    fs=16000,
    window="hamming",
    n_mels=80,
    frame_length=25,
    frame_shift=10,
    lfr_m=1,
    lfr_n=1,
    dither=1.0
)

  9. Extract Fbank features using fbank_online()

    main

    Use the fbank_online() method for streaming or incremental audio processing. It maintains an internal state (fbank_fn) to process incoming chunks of waveforms. It returns only the new features generated from the provided waveform chunk.

    # For streaming audio chunks
feat, feat_len = frontend.fbank_online(waveform_chunk)

  10. Initialize OmniSenseVoiceSmall

    main

    The OmniSenseVoiceSmall class is the main entry point for performing speech recognition using the SenseVoice model. You can initialize it by providing a model directory, specifying a device (CPU or CUDA), and optionally enabling quantization.

    Parameters:

    • model_dir (Union[str, Path]): Path to the pretrained model weights and configuration.
    • device_id (Union[str, int]): The CUDA device ID to use. Defaults to "-1" (which maps to CPU if CUDA is unavailable).
    • device (Optional[str]): Explicit device string (e.g., 'cuda:0'). If not provided, it defaults to CPU or the specified device_id if CUDA is available.
    • quantize (bool): Whether to use a quantized version of the model. Defaults to False.
    from omnisense.models.sensevoice import OmniSenseVoiceSmall

# Initialize with a local model directory on GPU 0
model = OmniSenseVoiceSmall(model_dir="/path/to/model", device_id=0)

# Or initialize with quantization on CPU
model = OmniSenseVoiceSmall(model_dir="/path/to/model", device="cpu", quantize=True)

  11. Load CMVN files with load_cmvn()

    main

    The load_cmvn() function loads Cepstral Mean and Variance Normalization data from a text file. The file is expected to contain specific tags: <AddShift> (for means) and <Rescale> (for variances), both followed by a <LearnRateCoef> line containing the numeric values.

    Returns a numpy.ndarray of shape (2, dim), where the first row contains means and the second row contains variances.

    from omnisense.utils.frontend import load_cmvn

cmvn_data = load_cmvn("path/to/cmvn_file.txt")
# cmvn_data shape is (2, dim)

  12. Use SentencepiecesTokenizer for text processing

    main

    The SentencepiecesTokenizer class provides an interface for converting text to tokens (pieces or IDs) and vice versa using a SentencePiece model. It handles lazy loading of the underlying SentencePieceProcessor to ensure compatibility with Python's multiprocessing module, avoiding TypeError: can't pickle SwigPyObject objects errors.

    from pathlib import Path
from omnisense.utils.sentencepiece_tokenizer import SentencepiecesTokenizer

# Initialize with the path to your BPE model
tokenizer = SentencepiecesTokenizer(bpemodel=Path("path/to/your/model.model"))

# Convert text to subword pieces
tokens = tokenizer.text2tokens("Hello world")

# Convert subword pieces back to text
text = tokenizer.tokens2text(tokens)

# Convert text to token IDs
ids = tokenizer.encode("Hello world")

# Convert token IDs back to text
text_from_ids = tokenizer.decode(ids)

# Get vocabulary size
vocab_size = tokenizer.get_vocab_size()