VOICEVOX CORE

repository·main·Indexed 22 days ago

https://github.com/voicevox/voicevox_core

The central speech synthesis engine for the VOICEVOX ecosystem, providing high-quality text-to-speech capabilities. It features a C API with language-specific bindings for Python and Java, and supports deployment across Windows, Linux, macOS, iOS, and Android.

Tokens
34.9K
Snippets
106
Records
153
Agent score
77%

What's inside voicevox_core

  1. Overview of VOICEVOX CORE

    main

    VOICEVOX CORE is the speech synthesis engine core for VOICEVOX. It provides the underlying logic for text-to-speech synthesis and is available as a library that can be integrated into various applications.

    Key components and related projects:

    • VOICEVOX: The editor application.
    • VOICEVOX ENGINE: The engine layer.
    • Pre-built Binaries: Available in the Releases section, including:
      • C API dynamic libraries (.so, .dll, .dylib)
      • Python API wheels (.whl)
  2. What is a VVM file

    main

    A VVM file is a ZIP-format archive containing all the voice information required for speech synthesis. It uses the .vvm extension.

    A VVM file typically contains a manifest.json file, a metas.json file, and various model files (such as .onnx or .bin) for duration, intonation, and decoding.

    Internal directory structure of a .vvm archive:

    • {filename}.vvm
      • manifest.json
      • metas.json
      • <duration_model>
      • <intonation_model>
      • <decode_model>

    Example: A sample.vvm might contain predict_duration.onnx, predict_intonation.onnx, and decode.onnx.

  3. What is a VVM manifest file

    main

    The manifest file is a JSON file located at the root of a VVM archive as manifest.json. It describes the composition of the VVM file and provides the necessary information to load and utilize the contained ONNX models and other assets.

    The schema for this file is defined by the Manifest struct within the voicevox_core source code.

  4. Understand consonant erosion in singing synthesis

    main

    When Synthesizer.create_sing_frame_audio_query generates FrameAudioQuery.phonemes from a Score, the Note.lyric is split into consonants and vowels.

    The vowel start position is aligned with the note's start position. Because consonants occur before vowels, they 'erode' (occupy time from) the preceding note.

    Example: If a note starts with the syllable "do", the consonant "d" will take a few frames from the end of the previous rest or note, and the vowel "o" will occupy the remainder of the current note's duration.

  5. Understand the Text-to-Speech (TTS) process flow

    main

    The standard TTS process in VOICEVOX CORE involves transforming Japanese text into a sequence of accent phrases, then into an AudioQuery, and finally into a WAV audio file.

    Standard Granular Workflow

    1. Analyze Text: Use OpenJtalk.analyze to convert Japanese text into a list of AccentPhrase objects (without pitch/mora length data).
    2. Enrich Accent Phrases: Use Synthesizer.replace_phoneme_length and Synthesizer.replace_mora_pitch to add pitch and mora length data to the accent phrases.
    3. Create AudioQuery: Convert the enriched accent phrases into an AudioQuery using AudioQuery.from_accent_phrases.
    4. Synthesize: Generate the final WAV audio from the AudioQuery using Synthesizer.synthesis.

    Shortcut API

    For most use cases, you can bypass the manual steps using the Synthesizer.tts method, which takes Japanese text directly and returns the synthesized WAV audio.

  6. Understand DLL/Shared Library loading

    main

    The Java API relies on loading native libraries. The method of loading depends on the platform:

    Android

    Libraries are loaded from jniLibs using System.loadLibrary.

    Non-Android Platforms

    On desktop platforms, the appropriate DLL/shared library from src/main/resources/dll is copied to a temporary directory and loaded via System.load.

    The expected library names are:

    • Windows: voicevox_core_java_api.dll
    • Linux: libvoicevox_core_java_api.so
    • macOS: libvoicevox_core_java_api.dylib

    If the library is not found automatically, System.loadLibrary can be used for debugging purposes.

  7. Minimum supported version policy for VOICEVOX CORE

    main

    VOICEVOX CORE follows a general policy of supporting language versions released approximately 3 years prior to the current release. This ensures compatibility with environments created around that time with minimal code changes. However, specific language ecosystems have different rules:

    • Python API: The minimum version is not updated even after 3 years unless issues arise. If problems occur, the minimum supported version may be raised to approximately 3 years old.
    • Rust API: This policy does not apply to Rust. The Rust ecosystem assumes the use of the latest versions, and backporting to older versions is not performed.
  8. API Design Principles for VOICEVOX CORE

    main

    VOICEVOX CORE's API design follows these core principles:

    1. Relationship with ENGINE API: The CORE API is based on the ENGINE API. While they are closely aligned, deviations are permitted if they significantly improve usability for CORE users. It is acceptable for CORE to include features not present in the ENGINE.
    2. Language Consistency: The core functionality is implemented in Rust. To maintain feature consistency, new features are generally not added via language-specific wrappers alone; they must be implemented in the core Rust layer.
    3. Idiomatic Language Wrappers: While features must remain consistent, wrappers should adapt to the idiomatic patterns of the target language (e.g., Python, Java, TypeScript) rather than strictly mimicking Rust's syntax.
  9. Use asynchronous processing with asyncio

    main

    For asynchronous workflows, use the voicevox_core.asyncio module instead of voicevox_core.blocking.

    Important Considerations:

    • Concurrency Limits: You cannot perform simultaneous synthesis using the same voice model instance (it is protected by a Mutex).
    • Performance: Because the underlying ONNX Runtime performs its own optimizations, using asyncio for performance gains is often ineffective. However, it can help with responsiveness if you reduce synthesizer.cpu_num_threads to allow other tasks to run while a long synthesis is in progress.

    Example:

    from voicevox_core.asyncio import Onnxruntime, OpenJtalk, Synthesizer, VoiceModelFile
    
    # 1. Synthesizer initialization (async)
    open_jtalk_dict_dir = "dict/open_jtalk_dic_utf_8-1.11"
    synthesizer = Synthesizer(await Onnxruntime.load_once(), await OpenJtalk.new(open_jtalk_dict_dir))
    
    # 2. Load voice model (async context manager)
    async with await VoiceModelFile.open("models/vvms/0.vvm") as model:
        await synthesizer.load_voice_model(model)
    
    # 3. Text-to-speech (async)
    wav = await synthesizer.tts("サンプル音声です", 0)
    from voicevox_core.asyncio import Onnxruntime, OpenJtalk, Synthesizer, VoiceModelFile
    
    # 1. Synthesizerの初期化
    open_jtalk_dict_dir = "dict/open_jtalk_dic_utf_8-1.11"
    synthesizer = Synthesizer(await Onnxruntime.load_once(), await OpenJtalk.new(open_jtalk_dict_dir))
    
    # 2. 音声モデルの読み込み
    async with await VoiceModelFile.open("models/vvms/0.vvm") as model:
        await synthesizer.load_voice_model(model)
    
    # 3. テキスト音声合成
    wav = await synthesizer.tts("サンプル音声です", 0)
  10. Build the VOICEVOX CORE Python API

    main

    The Python bindings are built using maturin. You can build the project in two ways:

    1. Editable Installation (Development)

    Use maturin develop to compile the Rust code into a .pyd (or .so/.dylib) file located under python/voicevox_core. This installs the package in editable mode, which is ideal for development.

    maturin develop --locked

    2. Wheel Build (Release)

    Use maturin build to create a standard Python wheel (.whl) file.

    maturin build --release --locked
    ❯ maturin develop --locked