Echogarden

This approach is used for aligning spoken audio in one language with a translated transcript in another. It is often more accurate for non-English target languages and complex source languages (e.g., Chinese, Japanese) than single-stage Whisper alignment.

The Two Stages

1. Native Alignment: The spoken audio is aligned with the native language transcript (e.g., using the fast dtw engine).
2. Semantic Alignment: The resulting timeline is aligned with the translated text using semantic text-to-text alignment. This uses a multilingual text embedding model (defaults to e5) to produce vector representations of tokens. DTW is then applied over these vector sequences to align the two languages.

Benefits:

• Faster than single-stage Whisper alignment when using the dtw engine in stage 1.
• Higher accuracy for specific language pairs and smaller Whisper models.

Echogarden uses a custom package system to manage large assets like TTS voices and STT models. Instead of including everything in the base installation, components are downloaded on demand as .tar.gz files and extracted to a specific data directory.

Package ID Format

Packages follow a specific naming convention: [engine name]-[package id]-[date as yyyymmdd]

Data Directory Locations

Depending on your operating system, the [data-folder] where packages are stored is located at:

• Windows: %AppData%\Local\echogarden
• macOS: Users/User/Library/Application Support/echogarden
• Linux: /home/user/.local/share/echogarden

This engine aligns an existing timeline (timestamps) with a translation of its text without requiring the original audio. This is useful for reusing a single native-language timeline to align with multiple different translations.

Mechanism:

• Uses the e5 engine (Microsoft's multilingual E5 text embedding model).
• Encodes text tokens into vectors where words with similar meanings across different languages are mapped to similar vector spaces.
• Applies DTW over the vector sequences to map translated words to the timestamps in the original timeline.

// Engine identifier: `e5`

This engine aligns a spoken recording in one language with a translated transcript in a different language (typically English).

Mechanism:

1. The translated transcript is tokenized.
2. Tokens are decoded in order using a guided approach with a multilingual Whisper model set to its translate task mode.
3. The approximate mapping between the spoken audio and each word of the translation is estimated.

These engines annotate a transcript with approximate start and end timestamps for a given audio recording:

• dtw: Uses Dynamic Time Warping. Synthesizes the transcript with espeak, then aligns the synthesized waveform with the original audio.
• dtw-ra: Dynamic Time Warping with Recognition Assist. Uses a recognition engine to assist the DTW process by mapping synthesized waveforms back to the original audio using timing information from the recognizer.
• whisper: Uses the Whisper model to tokenize the transcript and decode tokens in a guided approach to derive word timestamps.

Echogarden can split output into multiple files based on detected segment boundaries using the [segment] placeholder in the output filename.

Example:

echogarden speak text.txt parts/[segment].opus

This creates files like parts/001 Hello world... .opus, where the placeholder is replaced by the segment index and its initial text.

Use the translate-speech operation to transcribe audio directly into a second language without an intermediate text translation step.

Note: Currently, only English is supported as a target language due to whisper engine limitations.

Key Options:

• --sourceLanguage=[lang]: The language spoken in the audio.
• --targetLanguage=[lang]: The language to translate into (must be en).

# Translate Spanish speech to English text
echogarden translate-speech speech.mp3 translation.txt --sourceLanguage=es --targetLanguage=en

To update your global installation of Echogarden, use the standard npm update command. If you want to check for newer major versions, you can use npm-check-updates (ncu).

# Simple update
npm update -g echogarden

# Checking for major updates using ncu
npm install -g npm-check-updates
ncu -g echogarden

To use the cuda execution provider, you must satisfy the following system requirements and perform a specific reinstallation step:

1. Install CUDA Toolkit: Download and globally install CUDA Toolkit 12.x.
2. Install cuDNN: Download and globally install cuDNN 9.x.
3. Reinstall Echogarden: You must reinstall the project after installing the toolkit and cuDNN. This is required because the onnxruntime-node dependency runs a postinstall script that detects global CUDA installations to conditionally download the necessary binaries for GPU support.

All echogarden CLI commands follow a consistent structure. Operations can accept one or more inputs and one or more outputs. Options must be provided using the --[optionName]=[value] format (the = is required).

General Syntax:

echogarden [operation] [one or more inputs..] [one or more outputs...] [options...]

Keyboard Shortcuts during playback:

• esc: Exit immediately
• enter: Skip current audio
• space: Pause/Resume
• right: Skip 1 second forward
• left: Skip 1 second backward