SimulStreaming Documentation

SimulStreaming is a tool designed for simultaneous (streaming) processing of speech-to-text (ASR) and Large Language Model (LLM) translation. It is optimized for processing authentic long-form speech and is designed to be efficient enough for real-time applications.

Key Capabilities:

• Multilingual Support: Supports 99 Whisper source languages and 35 EuroLLM target languages.
• Processing Modes:
  • Direct Speech-to-Text: Uses Whisper to translate from 99 languages into English, or for transcription in the source language.
  • Cascade Translation: A pipeline of Whisper (speech-to-text) followed by EuroLLM (text-to-text) for translation into 35 target languages.
• Hardware Optimization: Optimized for 1–2 GPUs (e.g., running Whisper large-v3 1.5B and EuroLLM 9B simultaneously), though smaller distilled models can be used.
• Advanced Features: Supports flexible prompting, in-domain terminology, and Retrieval Augmented Generation (RAG).

The output is a JSONL stream where each line represents an update. The translation uses a LocalAgreement mechanism with a dual-part text buffer:

• text: Confirmed partial output. This can be extended but never changed.
• unconfirmed_text: A speculative buffer that may change in subsequent updates. A prefix of this text might eventually be confirmed and presented to users.

JSON Schema

{
  "emission_time": 8.838154792785645,  // Simulation time in seconds
  "end": 0.66,                        // End of the last processed audio segment (seconds)
  "status": "INCOMPLETE",            // "INCOMPLETE" (processing ongoing) or "COMPLETE" (update finished)
  "text": "",                        // Confirmed partial output
  "unconfirmed_text": "So...",       // Speculative output that may change
  "is_final": false                  // True if the upstream STT detected end of voiced segment
}

Note on is_final: When true, the last unconfirmed_text is immediately emitted as confirmed, and all buffers are cleared for a fresh context.

To use the LLM translation feature, you must install the required software dependencies and prepare a CTranslate2-compatible model.

1. Install dependencies:

   pip install -r requirements_translate.txt

2. Prepare the model: Download a model from Huggingface (e.g., EuroLLM-9B-Instruct) and convert it to the CTranslate2 format using ct2-transformers-converter.

   Example using EuroLLM-9B-Instruct:

   # Clone the model (requires HF access)
   git clone https://huggingface.co/utter-project/EuroLLM-9B-Instruct
   
   # Install converter dependencies
   pip install transformers[torch]
   
   # Convert to CTranslate2 format
   ct2-transformers-converter --model EuroLLM-9B-Instruct/ --output_dir ct2_EuroLLM-9B-Instruct

3. Reference paths: When running the translation script, use --model-dir ct2_EuroLLM-9B-Instruct and --tokenizer-dir EuroLLM-9B-Instruct to point to these directories.

Use simulstreaming_whisper.py to simulate live streaming from a local audio file.

Recommended Hardware: A GPU with at least 10GB VRAM is recommended for running the large-v3 model efficiently. While it works on CPU, it may be too slow for real-time applications.

Example Command:

python3 simulstreaming_whisper.py audio.wav --language en --task transcribe --comp_unaware --vac

Simulation Modes:

• Default (Computationally Aware): Real-time simulation where the chunk size is MIN_CHUNK_SIZE or larger depending on processing time.
• --comp_unaware: Computationally unaware simulation. The timer for emission times 'stops' during model computation, meaning latency is only caused by model ambiguity/confirmation, not hardware speed. This is used to find the lower bound for latency.
• --start_at START_AT: Starts processing at a specific timestamp in the audio file. Useful for debugging specific segments without waiting for the whole file to play.

You can simulate real-time LLM translation by feeding a JSONL file (typically generated by Whisper) into simulstreaming_translate.py.

Simulation Modes

• Default (Computationally Aware): Real-time simulation where the timer accounts for processing time. The chunk size is MIN_CHUNK_SIZE or larger if more words arrived during the last computation.
• --comp_unaware: Computationally unaware simulation. The timer 'stops' during model computation. This is used to find the lower bound of latency by isolating latency caused by language ambiguity from hardware/implementation speed.

Example Workflow

1. Process audio with Whisper to create a JSONL file:

   python3 simulstreaming_whisper.py audio.wav --language en --task transcribe --comp_unaware --vac > output.jsonl

2. Run the translation simulation:

   python3 simulstreaming_translate.py --src-lang en --tgt-lan de --comp_unaware --input-jsonl output.jsonl > output-llm.jsonl

The simulstreaming_translate.py script provides several options for controlling the translation simulation:

By default, the tool outputs JSONL to stdout. Each line represents a partial text update or an end-of-voice signal.

Partial Text Output Example:

{"start": 0.332, "end": 0.832, "text": " And so,", "tokens": [400, 370, 11], "words": [{"start": 0.332, "end": 0.332, "text": " And", "tokens": [400]}, {"start": 0.532, "end": 0.532, "text": " so", "tokens": [370]}, {"end": 0.832, "text": ",", "tokens": [11]}], "is_final": false, "emission_time": 2.24602843309326}

Field Definitions:

• start / end: Timestamps (seconds) of the audio segment where the output was detected.
• text: The partial text output produced in this update.
• tokens: List of token IDs used by the Whisper tokenizer.
• words: Detailed word-level view including individual start, end, text, and tokens.
• is_final: Boolean flag indicating end of voice (only used with --vac).
• emission_time: Simulation time in seconds. In computationally aware mode, this is real time from the start; in computationally unaware mode, it is the length of incoming audio.

The simulstreaming_whisper.py script accepts the following arguments:

General Options:

• -h, --help: Show help message.
• -l, --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}: Set log level.
• --logdir LOGDIR: Directory to save audio segments and generated texts for debugging.
• --out-txt: Output formatted as simple space-separated text instead of JSONL.
• --model_path MODEL_PATH: Path to Whisper .pt model (downloads automatically if not found).
• --beams BEAMS, -b BEAMS: Number of beams for beam search. If 1, GreedyDecoder is used.
• --decoder DECODER: Override automatic selection of beam or greedy decoder.

WhisperStreaming Processor Arguments:

• --min-chunk-size MIN_CHUNK_SIZE: Minimum audio chunk size in seconds. The system waits up to this time to process.
• --lan, --language LAN: Source language code (e.g., en, de, cs, or auto).
• --task {transcribe,translate}: Set task to transcribe or translate.
• --vac: Use Voice Activity Controller (requires torch).
• --vac-chunk-size VAC_CHUNK_SIZE: VAC sample size in seconds.

Audio Buffer & Alignment:

• --audio_max_len AUDIO_MAX_LEN: Max length of the audio buffer in seconds.
• --audio_min_len AUDIO_MIN_LEN: Skip processing if buffer is shorter than this.
• --frame_threshold FRAME_THRESHOLD: Threshold for attention-guided decoding (in frames; 1 frame = 0.02s for large-v3).

CIF (Truncation) Arguments:

• --cif_ckpt_path CIF_CKPT_PATH: Path to Simul-Whisper's CIF model checkpoint to detect end-of-word.
• --never_fire | --no-never_fire: Override CIF model. If --never_fire is True, the last word is NEVER truncated.