Whisper Flow Documentation

Metavox is an automated service designed to translate and narrate video content into different languages. It is intended for content creators (e.g., YouTube bloggers) to expand international reach without manual dubbing.

The service automates a four-step workflow:

1. Extract and Transcribe Audio: Uses Speech-to-Text (STT) to convert original video audio into text.
2. Translate Text: Uses Translation APIs to convert the transcribed text into target language(s).
3. Generate Narration: Uses Text-to-Speech (TTS) to synthesize natural-sounding narration in the target language.
4. Seamlessly Edit Video: Combines the new audio with original visuals, attempting to synchronize lip movement and maintain visual quality.

Metavox supports both local video files and direct input from YouTube.

Whisper Flow provides real-time transcription by receiving streaming media as a series of sequential data packets or 'chunks'. Unlike batch processing (where a full file is processed at once), Whisper Flow transcribes these chunks on the spot.

To manage the stream, Whisper Flow uses a tumbling window technique, gathering events into segments until specific conditions are met. The transcription is returned as a series of events. Each event contains a segment of transcribed speech, and the service provides updates as the segment grows until it is complete.

Each transcription event includes:

• Transcript: The transcribed text.
• EndTime: The timestamp of the segment.
• IsPartial: A boolean indicating if the result is a partial transcription (True) or a finalized segment (False).

While pursuing monetization, use supporting or guardrail metrics to ensure you are not eroding the project's foundation or user experience. Recommended guardrail metrics include:

• Activation rate: Users who reach their first transcription value.
• Retention / churn: User stickiness.
• Funnel drop-off: Identifying friction points in the visit → signup → activate → pay flow.
• Cost to serve: Inference/compute cost per active user (critical for maintaining WhisperFlow's latency goals).
• Community health: Monitoring stars, contributors, and issues to ensure monetization doesn't degrade the free experience.

To maintain a part-time pace and avoid building before demand is proven, follow these build/refuse guidelines:

Build (Phase B only):

• API keys
• Per-minute metering
• Stripe billing
• Landing page
• Basic onboarding

Refuse (Until pulled by a paying customer):

• Dashboards
• Multi-region support
• New/larger ML models
• Fancy UI
• Full product apps (e.g., MetaVox, Linga, LitMind, SlideCrafter)

Whisper-Flow provides two types of real-time transcription results to balance low latency with high accuracy:

• Partial results: These appear immediately as you speak. They provide instant feedback based on the system's current predictions, even before a sentence is finished.
• Non-partial results: These are the finalized transcriptions. Once the input is complete, Whisper-Flow refines and confirms the text to ensure higher accuracy.

When implementing client-side insertion logic for WhisperFlow, you must account for the fact that partials are not monotonic.

Whisper's partial results may rewrite previously emitted words as more context becomes available. To handle this correctly:

1. Maintain a pending range in the editor.
2. Replace the entire pending range with the new partial on every update.
3. Never append partials to existing text.
4. Only commit the text to the document when is_partial is false (indicating a final result).

Run the Whisper-Flow server using uvicorn. By default, the server listens on all interfaces (0.0.0.0) at port 8181 using the whisperflow.fast_server:app entrypoint.

uvicorn whisperflow.fast_server:app --host 0.0.0.0 --port 8181

You can deploy Whisper Flow using the provided helper script or manually via Docker commands.

Using the helper script

Running ./run.sh -docker will stop/remove existing containers/images, build a fresh image, and run the container on port 8888.

Manual Docker setup

To run the container on port 8181 manually:

1. Build the image using Dockerfile.test.
2. Run the container in detached mode.
3. Monitor logs or stop the container as needed.

# Using the helper script (runs on port 8888)
./run.sh -docker

# Manual Setup (runs on port 8181)
# Build the image
docker build -t whisperflow-image --file Dockerfile.test .

# Run the container
docker run --name whisperflow-container -p 8181:8181 -d whisperflow-image

# Check logs
docker logs whisperflow-container

# Stop the container
docker stop whisperflow-container