VideoLingo

VideoLingo is an automated AI-powered video translation and dubbing tool designed to achieve Netflix-level subtitle quality. It automates the entire workflow of video localization, including subtitle segmentation, translation, alignment, and voice dubbing.

Key Features:

• Intelligent Subtitle Segmentation: Uses NLP and LLM technology to segment subtitles based on semantic meaning.
• Context-Aware Translation: Utilizes GPT-based terminology knowledge bases to ensure coherent and natural translations.
• Three-Step Translation Process: Employs a 'Direct Translation $\rightarrow$ Reflection $\rightarrow$ Paraphrasing' workflow to mimic professional human subtitling.
• Precise Subtitle Alignment: Uses WhisperX for word-level timestamp recognition to ensure perfect synchronization.
• High-Quality Dubbing: Supports multiple TTS (Text-to-Speech) solutions, including high-quality personalized dubbing via GPT-SoVITS.
• Developer Friendly: Features a structured file design for easy customization and extension.

VideoLingo is an automated AI video translation team that provides Netflix-level subtitle segmentation, translation, synchronization, and dubbing.

Key features include:

• Intelligent Subtitle Segmentation: Uses NLP and LLM technology to segment subtitles based on semantic meaning.
• Context-Aware Translation: Uses GPT to summarize and extract terminology knowledge bases for consistent, natural, and fluent translations.
• 3-Step Translation Process: Employs a 'Direct Translation - Reflection - Paraphrasing' workflow to ensure professional quality.
• Precise Subtitle Synchronization: Utilizes WhisperX for word-level timeline recognition to ensure accurate synchronization.
• High-Quality Dubbing: Supports multiple TTS (Text-to-Speech) solutions, including GPT-SoVITS, for personalized and high-quality dubbing.
• Developer Friendly: Features a structured file design for easy customization and supports multiple deployment methods.

Videolingo is a modular video translation system that executes a multi-step pipeline. Developers can debug the system by executing individual modules (often named with a _X_*.py numbering scheme) within the core directory.

The Pipeline Stages:

1. Video Acquisition: Downloading via yt-dlp (core/_1_ytdlp.py).
2. Audio/ASR: Vocal separation via Demucs (core/asr_backend/demucs_vl.py) and transcription via WhisperX (local or 302.ai API) or ElevenLabs API (core/asr_backend/whisperX_local.py, core/asr_backend/whisperX_302.py, core/asr_backend/elevenlabs_asr.py).
3. Text Processing: Sentence splitting using spaCy (core/spacy_utils) and semantic splitting/translation using LLMs (core/_3_2_split_meaning.py, core/translate_lines.py).
4. Subtitles: Generating SRT files (core/_6_gen_sub.py) and optionally burning them into the video (core/_7_sub_into_vid.py).
5. Dubbing: Generating TTS audio via various backends (core/tts_backend/) and merging it with the original video/background music (core/_12_dub_to_vid.py).

Output Management:

• Intermediate/Final Outputs: Stored in the output directory.
• Archiving: The core/utils/onekeycleanup.py utility moves files from output to a structured history directory based on the video name.

Be aware of the following current limitations when using VideoLingo:

• Transcription Accuracy: WhisperX transcription can be affected by heavy background music. Use the vocal separation enhancement feature for such videos. Additionally, subtitles ending in numbers or special symbols might be truncated because wav2vec may fail to map characters like "1" to their phonetic form "one".
• LLM Strictness: Weaker models may fail due to strict JSON requirements. (See Troubleshooting).
• Dubbing Nuances: While engineering handles speech rate adjustments, dubbing may not be 100% perfect due to language-specific prosody and speed differences.
• Language Detection: Transcription only retains the primary language of a video. WhisperX uses language-specific models for word-level alignment, which may discard other languages present in the audio.
• Speaker Diarization: The tool currently cannot perform multi-character separate dubbing because WhisperX's speaker differentiation is not yet sufficient for this task.

To use GPT-SoVITS, place the GPT-SoVITS-v2-xxx folder in the same directory as VideoLingo (they should be sibling folders).

Configuration Methods:

1. Custom Trained Model:

   • Copy your trained tts_infer.yaml from GPT-SoVITS-v2-xxx/GPT_SoVITS/configs/ to the same directory and rename it to an English name (e.g., MyVoice.yaml).
   • Place your reference audio in the same directory named as {EnglishName}_{TextContent}.wav (e.g., MyVoice_Hello.wav).
   • In the VideoLingo web sidebar, set GPT-SoVITS 角色 to your English name.

2. Using Pre-trained Models:

   • Download and overwrite files in GPT-SoVITS-v2-xxx with provided models.
   • Set GPT-SoVITS 角色 to Huanyuv2 in the web sidebar.

3. Manual Model Path Configuration:

   • Place .ckpt files in GPT_weights_v2 and .pth files in SoVITS_weights_v2.
   • Edit your custom .yaml file to point to these paths:

   t2s_weights_path: GPT_weights_v2/Huanyu_v2-e10.ckpt
   version: v2
   vits_weights_path: SoVITS_weights_v2/Huanyu_v2_e10_s150.pth

   • Ensure the reference audio follows the {EnglishName}_{TextContent}.wav naming convention.

Important: Always use English for the character name to avoid errors. In the web sidebar, ensure 参考音频模式 (Reference Audio Mode) is selected.

# Example configuration for custom weights
t2s_weights_path: GPT_weights_v2/Huanyu_v2-e10.ckpt
version: v2
vits_weights_path: SoVITS_weights_v2/Huanyu_v2_e10_s150.pth

To use GPT-SoVITS-v2 for high-quality local voice cloning, follow these requirements:

1. Directory Structure: GPT-SoVITS-v2-xxx and VideoLingo must be parallel folders in the same parent directory.
2. Configuration Methods:
   • Self-trained model: Copy the auto-filled tts_infer.yaml from the GPT-SoVITS config folder, rename it to your preferred English character name, and place a reference audio file in the same directory named [character_name]_reference_audio_[text].wav (e.g., Huanyuv2_Hello, this is a test audio.wav).
   • Pre-trained model: Overwrite the GPT-SoVITS-v2-xxx folder with the provided model and set GPT-SoVITS Character to Huanyuv2 in the VideoLingo sidebar.
   • Other trained models: Place .ckpt and .pth files in their respective GPT_weights_v2 and SoVITS_weights_v2 folders. Modify the renamed tts_infer.yaml to point to these paths:

# Example config for custom models
t2s_weights_path: GPT_weights_v2/Huanyu_v2-e10.ckpt
version: v2
vits_weights_path: SoVITS_weights_v2/Huanyu_v2_e10_s150.pth

Warning: Always use English for the character_name to avoid errors. The reference_audio_text can be in Chinese.