DOCSEARCHbyAlgolia
Submit Docs

AutoSubs Documentation

repository·main·Indexed 25 days ago

https://github.com/tmoroney/auto-subs

A local-first AI subtitle generation tool providing transcription and speaker diarization without cloud dependencies. It features a cross-platform desktop app and integrations for Adobe Premiere Pro, After Effects, and DaVinci Resolve. The system utilizes Whisper, Parakeet, or Moonshine models for transcription and Pyannote for speaker identification, with a dedicated Adobe extension that bridges a React-based CEP panel and ExtendScript to manage audio export and SRT import.

Tokens
27.6K
Snippets
66
Records
167
Agent score
88%

What's inside AutoSubs

  2. Overview of AutoSubs App

    main

    AutoSubs is a cross-platform desktop application designed for generating subtitles using locally running AI transcription models. It supports speaker diarization, translation, and direct integration with video editing software like DaVinci Resolve and Adobe Premiere Pro / After Effects.

    Key Features:

    • Local AI Transcription: Powered by Whisper, Parakeet, or Moonshine models.
    • Speaker Diarization: Uses Pyannote to identify different speakers.
    • Translation: Integrates with Google Translate API.
    • Video Editor Integration: Includes a bundled CEP extension for Adobe products and Lua scripting for DaVinci Resolve.
    • Audio Processing: Uses FFmpeg for normalization and format conversion.

  3. Overview of pyannote-rs features

    main

    pyannote-rs provides audio diarization in Rust with the following capabilities:

    • High Performance: Can compute 1 hour of audio in less than a minute on CPU.
    • Hardware Acceleration: Supports DirectML on Windows and CoreML on macOS for faster performance.
    • Accurate Timestamps: Utilizes Pyannote segmentation for precise timing.
    • Speaker Identification: Uses wespeaker embeddings to identify different speakers.

  4. Understand the AutoSubs Animation System Architecture

    main

    The AutoSubs caption macro implements animations using Lua scripts stored in the macro's CustomData as long-bracket strings ([[ ... ]]), which are executed at runtime via loadstring. The system consists of three core components:

    1. Animations table: Contains named strings for ApplyX and ResetX functions for each animation type. Each function receives a ctx table containing:

      • follower: The StyledTextFollower tool.
      • animStretcher: The AnimationKeyframeStretcher tool.
      • animSpline: The BezierSpline connected to the stretcher.
      • animInEnd: The frame (0–100 range) where the in-animation ends.
      • animOutStart: The frame (0–100 range) where the out-animation starts.
      • mode: 0 = in only, 1 = out only, 2 = both.
      • level: 0 = line, 1 = word.

    2. AnimationRegistry: An ordered list of descriptors used by the orchestrator. Each descriptor includes:

      • controlKey: The UserControl checkbox that enables the animation.
      • usesFade: If true, fade is applied as a base layer when this animation is enabled.
      • applyKey: The key in the Animations table for the application function.
      • resetKey: The key in the Animations table for the reset function.

    3. SetAnimations (Orchestrator): A function that resets all registered animations, checks enabled states, applies fade (or flat opacity), and then applies each enabled animation.

  5. Understand the AutoSubs Adobe Extension Architecture

    main

    The Adobe Extension acts as a bridge between the AutoSubs desktop app and Adobe host applications (Premiere Pro or After Effects). It consists of three layers:

    1. AutoSubs Desktop App: Runs the main Tauri application and hosts a WebSocket server on 127.0.0.1:8185.
    2. CEP Panel: A React-based UI running inside Adobe that connects to the desktop app via WebSocket.
    3. ExtendScript Host Layer: A compiled layer that executes Adobe-specific APIs (like reading sequences or exporting audio) that the React panel cannot access directly.

    Communication flows from the Desktop App $\rightarrow$ WebSocket $\rightarrow$ CEP Panel $\rightarrow$ evalTS() bridge $\rightarrow$ ExtendScript $\rightarrow$ Adobe Host API.

  6. Quickstart with the Transcription Engine

    main

    To perform audio transcription using the transcription_engine, initialize an Engine with EngineConfig::default(), define your TranscribeOptions (including model, language, and VAD settings), and provide Callbacks to handle progress updates and new segments. The transcribe_audio method returns a tuple containing the original segments, the formatted segments, and the detected language.

    use transcription_engine::{
    Callbacks, ContentFormatting, Engine, EngineConfig, ProgressType, Segment, TextCase,
    TranscribeOptions,
};

#[tokio::main]
async fn main() -> eyre::Result<()> {
    whisper_rs::install_logging_hooks();

    let mut engine = Engine::new(EngineConfig::default());

    let options = TranscribeOptions {
        model: "base.en".into(),
        lang: Some("en".into()),
        enable_vad: Some(true),
        ..Default::default()
    };

    fn on_new_segment(seg: &Segment) {
        println!("{}", seg.text);
    }

    fn on_progress(percent: i32, ty: ProgressType, label: &str) {
        println!("{label}: {percent}% ({ty:?})");
    }

    let callbacks = Callbacks {
        progress: Some(&on_progress),
        new_segment_callback: Some(&on_new_segment),
        is_cancelled: None,
    };

    let (segments, formatted_segments, language) = engine
        .transcribe_audio(
            "./audio.wav",
            options,
            Some(2),
            None,
            None,
            Some(ContentFormatting {
                text_case: TextCase::None,
                remove_punctuation: false,
                censored_words: vec![],
            }),
            Some(callbacks),
        )
        .await?;

    println!("segments: {}, cues: {}, language: {}", segments.len(), formatted_segments.len(), language);
    Ok(())
}

  7. Install the gcc-arm-8.3-2019.03 toolchain

    main

    To use the gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf toolchain, download the archive from the official ARM developer website. Extract the archive to a software directory and add the bin directory to your PATH environment variable.

    mkdir /ceph-fj/fangjun/software
cd /ceph-fj/fangjun/software
tar xvf /path/to/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf.tar.xz

export PATH=/ceph-fj/fangjun/software/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin:$PATH

  8. Install kaldi-native-fbank via pip or source

    main

    To use the Kaldi-compatible online fbank feature extractor in Python, you can install kaldi-native-fbank using pip or by building from source.

    Using pip:

    pip install kaldi-native-fbank

    Building from source:

    git clone https://github.com/csukuangfj/kaldi-native-fbank
cd kaldi-native-fbank
python3 setup.py install
    pip install kaldi-native-fbank

  9. Build knf_ using CMake

    main

    To build the knf component, use cmake to configure the build directory with specific flags to disable Python and test builds for Kaldi, then build the Release configuration.

    cmake -B build .  -DKALDI_NATIVE_FBANK_BUILD_PYTHON=OFF -DKALDI_NATIVE_FBANK_BUILD_TESTS=OFF
cmake --build build --config Release
    cmake -B build .  -DKALDI_NATIVE_FBANK_BUILD_PYTHON=OFF -DKALDI_NATIVE_FBANK_BUILD_TESTS=OFF
cmake --build build --config Release

  10. Install the `autosubs` command on your PATH

    main

    To run autosubs from any terminal window, the binary must be in your system's PATH.

    • Linux: Automatically installed to /usr/bin/autosubs via .deb or .rpm packages.
    • macOS / Windows:
      1. Open the AutoSubs desktop application.
      2. Navigate to Settings → Command line.
      3. Click Install. This will symlink the command to /usr/local/bin (macOS) or update your user PATH (Windows).

    Development Setup: If building from source, the binary is located at src-tauri/target/debug/autosubs. You can manually symlink it: ln -s "$(pwd)/src-tauri/target/debug/autosubs" /usr/local/bin/autosubs

  11. Use the DaVinci Resolve Dev Launcher

    main

    Once the environment is set up, follow these steps to run the integration in development mode:

    1. Open DaVinci Resolve.
    2. Navigate to Workspace → Scripts → AutoSubs (Dev).
    3. The Lua server will start (note: no app window will appear).
    4. To apply changes made to files in src-tauri/resources/modules/, re-run the script from the Resolve menu.
    5. If you move your repository to a different location, you must re-run npm run setup-resolve to update the absolute paths.