DOCSEARCHbyAlgolia
Submit Docs

FunClip Documentation

repository·main·Indexed 27 days ago

https://github.com/modelscope/funclip

An open-source, locally deployed automated video clipping tool that uses Alibaba's FunASR models, such as Paraformer and SenseVoice, to clip video segments based on text transcripts or speakers. It features a Gradio UI and CLI for speech recognition and clipping, supports AI-assisted smart clipping via LLMs (e.g., Qwen, GPT), and integrates with TwelveLabs Pegasus for content-aware visual and audio reasoning.

Tokens
2.5K
Snippets
4
Records
11
Agent score
41%

What's inside FunClip

  1. Perform AI-assisted smart clipping with LLMs

    main

    FunClip supports smart clipping using Large Language Models (e.g., Qwen series, GPT series). Follow these steps in the Gradio UI:

    1. Recognition: Run the ASR process first.
    2. Configure LLM: Select the desired large model name and configure your apikey.
    3. Inference: Click the 'LLM Inference' button. FunClip will combine the video's SRT subtitles with your configured prompts.
    4. Clip: Click the 'AI Clip' button. FunClip will extract timestamps based on the LLM's output to perform the clipping.

  2. Install ImageMagick for subtitle clipping

    main

    If you need to clip video files with embedded subtitles, you must install ffmpeg and imagemagick and update the ImageMagick policy to allow reading/writing.

    Ubuntu

    apt-get -y update && apt-get -y install ffmpeg imagemagick
sed -i 's/none/read,write/g' /etc/ImageMagick-6/policy.xml

    MacOS

    brew install imagemagick
sed -i '' 's/none/read,write/g' "$(brew --prefix imagemagick)/etc/ImageMagick-7/policy.xml"

    Windows

    1. Download and install ImageMagick from here.
    2. Locate your Python installation path.
    3. Update the IMAGEMAGICK_BINARY variable to your ImageMagick installation path in the file: site-packages\moviepy\config_defaults.py.

    Font Setup

    After installing ImageMagick, download the required font file to the funclip/font directory:

    wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ClipVideo/STHeitiMedium.ttc -O font/STHeitiMedium.ttc
    # Ubuntu example
apt-get -y update && apt-get -y install ffmpeg imagemagick
sed -i 's/none/read,write/g' /etc/ImageMagick-6/policy.xml

  3. Enable Content-aware clipping with TwelveLabs Pegasus

    main

    For video understanding that reasons over actual visuals and audio (rather than just ASR transcripts), you can use TwelveLabs Pegasus. This is useful for picking highlights based on action or scene changes.

    Setup:

    1. Install the dependency: pip install twelvelabs.
    2. Obtain a free API key from twelvelabs.io.

    Usage in FunClip UI:

    1. Select the pegasus1.5 model name in the interface.
    2. Paste your TwelveLabs API key.
    3. Upload your video.
    4. Click 'LLM Inference'.

    Pegasus returns segments in the N. [start-end] text format, which is compatible with the existing 'AI Clip' functionality.

  4. Install ImageMagick for subtitle generation

    main

    If you want to use the feature that automatically generates subtitles in the clipped videos, you must install ImageMagick and configure its policy.

    Ubuntu:

    apt-get -y update && apt-get -y install ffmpeg imagemagick
sed -i 's/none/read,write/g' /etc/ImageMagick-6/policy.xml

    MacOS:

    brew install imagemagick
sed -i '' 's/none/read,write/g' "$(brew --prefix imagemagick)/etc/ImageMagick-7/policy.xml"

    Windows:

    1. Download and install from ImageMagick website.
    2. Locate your Python installation and modify site-packages\moviepy\config_defaults.py to set IMAGEMAGICK_BINARY to the path of your magick.exe.

    Download default font:

    wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ClipVideo/STHeitiMedium.ttc -O font/STHeitiMedium.ttc
    # Ubuntu
apt-get -y update && apt-get -y install ffmpeg imagemagick
sed -i 's/none/read,write/g' /etc/ImageMagick-6/policy.xml

# MacOS
brew install imagemagick
sed -i '' 's/none/read,write/g' "$(brew --prefix imagemagick)/etc/ImageMagick-7/policy.xml"

  5. Launch FunClip as a local Gradio Service

    main

    You can run FunClip as a local web service using Gradio. By default, it launches a service at localhost:7860. Use the funclip/launch.py script with various flags to select models and configurations:

    • -m <model_name>: Select the ASR model.
      • fun-asr-nano: Flagship model for Mandarin, English, Japanese, and various dialects/accents.
      • sensevoice: Multilingual ASR with emotion and audio event detection.
    • -l <lang>: Set the language for audio recognition (e.g., -l en for English).
    • -p <port>: Set the port number.
    • -s True: Enable public access for the service.

  6. Use LLM for Intelligent Video Clipping

    main

    FunClip supports intelligent video clipping using Large Language Models (e.g., Qwen, GPT series).

    Workflow:

    1. Perform ASR (speech recognition) on the video.
    2. In the UI, select an LLM name and configure your apikey.
    3. Click the 'LLM智能段落选择' (LLM Intelligent Paragraph Selection) button. FunClip will combine prompts with the video's SRT subtitles.
    4. Click the 'LLM智能裁剪' (LLM Intelligent Clipping) button. FunClip will extract timestamps from the LLM output to perform the clip.

  7. Install FunClip

    main

    To install FunClip, clone the repository and install the required Python dependencies. Note that model weights are downloaded automatically upon startup.

    Prerequisites:

    • Python and pip installed.
    • For Fun-ASR-Nano, SenseVoice, and subtitle compatibility, ensure funasr>=1.3.29 is installed.
    # Clone the repository
git clone https://github.com/modelscope/FunClip.git
cd FunClip

# Install dependencies
pip install -r ./requirements.txt

# Ensure funasr is up to date for specific models
pip install -U "funasr>=1.3.29"
    # 克隆funclip仓库
git clone https://github.com/modelscope/FunClip.git
cd FunClip
# 安装相关Python依赖
pip install -r ./requirements.txt

  8. Use FunClip via Command Line (videoclipper.py)

    main

    For automated workflows without a GUI, use funclip/videoclipper.py in two stages: recognition and clipping.

    Stage 1: Recognition Generates ASR results and SRT subtitles in the specified output directory.

    python funclip/videoclipper.py --stage 1 --file <video_path> --output_dir <output_dir>

    Stage 2: Clipping Extracts a specific segment based on text.

    python funclip/videoclipper.py --stage 2 \
                       --file <video_path> \
                       --output_dir <output_dir> \
                       --dest_text '<target_text>' \
                       --start_ost <start_offset> \
                       --end_ost <end_offset> \
                       --output_file '<output_video_path>'

    Arguments:

    • --stage 1: Recognition stage.
    • --stage 2: Clipping stage.
    • --file: Path to the input video.
    • --output_dir: Directory to save results.
    • --dest_text: The target text segment to clip.
    • --start_ost: Start offset.
    • --end_ost: End offset.
    • --output_file: Path for the resulting clipped video.

  9. Use FunClip via Command Line Interface

    main

    FunClip supports a two-stage CLI workflow for recognizing and clipping video files using funclip/videoclipper.py.

    Stage 1: Recognize Performs speech recognition on the video file and generates recognition results and an SRT file in the specified output directory.

    Stage 2: Clip Extracts specific segments based on text input.

    Required arguments for Stage 2:

    • --stage 2: Specifies the clipping stage.
    • --file: Path to the input video file.
    • --output_dir: Directory where results are stored.
    • --dest_text: The specific text segment you want to clip.
    • --start_ost: Start offset (timestamp).
    • --end_ost: End offset (timestamp).
    • --output_file: Path to the resulting clipped video file.
    # Step 1: Recognize
python funclip/videoclipper.py --stage 1 \
                       --file examples/2022云栖大会_片段.mp4 \
                       --output_dir ./output

# Step 2: Clip
python funclip/videoclipper.py --stage 2 \
                       --file examples/2022云栖大会_片段.mp4 \
                       --output_dir ./output \
                       --dest_text '我们把它跟乡村振兴去结合起来，利用我们的设计的能力' \
                       --start_ost 0 \
                       --end_ost 100 \
                       --output_file './output/res.mp4'

  10. Launch FunClip Gradio Service

    main

    You can start a local web interface using Gradio to interact with FunClip. Use the following command-line flags to select specific models or languages.

    Common Launch Commands:

    • Default (Paraformer): python funclip/launch.py (Best for precise text-based clipping in Chinese).
    • Fun-ASR-Nano: python funclip/launch.py -m fun-asr-nano (Supports Mandarin, English, Japanese, and dialects; high precision transcription but lacks reliable character-level timestamps).
    • SenseVoice: python funclip/launch.py -m sensevoice (Multilingual ASR + emotion recognition + audio event detection).
    • English Video Clipping: python funclip/launch.py -l en (Uses Paraformer English model).

    CLI Flags:

    • -m <model_name>: Select model (fun-asr-nano or sensevoice).
    • -l en: Start service for English audio recognition.
    • -p <port>: Set the port number.
    • -s True: Establish service for public access.

  11. Use different ASR models in FunClip

    main

    You can launch FunClip with different speech recognition models using the -m flag.

    • Fun-ASR-Nano: Use -m fun-asr-nano to load the flagship Fun-ASR-Nano-2512 checkpoint. This supports Mandarin, English, Japanese, and various Chinese dialects/accents. Note: For precise text-based clipping, use Paraformer as the Nano checkpoint may not provide reliable character-level timestamps.
    • SenseVoice: Use -m sensevoice to add emotion recognition and audio event detection.

    To support English audio files, use the -l flag:

    • English Support: python funclip/launch.py -l en