DOCSEARCHbyAlgolia
Submit Docs

Babit Multimedia Framework (BMF)

repository·master·Indexed 21 days ago

https://github.com/babitmf/bmf

A high-performance, multi-language video processing framework designed for large-scale production environments. BMF features strong GPU acceleration, seamless AI framework integration, and a C++ graph runtime. It includes bmf_lite for Android, iOS, and OpenHarmony, as well as integrations with ComfyUI using zero-copy tensors via DLPack. The framework supports diverse applications including RTMP broadcasting, aesthetic assessment, audio robot effects, and DeOldify colorization pipelines.

Tokens
36.2K
Snippets
105
Records
146
Agent score
74%

What's inside BMF

  1. Overview of Maxine Module Video Effects

    master

    The Maxine Module leverages NVIDIA Maxine GPU-accelerated AI SDKs to provide real-time video enhancement effects. Currently, the module supports the following video effects:

    • Virtual Background
    • Super Resolution (up to 4X scaling factor)
    • Upscaler (up to 4X scaling factor)
    • Artifact Reduction
    • Video Noise Removal

    Note: Audio and augmented reality effects are planned for future releases. For detailed programming specifications, refer to the NVIDIA VFX SDK programming guide.

  2. Overview of BMF (Babit Multimedia Framework)

    master

    BMF is a cross-platform, multi-language, and customizable multimedia processing framework designed for high-performance video processing. It is optimized for real-world production environments like video streaming, live transcoding, and cloud editing.

    Key capabilities include:

    • Multi-language support: Use Python, Go, or C++ APIs.
    • Cross-platform: Native support for Linux, Windows, and macOS on both x86 and ARM architectures.
    • High Performance: Features a powerful scheduler and heterogeneous hardware acceleration (CPU/GPU), with highly optimized GPU pipelines developed in cooperation with NVIDIA.
    • Extensibility: A decoupled architecture allows developers to add custom modules independently.
    • Data Conversion: Seamlessly converts data between FFmpeg, Numpy, PyTorch, OpenCV, and TensorRT, as well as between CPU and GPU devices.

  3. Overview of BMF ↔ ComfyUI Integration

    master

    This integration replaces the standard ComfyUI Python prompt executor with the BMF high-performance C++ graph runtime. It allows you to run existing ComfyUI workflows using BMF's engine without modifying the ComfyUI source code.

    How it works:

    • Runtime Hook: A hook is installed into execution.PromptExecutor.execute via run_bmf_comfy.py at startup.
    • Workflow Conversion: ComfyUI JSON workflows are converted into a BMF GraphConfig in topological order.
    • Node Mapping: Each ComfyUI node is mapped to a BMF Python module called ComfyNodeRunner, which instantiates the node and calls its execute/FUNCTION method.
    • Zero-Copy Tensors: Uses hmp/DLPack to bridge tensors between BMF and PyTorch (for IMAGE, LATENT, and MASK payloads) without expensive memory copies.
    • UI Integration: Maintains native WebUI features like progress bars, live previews, and history_result population.

  4. Understand BMF Graph Modes

    master

    BMF supports multiple execution modes for processing data within a graph:

    • Generator Mode: For generating data streams.
    • Sync Mode: Synchronous execution of modules.
    • Server Mode: Running BMF as a service.
    • Preload Mode: Preloading data for processing.
    • Subgraph Mode: Using subgraphs within a larger graph.
    • PushData Mode: Pushing data into the pipeline.

  5. How the video editing pipeline works in BMF

    master

    The editing demo utilizes a BMF graph topology that combines several modules to create a complex pipeline. The workflow follows these steps:

    1. Stream Creation: Three separate video streams are created.
    2. Overlaying: The video_overlay module is used to overlay a logo (e.g., Xigua Video) onto each of the three video streams.
    3. Concatenation: The three processed video streams, along with their corresponding audio streams, are passed to the video_concat module.
    4. Final Output: The video_concat module merges these channels to complete the editing process.

    This pattern demonstrates how to use BMF subgraphs to compose multiple specialized modules into a single complex processing graph.

  6. Broadcaster Demo Graph Architecture

    master

    The Broadcaster Demo is composed of several specialized BMF modules working in a pipeline:

    • wallclock module: Generates uniform audio/video timestamps (e.g., 25 fps, 44100Hz dual-channel LC-AAC). It acts as the stable clock driving the entire graph and provides background timestamps to the streamhub.
    • c_ffmpeg_decoder module: Acts as the source for media assets. In this demo, it supports RTMP input streams.
    • streamhub module: Manages jitter buffers for multiple input streams to ensure uniform output. It aligns frames from all input streams based on the background timestamps from the wallclock to prepare them for mixing.
    • audiomix module: Performs audio mixing. It accepts a framelist from streamhub and supports 1024-sample, dual-channel, 44100Hz audio input.
    • videolayout module: Handles video compositing (mixing multiple video streams into one). It uses parameters to control the display coordinates of multiple streams.
    • c_ffmpeg_encoder module: The output module that receives data from videolayout and audiomix to encode and output the final RTMP stream.

  7. How BMF Audio Modules process data

    master

    A BMF Audio Module follows a specific workflow to transform audio signals. The fundamental steps are:

    1. Data Validation: Verifies that input audio frames match required specifications for sample rate, audio layout, and data type.
    2. Data Caching: Buffers audio frames to satisfy the algorithm's requirements for efficient processing.
    3. Data Format Conversion: Converts AudioFrame tensor lists into ndarray format for compatibility with processing libraries.
    4. Algorithm Processing: Executes the core logic (e.g., using librosa for robot effects).
    5. Frame Splitting and Output: Processes the resulting audio buffer and splits it back into frames for output.

  8. How to implement a custom LLM for captioning

    master

    You can extend the LLM captioning capabilities by implementing your own model. The implementation path depends on the architecture type:

    1. Hugging Face Architectures: Extend BaseVisionModel (found in models/base_model.py) and implement the specific wrapper class required for the Hugging Face architecture.
    2. vLLM Architectures: Extend BaseVLLMVisionModel (found in models/vllm/base_vllm_model.py). Since vLLM maintains consistent chat logic, separate models simply extend this single base class.

    Once implemented, the model_factory.py will load your model when the corresponding name is passed during instantiation.

  9. How the Broadcaster Demo Pipeline Works

    master

    The broadcaster demo is composed of several BMF modules working in a graph to process and mix streams:

    • wall clock module: Generates audio and video timestamps (e.g., 25 fps, 44100 dual-channel lc-aac) to serve as a stable background source that drives the entire graph.
    • c_ffmpeg_decoder module: The input source. It decodes input streams (supporting RTMP) into audio and video.
    • streamhub module: Acts as a jitter buffer for each input stream to ensure smooth output. It aligns timestamps and frame rates from multiple streams based on the wall clock timestamps, binding them together for mixing.
    • audiomix module: Mixes the audio frames provided by streamhub. It requires audio input with 1024 samples, dual channel, and a 44100 sampling rate.
    • videolayout module: Blends the video frames from streamhub and allows configuration of display coordinates for multiple streams.
    • c_ffmpeg_encoder module: The output module. It receives processed audio from audiomix and video from videolayout, encodes them, and outputs the final RTMP stream.

  10. Understand BMFLite for client-side processing

    master

    BMFLite is a lightweight, efficient, cross-platform version of BMF designed specifically for client-side multimedia processing. It is used in high-scale applications (like Douyin/Xigua) for live streaming, video playback, and cloud gaming.

    Key use cases for BMFLite include:

    • Edge AI models: Integrating external algorithm modules into the BMFLite framework.
    • Real-time effects: Implementing pipelines that combine video capture, noise reduction (denoising), and rendering.

  11. Build and run bmf_lite for OpenHarmony (OHOS)

    master

    To use bmf_lite on OpenHarmony, follow these steps to build and run the sample project.

    Prerequisites

    • Huawei Developer Account: Registered on the Huawei developer platform.
    • DevEco-Studio: Installed and configured.
    • OpenHarmony SDK: Installed via DevEco-Studio.
    • CMake: Version 3.15 or newer.
    • Assets: Download bmf_lite_files.tar.gz and place test.mp4 in ohos/entry/src/main/resources/rawfile.
    • Device: A HarmonyOS mobile device or an emulator.

    Build and Run

    1. Open the bmf_lite/ohos project in DevEco-Studio.
    2. Build Library: Select the bmf_lite item (with an H icon) from the Run configurations and click the green run button.
    3. Run Sample:
      • Select the entry item (with a yellow square) from the Run configurations.
      • Click the green run button.
      • When prompted, click Open signing configs in the Run terminal and log in with your Huawei account.
      • Click the green run button again to execute.

  12. Explore BMF Quick Experience examples

    master

    BMF provides several high-level use cases and quick-start scenarios for video processing. You can explore implementations for:

    • Transcode: Converting video formats.
    • Edit: Performing video editing tasks.
    • Meeting/Broadcaster: Specialized workflows for real-time communication.
    • GPU acceleration:
      • GPU Video Frame Extraction
      • GPU Video Transcoding and Filtering
    • AI inference:
      • Deoldify (colorization)
      • Super Resolution
      • Video Quality Score
      • Face Detect With TensorRT