DOCSEARCHbyAlgolia
Submit Docs

PaddleMIX Documentation

repository·develop·Indexed 20 days ago

https://github.com/paddlepaddle/paddlemix

A comprehensive multimodal large model development suite built on PaddlePaddle. PaddleMIX supports multimodal understanding, text-to-image/video generation, and fine-tuning, covering the full lifecycle from data processing to distributed deployment. It includes applications such as Audio2Caption, Audio2Img, AudioChat, Grounded-SAM, and image-to-text generation via miniGPT4.

Tokens
208.5K
Snippets
531
Records
700
Agent score
72%

What's inside PaddleMIX

  1. Overview of EVA-02 Model Variants

    develop

    EVA-02 is a visual representation model implementation in PaddlePaddle. The available models are categorized into four training stages:

    1. MIM pre-trained EVA-02: Models trained using Masked Image Modeling (MIM). Input scale is 224x224 with a 14x14 patch size. Models with psz14to16 in their name have interpolated patch_embed kernels to 16x16 and pos_embed to 14x14, which is optimized for downstream tasks like object detection and segmentation.
    2. IN-21K intermediate fine-tuned EVA-02: Models that underwent intermediate fine-tuning on the ImageNet-21K dataset. Input scale is 448x448 with a 14x14 patch size.
    3. IN-1K fine-tuned EVA-02 (without IN-21K): Models fine-tuned directly on ImageNet-1K. Input sizes vary (e.g., 336x336 or 448x448).
    4. IN-1K fine-tuned EVA-02 (with IN-21K): Models that used IN-21K as an intermediate step before IN-1K fine-tuning. Input scale is 448x448.

  2. Overview of PPDiffusers capabilities

    develop

    PPDiffusers is a toolbox for training and inference of Diffusion Models across multiple modalities (text, image, video, audio) implemented on the PaddlePaddle framework and PaddleNLP library.

    Key Features

    • SOTA Pipelines: Over 100 integrated pipelines supporting tasks like Text-to-Image, Text-Guided Image Inpainting, Image-to-Image, Text-to-Video, Super-resolution, and Text-to-Audio.
    • Noise Schedulers: 14+ integrated schedulers (e.g., DDPM, DDIM, PNDM, DPMSolver) to balance generation speed and quality.
    • Diffusion Components: A wide range of model components including UNet1DModel, UNet2DModel, UNet2DConditionModel, UNet3DConditionModel, VQModel, and AutoencoderKL.
    • Training & Inference Tutorials: Support for fine-tuning via Textual Inversion and DreamBooth, as well as training for Latent Diffusion Model, ControlNet, and T2I-Adapter.
    • High-Performance Deployment: Integration with FastDeploy for high-performance Stable Diffusion pipelines.

  3. Introduction to DeepSeek-VL2

    develop

    DeepSeek-VL2 is a Mixture-of-Experts (MoE) vision-language model designed for tasks including visual question answering, OCR, document/table/chart understanding, and visual grounding.

    Supported model variants include:

    • deepseek-ai/deepseek-vl2-tiny (1B activated parameters)
    • deepseek-ai/deepseek-vl2-small (2.8B activated parameters)
    • deepseek-ai/deepseek-vl2 (4.5B activated parameters)

    Note: While these share names with HuggingFace weights, they are Paddle-compatible tensors. You can automatically download them using xxx.from_pretrained("deepseek-ai/deepseek-vl2-tiny").

  4. Introduction to PP-DocBee2

    develop

    PP-DocBee2 is a multimodal large model developed by the PaddleMIX team specifically for document understanding. It is optimized for Chinese document tasks using a proprietary data synthesis strategy.

    Available Model:

    • PPDocBee2-3B: 3B parameter model.

    You can automatically download the weights by using xxx.from_pretrained("PaddleMIX/PPDocBee2-3B").

  5. Overview of PP-DocBee

    develop

    PP-DocBee is a multimodal large model developed by the PaddleMIX team, specifically optimized for document understanding tasks in Chinese.

    Key characteristics:

    • Architecture: Based on the Qwen2-VL-2B architecture.
    • Training: Fine-tuned on nearly 5 million multimodal document understanding datasets, including General VQA, OCR, charts, text-rich documents, mathematics, complex reasoning, synthetic data, and pure text data.
    • Performance: Achieves SOTA (State-of-the-Art) performance on authoritative English document understanding benchmarks for its parameter scale and outperforms popular open-source and closed-source models in internal Chinese business scenarios.

  6. Overview of Latent Diffusion Model Training

    develop

    This tutorial provides code for training a 32-layer Latent Diffusion Model from scratch. The implementation supports switching between 中文 (Chinese) and 英文 (English) tokenizers.

    Note on VAE/VQVAE naming: While the official 32-layer CompVis/ldm-text2im-large-256 model uses a VAE, this implementation uses the folder name vqvae to maintain consistency with the Huggingface directory structure.

  7. Overview of Latent Consistency Models (LCM)

    develop

    Latent Consistency Models (LCMs) are a next-generation generative model architecture designed for high-resolution image synthesis with few-step inference. Based on the research from Tsinghua University, LCMs can increase image generation speed by 2-5x compared to standard Latent Diffusion Models (LDMs) while requiring significantly less computational power.

    Supported Models in Model Zoo

    model nameparamsweight
    latent-consistency/lcm-sdxlTODOTODO
    latent-consistency/lcm-lora-sdv1-5TODOTODO
    latent-consistency/lcm-lora-sdxlTODOTODO

    Note: More LCM models will be supported in future updates. Pre-trained weights will be provided via AI Studio.

  8. Overview of PaddleMIX

    develop

    PaddleMIX is a multimodal large model development suite based on PaddlePaddle. It aggregates multiple modalities including image, text, and video, covering a wide range of tasks such as:

    • Visual-Language Pre-training
    • Fine-tuning
    • Text-to-Image Generation
    • Text-to-Video Generation
    • Multimodal Understanding

    It provides an out-of-the-box development experience with support for flexible customization, aiming to facilitate the exploration of Artificial General Intelligence (AGI).

  9. Overview of Audio-to-Image Generation

    develop

    The Audio2Img application enables generating images from audio using ImageBind's unified latent space combined with stable-diffusion-2-1-unclip. This approach allows for cross-modal generation without requiring additional training. It is integrated with ppdiffusers and supports three primary modes of operation:

    1. Audio to Image: Generate an image based solely on an audio input.
    2. Audio + Text to Image: Use both an audio file and a text prompt to guide the image generation.
    3. Audio + Image to Image: Use an audio file and an existing image as inputs to generate a new image.

  10. Overview of PPDiffusers Pipelines

    develop

    PPDiffusers Pipelines provide a simplified, end-to-end interface for performing inference on various State-of-the-Art (SOTA) diffusion models. Instead of manually managing individual components like Autoencoders, Unets, Text Encoders, and Schedulers, the Pipeline encapsulates the entire system into a single API.

    Key capabilities include:

    • Weight Loading: Loading official weights to reproduce outputs identical to original implementations.
    • Unified API: A consistent interface for running inference across different diffusion model systems.
    • Multi-modal Support: Support for 10+ tasks across multiple modalities.
    • Code Transparency: Implementation designed to be easily readable alongside official documentation.

    Note: Pipelines are designed for inference only. They do not provide training functionality. For training examples, refer to the examples directory in the repository.

  11. Overview of PaddleMIX multimodal capabilities

    develop

    PaddleMIX is a multimodal large model development suite built on PaddlePaddle. It provides a complete toolchain for integrating various modalities like images, text, and video.

    Key capabilities include:

    • Multimodal Understanding: Includes basic perception, fine-grained image understanding, and complex visual reasoning. Supported tasks include natural image summarization, question answering (QA), OCR, sentiment recognition, specialized image analysis, and code interpretation for single images, multiple images, or video.
    • Multimodal Generation: Covers text-to-image and text-to-video technologies. It supports single-image generation, multi-image synthesis, and video generation (e.g., using Stable Diffusion 3 or Open-Sora).
    • Task Coverage: Supports vision-language pre-training, fine-tuning, and inference deployment for mainstream models like EVA-CLIP, BLIP-2, and Stable Diffusion.

  12. Overview of PaddleMIX Data Processing and Analysis Enhancements

    develop

    This RFC (Request for Comments) outlines the enhancement of PaddleMIX's data processing and analysis capabilities, specifically targeting multimodal datasets like LLaVA v1.5 mix665k. The goal is to provide fine-grained analysis and efficient processing for image-text pairs to improve model training quality, convergence speed, and generalization.

    Key focus areas include:

    • Data Quality: Addressing blurry images, low resolution, and poor text grammar.
    • Multimodal Alignment: Measuring and filtering low image-text matching scores.
    • Data Distribution: Analyzing and balancing sample categories and text lengths.