Align-Anything Documentation

JADE 3.0 is a high-quality Chinese dataset designed for the safety fine-tuning of Large Language Models (LLMs). It consists of triplets containing a "High-risk Question", a "Violating Response", and a "Safe and Useful Response". These triplets are generated based on high-risk scenarios identified by the JADE 1.0 testing platform, using LLM self-reflection and correction to ensure high quality.

Note: The dataset contains examples of harmful and violating content for training purposes and does not represent the stance of the development team.

The align-anything evaluation module supports multimodal benchmarks across various modalities:

• Text→Text
• Text+Image/Video/Audio→Text
• Text→Image/Video/Audio

Generation Backends

For most modalities, you can choose between the following backends based on your hardware and environment:

• vLLM: Supported for most text and multimodal tasks.
• DeepSpeed: Supported for specific multimodal tasks (e.g., A-OKVQA, MathVista, MMBench).
• Accelerate: Used for diffusion-based generation tasks (Text→Image/Audio/Video).

Local File Loading

For Text→Image/Audio/Video tasks, where diffusion models often rely on specific frameworks, the module provides an option to load local files directly for evaluation.

Align-Anything is a highly modular framework designed to align any-to-any large models with human intentions and values. Key capabilities include:

• Modular Framework: Easily customize code for different tasks.
• Multi-Modal Fine-Tuning: Supports diverse modalities including image, video, and audio.
• Alignment Methods: Supports SFT (Supervised Fine-Tuning), DPO (Direct Preference Optimization), PPO (Proximal Policy Optimization), and GRPO (Group Relative Policy Optimization).
• Multi-Modal CLI: Command-line interface for image, audio, and video modalities.
• O1-like Training: Supports reasoning-based training (e.g., using DollyTails datasets).
• Rule-based RL: Reinforcement Learning encouraged by Deepseek-R1 patterns.
• Hardware Support: Optimized for Nvidia GPUs and Huawei Ascend NPUs.

JADE (Linguistics-based Safety Evaluation Platform) provides targeted safety evaluation datasets for Large Language Models (LLMs). The platform uses linguistic mutation to transform low-trigger seed questions into high-risk questions, covering four major categories: Core Values, Illegal Activities/Crimes, Rights Infringement, and Discrimination/Bias.

Note: The datasets contain examples of harmful and violating content. Due to regulations, politically sensitive test questions are not included in the public downloadable sets; users requiring such evaluations should contact mi_zhang@fudan.edu.cn.

The Chameleon Plus Fine-Tuning Pipeline describes the process of enhancing the Chameleon-7B model for improved text-image interleaved input and output tasks.

Key stages include:

1. Base Model Training: Training the original Chameleon-7B model using a 4.6k subset from the laion-art dataset to create AA-Chameleon-7B-Base, which possesses image generation capabilities.
2. Plus Model Training: Using the Align-Anything framework and specialized text-image interleaved datasets to train AA-Chameleon-7B-Plus, which significantly improves performance on interleaved I/O tasks.

Eval-anything is a framework designed to track the performance and safety of any-to-any modality large models.

Key features include:

• Safety Assessment: Uses self-developed datasets and over 50 integrated open-source datasets across five core dimensions (35 sub-dimensions).
• Embodied Safety Evaluation: Evaluates text, image, video, speech, and action modalities, covering categories like corner cases, blind spots, and dangerous equipment. It focuses on execution safety, long-range trajectory safety, and hardware safety.
• Platform Integration: Integrates with FlagEval for enhanced assessment.

Align-Anything is an open-source infrastructure designed to align any-to-any modality models (including LLMs and VLMs) with human values. It provides a highly modular framework for training and evaluation across diverse modalities such as image, video, and audio.

Key capabilities include:

• Modular Training: Supports various alignment algorithms including SFT, RM, DPO, PPO, KTO, SimPO, and ORPO.
• Multi-Modality Support: Fine-tuning scripts for diverse models (image/video/audio).
• Scalable Model Registration: Supports training and deploying over 25 different models.
• Rule-based RL: Implementation of rule-based reinforcement learning inspired by Deepseek-R1.