HPSv3: Human Preference Score v3

Use the evaluate/evaluate.py script to assess model performance on a test set. The script supports two evaluation modes:

1. pair mode: Calculates pairwise accuracy. The input JSON must provide a path1 (assumed better) and a path2 for each prompt.
2. ranking mode: Calculates ranking accuracy. The input JSON must provide a list of image paths in generations and their corresponding ground-truth order in ranking.

Input Data Formats

Pair-wise JSON format (--mode pair):

[
    {
        "prompt": ".....",
        "path1": ".....",
        "path2": "....."
    }
]

Rank-wise JSON format (--mode ranking):

[
    {
        "id": "005658-0040",
        "prompt": ".....",
        "generations": [
            "path/to/image1",
            "path/to/image2",
            "path/to/image3",
            "path/to/image4"
        ],
        "ranking": [1, 2, 5, 3]
    }
]

python evaluate/evaluate.py \
  --test_json /path/to/your/test_data.json \
  --config_path config/HPSv3_7B.yaml \
  --checkpoint_path checkpoints/HPSv3_7B/model.pth \
  --mode pair \
  --batch_size 8 \
  --num_processes 8

To add a new text-to-image model, you must define a new PipelineParam instance in utils/pipelines.py.

1. Open utils/pipelines.py.
2. Import PipelineParam.
3. Create a new instance of PipelineParam with your model's configuration.
4. Use the variable name of your new instance in the --pipeline_name CLI argument.

Available PipelineParam fields:

• pipeline_name: Model path (Hugging Face Hub or local directory).
• generation_path: Subdirectory name for output images.
• pipeline_type: Type of pipeline (e.g., 't2i' for text-to-image, 't2v' for text-to-video). Defaults to 't2i'.
• pipe_init_kwargs: Dictionary of arguments for initializing the pipeline (e.g., {"torch_dtype": torch.float16}).
• generation_kwargs: Dictionary of arguments for the generation process (e.g., {"guidance_scale": 7.0}).
• base_resolution: The base resolution the model was trained on.
• force_aspect_ratio: (Optional) Force a specific aspect ratio (e.g., 1 for square).

from pydantic import BaseModel, Field
import torch

# ... (PipelineParam class definition)

# Add your new model configuration
my_new_model_pipe = PipelineParam(
        pipeline_name='organization/my-cool-model',
        generation_path=f'generation/my_cool_model',
        pipe_init_kwargs={
            "torch_dtype": torch.float16,
        },
        base_resolution=1024,
        generation_kwargs={
            "guidance_scale": 5.0,
            "num_inference_steps": 30,
        }
    )

CoHP is a reasoning approach for iterative image refinement. It generates images using multiple diffusion models, selects the best one using a reward model, and performs iterative image-to-image generation to improve quality.

Usage

Run the run_cohp.py script with the required --prompt and --index arguments.

CLI Arguments

• --prompt (required): The text prompt for image generation.
• --index (required): A unique identifier for saving the results.
• --device: The GPU device to use (default: 'cuda:1').
• --reward_model: The reward model used for scoring. Supported values:
  • hpsv3 (default, recommended)
  • hpsv2
  • imagereward
  • pickscore

Process Details

1. Multi-Model Generation: Uses FLUX.1 dev, Kolors, Stable Diffusion 3 Medium, and Playground v2.5.
2. Reward Scoring: Evaluates images via the chosen reward model.
3. Selection: Picks the model with the highest average score.
4. Refinement: Performs 4 rounds of image-to-image generation. Strength is set to 0.8 for rounds 1-2 and 0.5 for rounds 3-4.

python hpsv3/cohp/run_cohp.py \
    --prompt "A beautiful sunset over mountains" \
    --index "sample_001" \
    --device "cuda:0" \
    --reward_model "hpsv3"

When training HPSv3, you can adjust several key parameters in your configuration file.

Model Configuration

• rm_head_type: The architecture of the reward model head (e.g., "ranknet").
• lora_enable: Boolean. If False, the language tower is fully trainable. If True, enables LoRA for efficient fine-tuning.
• vision_lora: Boolean. If False, the vision tower is fully trainable. If True, applies LoRA specifically to vision components.
• model_name_or_path: The path to the base model checkpoint (e.g., "Qwen/Qwen2-VL-7B-Instruct").

Data Configuration

• confidence_threshold: Minimum confidence score required for training data (e.g., 0.95).
• train_json_list: A list of training data files (e.g., ["example_train.json"]).
• test_json_list: A list of validation datasets.
• output_dim: The output dimension of the reward head for $\mu$ and $\sigma$ (e.g., 2).
• loss_type: The loss function type used for training (e.g., "uncertainty").