InstantCharacter

Use the InstantCharacterFluxPipeline to generate images that preserve a character from a reference image.

Workflow:

1. Initialize Pipeline: Load the InstantCharacterFluxPipeline with a base model (e.g., black-forest-labs/FLUX.1-dev).
2. Initialize Adapter: Call pipe.init_adapter() with paths to the IP-Adapter, and two image encoders (google/siglip-so400m-patch14-384 and facebook/dinov2-giant).
3. Load Reference: Open a reference image (ideally with a white background).
4. Run Inference: Call the pipeline with your prompt, subject_image, and subject_scale.

import torch
from PIL import Image
from pipeline import InstantCharacterFluxPipeline

# Step 1 Load base model and adapter
ip_adapter_path = 'checkpoints/instantcharacter_ip-adapter.bin'
base_model = 'black-forest-labs/FLUX.1-dev'
image_encoder_path = 'google/siglip-so400m-patch14-384'
image_encoder_2_path = 'facebook/dinov2-giant'
seed = 123456
pipe = InstantCharacterFluxPipeline.from_pretrained(base_model, torch_dtype=torch.bfloat16)
pipe.to("cuda")
pipe.init_adapter(
    image_encoder_path=image_encoder_path, 
    image_encoder_2_path=image_encoder_2_path, 
    subject_ipadapter_cfg=dict(subject_ip_adapter_path=ip_adapter_path, nb_token=1024), 
)

# Step 2 Load reference image
ref_image_path = 'assets/girl.jpg'  # white background
ref_image = Image.open(ref_image_path).convert('RGB')

# Step 3 Inference without style
prompt = "A girl is playing a guitar in street"
image = pipe(
    prompt=prompt, 
    num_inference_steps=28,
    guidance_scale=3.5,
    subject_image=ref_image,
    subject_scale=0.9,
    generator=torch.manual_seed(seed),
).images[0]
image.save("flux_instantcharacter.png")

You can apply specific artistic styles to your character generation by using the with_style_lora method. This requires downloading a style LoRA (e.g., Ghibli or Makoto Shinkai) and providing a trigger word.

Steps:

1. Download the desired LoRA to your checkpoints/style_lora/ directory.
2. Use pipe.with_style_lora(...) instead of calling the pipeline directly. This method accepts the lora_file_path, a trigger string, and the standard inference parameters.

# Step 3 Inference with style
lora_file_path = 'checkpoints/style_lora/ghibli_style.safetensors'
trigger = 'ghibli style'
prompt = "A girl is playing a guitar in street"
image = pipe.with_style_lora(
    lora_file_path=lora_file_path,
    trigger=trigger,
    prompt=prompt, 
    num_inference_steps=28,
    guidance_scale=3.5,
    subject_image=ref_image,
    subject_scale=0.9,
    generator=torch.manual_seed(seed),
).images[0]
image.save("flux_instantcharacter_style_ghibli.png")

To use InstantCharacter, you must first initialize the InstantCharacterFluxPipeline using a base model (e.g., black-forest-labs/FLUX.1-dev) and then call init_adapter to load the necessary IP-Adapter and image encoders. This setup prepares the pipeline for character personalization.

Required components for init_adapter:

• image_encoder_path: Path to the first image encoder (e.g., SigLIP).
• image_encoder_2_path: Path to the second image encoder (e.g., DINOv2).
• subject_ipadapter_cfg: A dictionary containing subject_ip_adapter_path and nb_token (typically 1024).

from pipeline import InstantCharacterFluxPipeline
import torch

base_model = 'black-forest-labs/FLUX.1-dev'
pipe = InstantCharacterFluxPipeline.from_pretrained(base_model, torch_dtype=torch.bfloat16)
pipe.to("cuda")

pipe.init_adapter(
    image_encoder_path='google/siglip-so400m-patch14-384', 
    image_encoder_2_path='facebook/dinov2-giant', 
    subject_ipadapter_cfg=dict(subject_ip_adapter_path='path/to/instantcharacter_ip-adapter.bin', nb_token=1024), 
)

The InstantCharacterFluxPipeline is the primary interface for character-preserving generation.

init_adapter

Initializes the IP-Adapter and image encoders.

• image_encoder_path (str): Path to the first image encoder (e.g., SigLIP).
• image_encoder_2_path (str): Path to the second image encoder (e.g., DINOv2).
• subject_ipadapter_cfg (dict): Configuration for the subject IP-Adapter. Must contain:
  • subject_ip_adapter_path (str): Path to the .bin adapter file.
  • nb_token (int): Number of tokens.

__call__ (Inference)

Standard inference call.

• prompt (str): Text description of the scene.
• num_inference_steps (int): Number of diffusion steps.
• guidance_scale (float): CFG scale.
• subject_image (PIL.Image): The reference character image.
• subject_scale (float): Strength of the character preservation.
• generator (torch.Generator): Random seed generator.

with_style_lora

Wraps inference with a LoRA style.

• lora_file_path (str): Path to the .safetensors LoRA file.
• trigger (str): The trigger word for the LoRA style.
• All other arguments are passed to the standard inference call.

The pipeline provides two main ways to generate images: standard inference and inference with a specific style using LoRA.

Standard Inference

Use the __call__ method (via pipe(...)) to generate images based on a prompt and a subject image.

Parameters:

• prompt: Text description of the scene.
• subject_image: The reference character image (ideally with background removed).
• subject_scale: Controls the strength of the subject adaptation.
• num_inference_steps: Number of diffusion steps.
• guidance_scale: CFG scale.
• width / height: Output dimensions (default 1024x1024).
• generator: A torch.Generator object for reproducibility.

Inference with Style (LoRA)

Use pipe.with_style_lora(...) to apply a specific artistic style (like Makoto Shinkai or Ghibli) while maintaining character identity.

Parameters:

• lora_file_path: Path to the .safetensors LoRA file.
• trigger: The trigger word required for the LoRA style.
• All other parameters are identical to standard inference.

# Standard inference
images = pipe(
    prompt="a character is riding a bike", 
    num_inference_steps=28,
    guidance_scale=3.5,
    width=1024,
    height=1024,
    subject_image=input_image,
    subject_scale=1.0,
    generator=torch.manual_seed(123456),
).images

# Inference with style
images = pipe.with_style_lora(
    lora_file_path="path/to/Makoto_Shinkai_style.safetensors",
    trigger="Makoto Shinkai style",
    prompt="a character is riding a bike", 
    num_inference_steps=28,
    guidance_scale=3.5,
    width=1024,
    height=1024,
    subject_image=input_image,
    subject_scale=1.0,
    generator=torch.manual_seed(123456),
).images