DOCSEARCHbyAlgolia
Submit Docs

Compel

repository·main·Indexed 20 days ago

https://github.com/damian0815/compel

A prompting enhancement library for transformer-based text embedding systems, optimized for the Hugging Face diffusers ecosystem. Compel provides intuitive syntax (such as ++ and --) for weighting and blending specific parts of a prompt to influence resulting embedding tensors. It includes specialized classes like CompelForSD, CompelForSDXL, and CompelForFlux to streamline embedding management, handle long prompts without truncation, and support complex prompt conjunctions via the .and() method.

Tokens
17.9K
Snippets
64
Records
74
Agent score
67%

What's inside compel

  1. Apply attention weights to prompt tokens

    main

    You can increase or decrease the importance of specific words or phrases in a prompt using attention weights. Weights can be applied to single words or groups of words enclosed in parentheses.

    Weighting Methods

    • Incremental Symbols: Use + to increase importance and - to decrease it. Multiple symbols stack multiplicatively (e.g., ++ is more powerful than +).
      • + is equivalent to a weight of 1.1.
      • - is equivalent to a weight of 0.9.
    • Explicit Numerical Weights: Use a number between 0 and 2 (where 1 is the default) to set an exact importance level.
      • (token)0.5 reduces importance to half.
      • (token)1.5 increases importance by 1.5x.

    Syntax Rules

    • Single words: word+ or word-.
    • Phrases: Use parentheses to group words before applying the weight, e.g., (picking apricots)+.
    • Nesting: You can nest weights. For example, (picking (apricots)1.3)1.1 applies a 1.3 weight to apricots, and then applies a 1.1 weight to the entire group.
    a tall thin man (picking apricots)1.5
a tall thin man (picking apricots)++
apricot++
apricot--
a tall thin man (picking (apricots)1.3)1.1

  2. Concatenate embeddings using .and()

    main

    Compel allows you to break complex prompts into segments and concatenate them using the .and() method. This is particularly useful for Stable Diffusion 2.1 to improve generation quality by feeding different parts of the prompt to the text encoder separately.

    You can pass a tuple of strings to .and() and optionally provide weights for each segment.

    Example syntax:

    • ("part 1", "part 2").and()
    • ("part 1", "part 2", "part 3").and(1, 0.5, 0.5)
    ("A dream of a distant galaxy, by Caspar David Friedrich, matte painting", "trending on artstation, HQ").and()

  3. Use CrossAttentionControlSubstitute for prompt-to-prompt editing

    main

    The CrossAttentionControlSubstitute object allows for 'prompt2prompt' style editing. It takes an original sequence of fragments and an edited sequence. The attention maps from the original sequence are applied to the edited sequence, allowing you to swap concepts while maintaining the structure and composition of the original image.

    You can use this for specific token swaps or for larger portions of the prompt. It must be embedded within a Prompt or FlattenedPrompt.

    # Example: Swapping a cat for a dog in a specific phrase
flattened = FlattenedPrompt([
        Fragment('a'),
        CrossAttentionControlSubstitute(original=[Fragment('cat')], edited=[Fragment('dog')]),
        Fragment('sitting on a car')
    ])

  4. How weighting, blending, and conjunctions work in Compel

    main

    Compel works by parsing a prompt into a Conjunction structure and then constructing a conditioning tensor based on that structure.

    Weighting

    Weighting uses masked scaling of the conditioning tensor.

    • Upweighting: Works well up to approximately 1.5 or 1.6 depending on the model and CFG.
    • Downweighting: Can go down to 0 (where terms disappear). There is a non-linear inflection point around 0.5 where the model's attention to the term changes significantly.

    Blending

    Blends are implemented as a mathematical linear interpolation (lerp) of the conditioning tensors.

    Conjunctions

    Conjunctions work by concatenating prompts together. When using conjunctions, you must use pad_conditioning_tensors_to_same_length to ensure compatibility with negative prompts.

  5. How the prompt parsing hierarchy works

    main

    Compel uses a tree-like structure to represent complex prompts. Understanding the relationship between these objects is key to using the library correctly:

    1. Conjunction: The top-level container. It holds one or more Prompt or Blend objects. These are intended to be diffused separately and then merged via a weighted sum in latent space.
    2. Prompt: A mid-level structure representing a parsed segment of a prompt. Because it can be nested (e.g., containing Blend or Attention objects), it is not suitable for direct tokenization.
    3. FlattenedPrompt: The result of calling .flatten() on a Conjunction. It converts the nested tree into a linear sequence of Fragment or CrossAttentionControlSubstitute objects that can be directly tokenized.
    4. Fragment: The leaf node. A chunk of plain text and an optional weight, intended to be passed to the CLIP tokenizer.
    5. Attention: A mechanism for nestable weight control. Weights accumulate as you traverse deeper into the tree.
    6. Blend: Represents a weighted interpolation (lerp) between multiple Prompt objects in feature vector space.

    Workflow Tip: Do not attempt to traverse or tokenize Prompt objects directly. Always start with a Conjunction and use the flatten method to obtain a FlattenedPrompt.

  6. Quickstart: Use Compel with Stable Diffusion (SD v1.5)

    main

    To use Compel with standard Stable Diffusion pipelines (diffusers >=0.12), use the CompelForSD class. This allows you to use syntax like ++ to upweight specific parts of a prompt. The compel() call returns a conditioning object containing embeds and optionally negative_embeds.

    from diffusers import StableDiffusionPipeline
from compel import CompelForSD

pipeline = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")
compel = CompelForSD(pipeline)

# upweight "ball"
prompt = "a cat playing with a ball++ in the forest"
conditioning = compel(prompt)

# generate image
images = pipeline(prompt_embeds=conditioning.embeds, num_inference_steps=20).images
images[0].save("image.jpg")

  8. Escape parentheses and speech marks in prompts

    main

    Because parentheses () and speech marks "" are used for syntax (weighting and blending), you must escape them with a backslash \ if you want them to be treated as literal text in your prompt.

    Example: To use the literal text (my_keyword), write \(my_keyword\).

    \(my_keyword\)

  9. Use Compel with SDXL

    main

    To use Compel with Stable Diffusion XL (SDXL), you must provide both tokenizers and both text encoders to the Compel constructor. You should also specify the returned_embeddings_type and set requires_pooled to account for the two different text encoders used by SDXL.

    Note: If you are using pipeline.enable_sequential_cpu_offloading(), you must pass device='cuda' (or your specific device) during Compel initialization.

    from compel import Compel, ReturnedEmbeddingsType
from diffusers import DiffusionPipeline
import torch

pipeline = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", variant="fp16", use_safetensors=True, torch_dtype=torch.float16).to("cuda")
compel = Compel(
    tokenizer=[pipeline.tokenizer, pipeline.tokenizer_2], 
    text_encoder=[pipeline.text_encoder, pipeline.text_encoder_2], 
    returned_embeddings_type=ReturnedEmbeddingsType.PENULTIMATE_HIDDEN_STATES_NON_NORMALIZED, 
    requires_pooled=[False, True]
)

# upweight "ball"
prompt = "a cat playing with a ball++ in the forest"
conditioning, pooled = compel(prompt)

# generate image
image = pipeline(prompt_embeds=conditioning, pooled_prompt_embeds=pooled, num_inference_steps=30).images[0]

  10. Use Compel with Flux

    main

    For Flux models, use the CompelForFlux class. Similar to SDXL, Flux requires pooled_prompt_embeds in the pipeline call.

    from diffusers import FluxPipeline
from compel import CompelForFlux
import torch

device = "mps"
pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", torch_dtype=torch.float32).to(device)
compel = CompelForFlux(pipe)
prompt = "Astronaut---- in a jungle++++, cold color palette, muted colors, detailed, 8k"

conditioning = compel(prompt)
generator = torch.Generator().manual_seed(42)
images = pipe(
    prompt_embeds=conditioning.embeds, 
    pooled_prompt_embeds=conditioning.pooled_embeds,
    num_inference_steps=4, width=512, height=512, generator=generator
)

  11. Use Weighting to adjust prompt attention

    main

    You can increase or decrease the attention (CFG weighting multiplier) the model pays to specific words or phrases using +, -, or explicit numeric weights between 0 and 2.

    • Increasing attention: Use + or a number > 1. Each + is equivalent to a 1.1 multiplier (e.g., ++ is 1.1^2).
    • Decreasing attention: Use - or a number < 1. Each - is equivalent to a 0.9 multiplier (e.g., -- is 0.9^2).
    • Syntax options:
      • Single words: word+ or word-.
      • Phrases with parentheses: (multiple words)+ or (multiple words)-.
      • Nesting: (word+)+ effectively applies multiple weights to the inner term.

    This is useful for controlling the intensity of specific subjects or balancing the relationship between different parts of a prompt (e.g., mountain+ man vs mountain man+).

    # Examples of weighting syntax
a tall thin man picking apricots+
(apricots)+
(apricots)++
(apricots)1.3
mountain+ man
mountain man++

  12. Handle long prompts without truncation

    main

    By default, Compel truncates prompts that exceed the model's maximum token length. To use the full prompt, initialize Compel with truncate_long_prompts=False.

    When using non-truncated prompts, you must ensure that your positive and negative conditioning tensors have the same length. If you are not using a negative prompt, you must still create an empty conditioning tensor for it and use compel.pad_conditioning_tensors_to_same_length() to align them.

    compel = Compel(..., truncate_long_prompts=False)
prompt = "a cat playing with a ball++ in the forest, ... [very long prompt] ..."
conditioning = compel.build_conditioning_tensor(prompt)

negative_prompt = "" # necessary to create an empty prompt
negative_conditioning = compel.build_conditioning_tensor(negative_prompt)

[conditioning, negative_conditioning] = compel.pad_conditioning_tensors_to_same_length([conditioning, negative_conditioning])