JoyCaption

The user prompt used in your training data determines how the model behaves.

• To adjust style only: Use prompts similar to the original JoyCaption prompts (e.g., "Write a descriptive caption for this image in a formal tone.") to build on existing knowledge.
• To change behavior/task: Use distinct, detailed prompts to force the model out of its default behavior (e.g., "I want you to write a JSON object with all the details of this image..."). Since the base model is Llama 3.1, very long, detailed prompts are effective for guiding specific outputs.

Instead of a single string, you can provide a JSON file via --prompt-file. This allows for randomized prompt selection. The JSON file can contain either simple strings or objects with prompt and weight fields. If weights are provided, the probability of a prompt being selected is proportional to its weight (e.g., a weight of 2.0 is twice as likely to be chosen as a weight of 1.0).

[
  { "prompt": "Describe the scene in detail.", "weight": 2.0 },
  { "prompt": "Summarize the main elements of the image.", "weight": 1.0 }
]

To process multiple images at once:

1. Navigate to the "Batch Processing" tab.
2. Upload multiple images (supported formats: PNG, JPEG, WEBP).
3. Configure DataLoader Workers (CPU processes) and Batch Size according to your hardware.
4. Click "Start Batch Process & Create ZIP".
5. Download the resulting ZIP file containing all generated captions once finished.

To set up the JoyCaption Gradio interface, ensure you have Python 3.8+ and a CUDA-capable GPU. You will need at least 24GB of VRAM for bf16 precision or 8GB for nf4 quantized mode.

Follow these steps to install and run the application:

1. Clone the repository and enter the gradio-app directory.
2. Create and activate a Python virtual environment.
3. Install dependencies from requirements.txt.
4. Run app.py and access the interface via the local URL (usually http://127.0.0.1:7860).

git clone https://github.com/yourusername/joy-caption.git
cd joy-caption/gradio-app

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

pip install -r requirements.txt

python app.py

You can improve performance and reduce memory usage by installing optional dependencies:

• Liger Kernel: Install for faster inference.
• bitsandbytes: Required for 4-bit (nf4) and 8-bit quantization modes.

pip install bitsandbytes

vLLM does not currently support LoRA on VLM-type models. To use a fine-tuned JoyCaption model in vLLM, you must first merge the LoRA weights into the base model and save the resulting model and processor.

Step 1: Merge and Save

model = model.merge_and_unload(progressbar=True)
model.save_pretrained("./merged-model")
processor.save_pretrained("./merged-model")

Step 2: Serve with vLLM

vllm serve ./merged-model --max-model-len 4096 --enable-prefix-caching

vllm serve ./questions-cuu2y0sx --max-model-len 4096 --enable-prefix-caching

The batch-caption.py script allows for bulk captioning of images. You must provide an image source (via --glob or --filelist) and a prompt source (via --prompt or --prompt-file). By default, the script writes .txt files containing the captions in the same directory as the source images. If a .txt file already exists for an image, the script will skip it.

./batch-caption.py --glob "path/to/images/*.jpg" --prompt "Write a descriptive caption for this image in a formal tone."

To generate a caption for a single image:

1. Upload an image in the left-hand panel.
2. Choose a Caption Type and desired length.
3. (Optional) Use "Extra Options" for additional parameters or adjust generation settings like temperature and top-p.
4. Click "Caption".
5. Copy or edit the result from the output box.

The PROMPT constant defines the system instructions for a 'Judge' model tasked with comparing two AI responses (Response A and Response B).

Evaluation Criteria:

1. Adherence to Original System Prompt
2. Addressing the User Query
3. Image Integration
4. Helpfulness and Usefulness
5. Accuracy and Factual Correctness
6. Clarity, Conciseness, and Structure
7. Overall Quality

Required Output Format: The judge must provide a structured response including Context Summary, Analysis of Response A, Analysis of Response B, Comparison, Judgment, and Justification. Crucially, it must end with machine-parsable tags:

• <best>Response A</best> or <best>Response B</best>
• <response_a_score>1-10</response_a_score>
• <response_b_score>1-10</response_b_score>

When generating tags for Danbooru, e621, or Rule34, follow these formatting rules:

• Separation: Use commas (e.g., tag1, tag2).
• Case: All tags must be lowercase.
• Spaces: Use underscores instead of spaces (e.g., tag_me).
• Categorization: Prepend the category to tags (e.g., artist:tag1, character:tag2), except for the 'general' category.
• Ordering (Rule34/e621): artist $\rightarrow$ copyright $\rightarrow$ character $\rightarrow$ species $\rightarrow$ meta $\rightarrow$ lore $\rightarrow$ general.
• Ordering (Danbooru): artist $\rightarrow$ copyright $\rightarrow$ character $\rightarrow$ meta.
• Sorting: Within each category, tags must be sorted alphabetically.