Grounding DINO Documentation

Inputs

Grounding DINO requires an (image, text) pair.

Outputs

• Object Boxes: By default, the model outputs 900 object boxes.
• Similarity Scores: Each box contains similarity scores for all input words.
• Filtering:
  • Use box_threshold to select boxes whose highest similarity scores exceed the threshold.
  • Use text_threshold to extract predicted labels from words with high similarity.
• Phrase Selection: To target specific phrases (e.g., "dogs" in "two dogs with a stick"), select boxes with the highest text similarity to the target phrase.

Tips

• Tokenization: A single word may be split into multiple tokens depending on the tokenizer; therefore, the number of words in a sentence may not equal the number of text tokens.
• Category Separation: It is recommended to separate different category names using a period (.) for better performance.

Combine Grounding DINO detections with a StableDiffusionInpaintPipeline to edit images.

1. Convert the source image and the generated mask to PIL images.
2. Resize them to 512x512 (required by the pipeline).
3. Pass the prompt, image, and mask to the pipeline.
4. Resize the resulting image back to the original dimensions.

Note: The mask must be white (255) for the area to be inpainted and black (0) for the area to be kept.

# image_source, image_mask are PIL images
image_source_for_inpaint = image_source.resize((512, 512))
image_mask_for_inpaint = image_mask.resize((512, 512))

prompt = "a cute dinosaur"

# image and mask_image should be PIL images.
image_inpainting = pipe(prompt=prompt, image=image_source_for_inpaint, mask_image=image_mask_for_inpaint).images[0]

# Resize back to original size
image_inpainting = image_inpainting.resize((image_source.size[0], image_source.size[1]))

Import the inference utilities to integrate Grounding DINO into your Python applications.

Workflow:

1. load_model: Load the model using a config file and weights.
2. load_image: Load the image and return both the source and the processed image.
3. predict: Perform inference with box_threshold and text_threshold.
4. annotate: Generate an annotated image frame.
5. Use cv2 to save the result.

from groundingdino.util.inference import load_model, load_image, predict, annotate
import cv2

model = load_model("groundingdino/config/GroundingDINO_SwinT_OGC.py", "weights/groundingdino_swint_ogc.pth")
IMAGE_PATH = "weights/dog-3.jpeg"
TEXT_PROMPT = "chair . person . dog ."
BOX_TRESHOLD = 0.35
TEXT_TRESHOLD = 0.25

image_source, image = load_image(IMAGE_PATH)

boxes, logits, phrases = predict(
    model=model,
    image=image,
    caption=TEXT_PROMPT,
    box_threshold=BOX_TRESHOLD,
    text_threshold=TEXT_TRESHOLD
)

annotated_frame = annotate(image_source=image_source, boxes=boxes, logits=logits, phrases=phrases)
cv2.imwrite("annotated_image.jpg", annotated_frame)

You can perform object detection by loading a model with a specific configuration and weight file, then using the predict function with a text prompt. The predict function returns bounding boxes, confidence logits, and the associated text phrases. Use annotate to draw the detected boxes and labels onto the original image source.

from groundingdino.util.inference import load_model, load_image, predict, annotate
import cv2

# Load model with config and weights
model = load_model("groundingdino/config/GroundingDINO_SwinT_OGC.py", "../04-06-segment-anything/weights/groundingdino_swint_ogc.pth")

IMAGE_PATH = ".asset/cat_dog.jpeg"
TEXT_PROMPT = "chair . person . dog ."
BOX_TRESHOLD = 0.35
TEXT_TRESHOLD = 0.25

# Load and preprocess image
image_source, image = load_image(IMAGE_PATH)

# Run inference
boxes, logits, phrases = predict(
    model=model,
    image=image,
    caption=TEXT_PROMPT,
    box_threshold=BOX_TRESHOLD,
    text_threshold=TEXT_TRESHOLD
)

# Annotate and save result
annotated_frame = annotate(image_source=image_source, boxes=boxes, logits=logits, phrases=phrases)
cv2.imwrite("annotated_image.jpg", annotated_frame)

Use the demo/inference_on_a_image.py script to perform object detection from the command line. You can specify the GPU via CUDA_VISIBLE_DEVICES and use the --cpu-only flag to run on CPU.

Arguments:

• -c: Path to the configuration file.
• -p: Path to the model weights.
• -i: Path to the input image.
• -o: Directory to save the output.
• -t: Text prompt (phrases to detect).
• --token_spans: (Optional) Specifies start and end positions of phrases for precise detection. Format: [[[start, end], [start, end]], ...].

# Basic detection
CUDA_VISIBLE_DEVICES={GPU ID} python demo/inference_on_a_image.py \
-c groundingdino/config/GroundingDINO_SwinT_OGC.py \
-p weights/groundingdino_swint_ogc.pth \
-i image_you_want_to_detect.jpg \
-o "dir you want to save the output" \
-t "chair"

# Detection with specific token spans
CUDA_VISIBLE_DEVICES={GPU ID} python demo/inference_on_a_image.py \
-c groundingdino/config/GroundingDINO_SwinT_OGC.py \
-p ./groundingdino_swint_ogc.pth \
-i .asset/cat_dog.jpeg \
-o logs/1111 \
-t "There is a cat and a dog in the image ." \
--token_spans "[[[9, 10], [11, 14]], [[19, 20], [21, 24]]]"