Eagle Vision-Language Models

Eagle is a family of frontier vision-language models (VLMs) from NVIDIA designed for multimodal understanding, long-context reasoning, and embodied applications. The family includes several specialized models:

• LocateAnything: A generalist grounding model focused on detection and pointing using Parallel Box Decoding.
• Eagle 2.5: A frontier VLM optimized for state-of-the-art image and video understanding, utilizing specific frameworks and data strategies for long-context multimodal reasoning.
• Eagle 2: A frontier VLM focused on state-of-the-art image understanding and exploring post-training data strategies.
• Eagle: A VLM architecture utilizing a mixture-of-encoders to explore the design space for vision-centric models.

Eagle is a family of Vision-Centric High-Resolution Multimodal LLMs designed to strengthen perception through a mixture of vision encoders and varying input resolutions.

Key technical features include:

• Channel-concatenation-based "CLIP+X" fusion: Integrates vision experts with different architectures (ViT/ConvNets) and knowledge (detection, segmentation, OCR, SSL).
• High Resolution: Supports up to over 1K input resolution.
• Specialized Capabilities: Strong performance on resolution-sensitive tasks such as Optical Character Recognition (OCR) and document understanding.

Eagle models support long-context multimodal reasoning, specifically for video analysis. A primary use case is VLM Captioning, where the model can:

• Analyze long videos.
• Divide videos into distinct sections.
• Generate a title and a detailed caption for each section.
• Indicate the timestamp (in seconds) for the beginning of each section.

Example output format for a section: [timestamp] seconds, [Title] detailed caption: [Description]

The training pipeline uses online streaming packing to efficiently batch variable-length sequences without padding waste. It uses two main strategies:

• Best-Fit: Fills remaining batch space with the largest fitting sample from a buffer.
• Big-Rocks-First: Starts each new batch with the largest buffered sample.

Key Implementation Details:

• Batch Size: Always set --per_device_train_batch_size 1; the packing logic handles effective batch sizing.
• Buffer Size: Controlled by --packing_buffer_size (default 32). Larger buffers improve efficiency but increase memory usage.
• Token Budgets: --max_num_tokens defines the total token budget per packed batch, and --max_num_tokens_per_sample defines the threshold above which samples are dropped.

Eagle2.5 deployment on x86_64 Linux platforms uses a split strategy to enhance inference efficiency by building separate engines for the vision and LLM components.

Vision Component Pipeline:

1. Export the model to ONNX format.
2. Build TensorRT engines from the ONNX model.

LLM Component Pipeline:

1. Convert checkpoints to Hugging Face safetensors using TensorRT-LLM.
2. Build engines using the trtllm-build command.

Requirements:

• TensorRT 10.11
• TensorRT-LLM 0.20 (requires a specific patch for Eagle2.5 inference)
• X86_64 Linux platform

The LocateAnything model provides a unified Vision-Language Model (VLM) capable of diverse localization tasks. Key capabilities include:

• Dense Object Detection: Performs tasks such as document understanding, GUI grounding, dense object detection, and OCR.
• Fast Decoding Speed: Utilizes Parallel Box Decoding (PBD), which predicts bounding boxes atomically in a single forward pass. This provides substantially higher throughput compared to traditional Quantized Coordinate Decoding.

The Eagle family of models is designed for various real-world applications:

• Document & General VQA: Visual Question Answering on graphs, charts, and documents (e.g., analyzing population trends from a graph).
• 3D Perception & Spatial Intelligence: 3D object detection in wild environments using LocateAnything3D.
• Smart City & Metropolis: Zero-shot ultra-dense pedestrian detection in complex urban environments (e.g., busy road crossings).

LocateAnything-3B is a high-throughput vision-language grounding model. Key performance metrics include:

• Throughput: 12.7 BPS on a single H100 (approx. 10x speedup vs Qwen3-VL and 2.5x vs Rex-Omni).
• Common Object Detection: 50.7 LVIS F1@Mean and 54.7 COCO F1@Mean.
• Dense Object Detection: 58.7 Dense200 F1@Mean and 39.9 VisDrone F1@Mean.
• Document Layout & OCR: 76.8 DocLayNet F1@Mean and 70.1 M6Doc F1@Mean.
• GUI Grounding (ScreenSpot-Pro): 60.3 Avg (SOTA).
• Referring Expression Comprehension: 76.7 RefCOCOg val F1@Mean (SOTA).
• Pointing: Achieves best results across all 7 tested benchmarks.

Eagle currently supports Linux OS only. It does not support Windows or macOS. Follow these steps to set up your environment:

1. Clone the repository:

   git clone https://github.com/NVlabs/EAGLE.git
   cd EAGLE

2. Create a Conda environment and install dependencies:

   conda create -n eagle python=3.10 -y
   conda activate eagle
   pip install --upgrade pip  # enable PEP 660 support
   pip install -r requirements.txt
   pip install .

3. Install additional packages for training (if required):

   pip install flash-attn --no-build-isolation

LocateAnything evaluation requires specific datasets and a specific directory structure.

1. Download datasets from:
   • https://huggingface.co/datasets/Mountchicken/Rex-Omni-EvalData
   • https://huggingface.co/datasets/likaixin/ScreenSpot-Pro (for ScreenSpot Pro evaluation)
2. Place converted_box.jsonl inside the ScreenSpot-Pro/ directory.
3. Unpack all .tar.gz image archives in the EvalData directory.

Expected directory layout:

/.../EvalData
  *.tar.gz               # per-dataset image archives
  _annotations/          # JSONL annotations
  ScreenSpot-Pro/         # ScreenSpot-Pro dataset
    images/              # ScreenSpot-Pro images
    converted_box.jsonl  # Downloaded converted_box.jsonl
  _locate_anything_eval_results # Evaluation results output

cd path/to/EvalData
for f in *.tar.gz; do
  echo "Extracting $f" && tar -xzf "$f"
done

Use the provided shell script for parameter-efficient LoRA fine-tuning. You can control the LoRA rank for the LLM and backbone via environment variables.

Environment Variables:

• MODEL_PATH: Base checkpoint (default: nvidia/LocateAnything-3B).
• USE_LLM_LORA: LLM LoRA rank (default: 64; 0 to disable).
• USE_BACKBONE_LORA: Vision-backbone LoRA rank (default: 0).
• FREEZE_LLM, FREEZE_BACKBONE, FREEZE_MLP: Control frozen modules.

export HF_TOKEN=your_hf_token
export META_PATH=./locany_recipe/visual_prompt_recipe.json

bash shell/locate-anything-lora-visual-prompt.sh 1 work_dirs/locany_lora_visual_prompt