The All-Seeing Project

The All-Seeing Project is a research initiative focused on panoptic visual recognition and general relation comprehension in an open world. It consists of two main versions, each providing foundation models and large-scale datasets:

All-Seeing Project (V1)

• ASM (All-Seeing Model): A unified vision-language foundation model designed for open-world panoptic visual recognition and understanding. It supports image-text retrieval and generation with strong zero-shot capabilities.
• AS-1B (All-Seeing 1B) Dataset: A large-scale dataset for open-world panoptic visual recognition, created using a semi-automatic data engine. Subsets include AS-100M, AS-10M, and AS-Core (a human-verified subset).

All-Seeing Project V2

• ASMv2 (All-Seeing Model v2): An upgraded model that integrates Relation Conversation (ReC) capabilities. It excels at grounding, referring, and scene graph generation tasks.
• AS-V2 (All-Seeing Dataset V2): A dataset containing 127K high-quality relation conversation samples designed to unlock ReC capabilities in Multi-modal Large Language Models (MLLMs).
• CRPE Benchmark: The Circular-based Relation Probing Evaluation benchmark, used to systematically evaluate relation comprehension across (subject, predicate, object) triplets.

ASMv2 (All-Seeing Model V2) is a multimodal model capable of several advanced visual-language tasks. Its primary capabilities include:

• Multimodal Dialogue with Grounded Relation: Engaging in dialogue that includes grounding specific visual relations.
• Multimodal Dialogue: General visual-language reasoning across various benchmarks (e.g., VQAv2, GQA, TextVQA).
• Multimodal Dialogue with Pointer Instructions: Performing Referring Expression Comprehension (REC) to identify specific objects based on text descriptions.
• Region Captioning: Generating descriptive text for specific regions within an image.
• Visual Commonsense Reasoning: Performing tasks like Question-to-Answer (Q $\rightarrow$ A), Question-Answer-to-Relation (QA $\rightarrow$ R), and Question-to-Attribute-Relation (Q $\rightarrow$ AR).
• CRPE (Commonsense Relation Perception and Extraction): Evaluating existence, subjects, predicates, and objects in visual relations.
• Open-ended Scene Graph Generation: Generating scene graphs (tuples of relations) from visual input.

The AS-V2 dataset uses a Relation Conversation (ReC) formulation where objects and predicates are explicitly marked within text to facilitate scene graph parsing.

Tagging Convention

• Objects: Marked using <ref></ref> tags. Each marked object is followed by its corresponding bounding box.
• Predicates: Marked using <pred></pred> tags. Each marked predicate is followed by two bounding boxes representing the subject and the object of that predicate.

Bounding Box Format

All bounding boxes are normalized to integer values in the range $[0, 1000)$ and follow this structure:

<box>[[x1, y1, x2, y2], ..., [x1, yn, x2, yn]]</box>

Where [x1, y1, x2, y2] denotes the $i$-th bounding box linked to the object or predicate.

For detailed parsing logic, refer to the project's paper or the scripts_asmv2/visualize_rec.py script.

<ref>object_name</ref><box>[[x1, y1, x2, y2]]</box>
<pred>predicate_name</pred><box>[[subject_x1, subject_y1, subject_x2, subject_y2], [object_x1, object_y1, object_x2, object_y2]]</box>

To run the LLaVA LoRA technical preview demo, you must launch three components sequentially: a controller, a Gradio web server, and a model worker.

Prerequisites:

• You must use the latest LLaVA codebase.
• You need the Vicuna v1.1 (7B) checkpoint as the --model-base.
• The --model-base must correspond to the LoRA checkpoint being used.

Steps:

1. Launch the Controller: This manages the communication between workers and the UI.
2. Launch the Gradio Web Server: This provides the user interface. Note that the model list will be empty until a worker is launched.
3. Launch a Model Worker: This loads the actual LoRA weights and the base model. Once the worker is running, refresh the Gradio UI to see the model in the list.

You can launch multiple workers to compare different checkpoints in the same interface by using different --port and --worker values while keeping the same --controller address.

# 1. Launch a controller
python -m llava.serve.controller --host 0.0.0.0 --port 10000

# 2. Launch a gradio web server
python -m llava.serve.gradio_web_server --controller http://localhost:10000 --model-list-mode reload

# 3. Launch a model worker
python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path liuhaotian/llava-vicuna-7b-v1.1-lcs_558k-instruct_80k_3e-lora-preview-alpha --model-base /path/to/vicuna-v1.1

To run the Gradio demo, you must first prepare LLaVA checkpoints locally. The architecture consists of a Controller (API Server), a Gradio Web Server (UI), and one or more Model Workers (Inference Engines).

Follow these steps in order:

1. Launch the Controller: This acts as the central hub.
2. Launch the Gradio Web Server: This provides the user interface. It will initially show an empty model list.
3. Launch Model Workers: Each worker performs inference for a specific model. Once a worker is running, refresh the Gradio UI to see the model in the list.

You can launch multiple workers on different ports to compare different checkpoints within the same interface.

# 1. Launch the controller
python -m llava.serve.controller --host 0.0.0.0 --port 10000

# 2. Launch the gradio web server
python -m llava.serve.gradio_web_server --controller http://localhost:10000 --model-list-mode reload

# 3. Launch a model worker (e.g., LLaVA-1.5-13B)
python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path liuhaotian/llava-v1.5-13b