Describe Anything Model (DAM)

The demo requires the segment-anything package. Install it directly from the GitHub repository using pip:

pip install git+https://github.com/facebookresearch/segment-anything.git

The web demo requires a quantized ONNX model. You can generate this using the quantize_dynamic function from the ONNX tools.

Important: If you change the ONNX model by using a new checkpoint, you must also re-export the image embedding to match.

Example quantization code:

onnx_model_path = "sam_onnx_example.onnx"
onnx_model_quantized_path = "sam_onnx_quantized_example.onnx"
quantize_dynamic(
    model_input=onnx_model_path,
    model_output=onnx_model_quantized_path,
    optimize_model=True,
    per_channel=False,
    reduce_range=False,
    weight_type=QuantType.QUInt8,
)

After exporting, copy the file to /model/sam_onnx_quantized_example.onnx in the project directory.

To use a custom image in the web demo, you must generate a corresponding .npy embedding file using the Segment Anything (SAM) Python predictor.

1. Initialize the SAM predictor in Python:

checkpoint = "sam_vit_h_4b8939.pth"
model_type = "vit_h"
sam = sam_model_registry[model_type](checkpoint=checkpoint)
sam.to(device='cuda')
predictor = SamPredictor(sam)

2. Set the image and export the embedding:

image = cv2.imread('src/assets/dogs.jpg')
predictor.set_image(image)
image_embedding = predictor.get_image_embedding().cpu().numpy()
np.save("dogs_embedding.npy", image_embedding)

3. Place the resulting .npy file and the original image in the src/assets/data directory of the frontend project.

You can install the Describe Anything Model (DAM) package either directly via pip or by cloning the repository for local installation.

Direct Installation:

pip install git+https://github.com/NVlabs/describe-anything

Local Installation:

git clone https://github.com/NVlabs/describe-anything
cd describe-anything
pip install -v .

To evaluate models on DLC-Bench, you need the dam package and vllm to serve Llama 3.1 8B for output evaluation.

1. Install the dam package following the general installation instructions in the main repository README.
2. Install vllm compatible with your CUDA version. For CUDA 11.8 and Python 3.10, use the specific wheel provided below.

export VLLM_VERSION=0.5.3.post1
export PYTHON_VERSION=310
pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux1_x86_64.whl --extra-index-url https://download.pytorch.org/whl/cu118

The evaluation process consists of three main steps: starting a vLLM backend, generating model outputs, and evaluating those outputs.

1. Start a vLLM backend

The vLLM backend evaluates the generated model outputs. It can run on a different GPU than the model inference. If running on the same GPU, use --gpu-memory-utilization 0.5 to prevent out-of-memory errors.

2. Obtain model outputs

Use get_model_outputs.py to run inference. Outputs are cached in model_outputs_cache/.

3. Evaluate model outputs

Use eval_model_outputs.py to perform the final evaluation against the vLLM backend.

The frontend is pre-built. If you make modifications to the frontend code, you must rebuild it by navigating to the frontend directory and using yarn:

cd frontend
yarn
yarn build

The Segment Anything Simple Web demo is a front-end only React application that runs the SAM ONNX model in the browser using WebAssembly with multithreading (via SharedArrayBuffer, Web Worker, and SIMD128).

To run the application, you must have yarn installed. Use the following commands to build and start the development server:

1. Install Yarn globally if you haven't already:

   npm install --g yarn

2. Build and start the app:

   yarn && yarn start

Once running, navigate to http://localhost:8081/. You can interact with the demo by moving your cursor to see mask predictions update in real time.

To run the full interactive demo, navigate to the demo directory and execute app.py.

Note: There are also specific Gradio demos available for image and video tasks:

• demo_simple.py: Interactive Gradio interface for drawing masks on images (tested with gradio 5.5.0).
• demo_video.py: Interactive Gradio interface for drawing masks on videos (tested with gradio 5.5.0).

cd demo
python app.py

Download the DLC-Bench dataset from Hugging Face and place the DLC-bench folder in the evaluation directory using Git LFS.

git lfs install
git clone https://huggingface.co/datasets/nvidia/DLC-Bench

To start the demo, execute the following command from the root of the demo directory:

python app.py