UForm Multimodal AI Library

To reduce memory usage and increase speed, you can convert embeddings into smaller numeric representations.

• f16 (Half-precision): Recommended in almost all cases for a ~2x speedup with minimal accuracy loss, unless using very old hardware.
• i8 (Integer 8-bit): Possible via linear scaling, but may impact recall on very large collections (millions of entries).
• b1 (Binary): Quantizing high-dimensional embeddings (512 or 768) into single-bit representations for faster search.

Note: These approaches are natively supported by the USearch vector-search engine and SimSIMD numerics libraries.

import numpy as np

f32_embedding: np.ndarray = model.encode_text(text_data, return_features=False)
f16_embedding: np.ndarray = f32_embedding.astype(np.float16)
i8_embedding: np.ndarray = (f32_embedding * 127).astype(np.int8)
b1_embedding: np.ndarray = np.packbits((f32_embedding > 0).astype(np.uint8))

To increase throughput, wrap the model encoders in torch.nn.DataParallel.

Important Notes:

1. Set model.return_features = False on the original models before wrapping.
2. When using the DataParallel wrapper, you must use the .forward() method instead of .encode().
3. Use .detach().cpu().numpy() to convert the resulting tensors back into NumPy arrays.

from uform import get_model, Modality
import torch.nn as nn

# Load model
processors, models = get_model('unum-cloud/uform-vl-english-small', backend='torch')

model_text = models[Modality.TEXT_ENCODER]
model_image = models[Modality.IMAGE_ENCODER]
processor_text = processors[Modality.TEXT_ENCODER]
processor_image = processors[Modality.IMAGE_ENCODER]

# Prepare for parallel
model_text.return_features = False
model_image.return_features = False
model_text_parallel = nn.DataParallel(model_text)
model_image_parallel = nn.DataParallel(model_image)

# Usage pattern
def get_image_embedding(images):
    preprocessed = processor_image(images)
    embedding = model_image_parallel.forward(preprocessed)
    return embedding.detach().cpu().numpy()

UForm generative models are designed for Chat, Image Captioning, and Visual Question Answering (VQA). They are natively compatible with the transformers library. You can load them using AutoModel and AutoProcessor with trust_remote_code=True.

from transformers import AutoModel, AutoProcessor
import torch
from PIL import Image

model = AutoModel.from_pretrained('unum-cloud/uform-gen2-dpo', trust_remote_code=True)
processor = AutoProcessor.from_pretrained('unum-cloud/uform-gen2-dpo', trust_remote_code=True)

prompt = 'Question or Instruction'
image = Image.open('image.jpg')

inputs = processor(text=[prompt], images=[image], return_tensors='pt')

with torch.inference_mode():
     output = model.generate(
        **inputs,
        do_sample=False,
        use_cache=True,
        max_new_tokens=256,
        eos_token_id=151645,
        pad_token_id=processor.tokenizer.pad_token_id
    )

prompt_len = inputs['input_ids'].shape[1]
decoded_text = processor.batch_decode(output[:, prompt_len:])[0]

When using CUDA 12 or newer with ONNX backends, you must install the Nvidia toolkit and the specific onnxruntime-gpu package from the custom repository.

Follow these steps to set up the environment:

1. Install the Nvidia CUDA keyring and toolkit.
2. Install onnxruntime-gpu using the specific extra index URL.
3. Export CUDA_PATH, PATH, and LD_LIBRARY_PATH to point to your CUDA installation.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12
pip install onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/
export CUDA_PATH="/usr/local/cuda-12/bin"
export PATH="/usr/local/cuda-12/bin${PATH:+:${PATH}}"
export LD_LIBRARY_PATH="/usr/local/cuda-12/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}"