NuExtract

NuExtract3 is a 4B vision-language reasoning model designed for document understanding. It provides two primary capabilities:

1. Structured Extraction: Converts input (text, images, or both) into structured JSON output based on a provided JSON template and optional instructions.
2. Markdown Conversion: Converts document images into clean Markdown, using HTML for tables, LaTeX for math, and a specific <figure> tag for images.

The model is multilingual and supports both reasoning and non-reasoning inference modes. It is suitable for OCR, RAG preprocessing, and extraction pipelines for documents like receipts, invoices, contracts, and forms.

When using NuExtract for document-to-Markdown conversion, the output follows these formatting rules:

• Text: Standard Markdown (e.g., headers, bold text).
• Tables: Represented using HTML <table> tags.
• Math: Represented using LaTeX.
• Images: Represented using a specific <figure> tag containing an <img> element.

Example image tag format:

<figure data-type="image" data-id="img_n">
  <img src="path/to/image.png" alt="Description of the image"/>
</figure>

Structured extraction in NuExtract requires four components:

1. Input Document: Can be text, an image, or a combination of both.
2. JSON Template: A template that defines the structure and types of the desired output. The template's structure must be identical to the expected output JSON.
3. Instructions (Optional): Provided via the instructions chat template kwarg to specify expected output formats or specific values.
4. In-Context Learning (ICL) examples (Optional): Examples provided to guide the model's extraction behavior.

NuExtract supports two inference modes controlled via enable_thinking in chat_template_kwargs:

• Non-thinking mode (enable_thinking = False): Best for fast, deterministic extraction or Markdown conversion. Recommended temperature is low (e.g., 0.2).
• Thinking mode (enable_thinking = True): Use for difficult documents, complex layouts, or ambiguous fields requiring additional reasoning. Recommended temperature is higher (e.g., 0.6).

For production, start with non-reasoning mode and only enable reasoning for difficult examples.

Any arguments appended to the docker run command are passed directly to the underlying vLLM engine. It is strongly recommended to provide an --api-key for security.

# Run with an API key
docker run -it --rm --name nuextract --network=host \
    --runtime nvidia --gpus all \
    NuExtract-2.0-8B --api-key putarealsecret

# Verify with curl
curl http://localhost:8000/v1/models -H "Authorization: Bearer putarealsecret"

After building the images, you can run them using docker run. Use --network=host to allow access to the API on your localhost.

• For CPU: Run the CPU-tagged image.
• For GPU: You must include the --runtime nvidia and --gpus all flags.

# Run CPU image
docker run -it --rm --name nuextract --network=host \
    NuExtract-2.0-8B.cpu

# Run GPU image
docker run -it --rm --name nuextract --network=host \
    --runtime nvidia --gpus all \
    NuExtract-2-8B

NuExtract provides optimized Dockerfiles for different hardware targets.

• CPU Target: Requires an AVX512 capable CPU. Note that GPTQ models are currently not supported by the vLLM CPU implementation.
• GPU Target: Uses CUDA and automatically sets tensor-parallel-size based on the number of available NVIDIA GPUs.

To build the images, use the docker build command with the appropriate Dockerfile and build arguments.

# Build for CPU
docker build . \
    --build-arg CPU=1 \
    -t NuExtract-2.0-8B.cpu \
    -f NuExtract-2.0-8B.dockerfile

# Build for GPU
docker build . \
    -t NuExtract-2-8B \
    -f NuExtract-2-8B.dockerfile

To reduce container startup time, you can use RClone to synchronize the vLLM compilation cache (e.g., from an S3 bucket).

1. Create a rclone.conf file configured with your remote storage.
2. Mount the config file into the container using a bind mount when running the Docker command.

# Example: Running CPU image with RClone config mounted
docker run -it --rm --name nuextract --network=host \
    --mount type=bind,source=rclone.conf,target=/root/.config/rclone/rclone.conf,readonly=true \
    NuExtract-2.0-8B.cpu

NuExtract can be served using vLLM via an OpenAI-compatible API. Use the following command to deploy the model with Multi-Token Prediction (MTP) enabled for improved throughput. If you encounter memory issues, reduce --max-model-len and the number of images allowed via --limit-mm-per-prompt.

vllm serve numind/NuExtract3 \
  --trust-remote-code \
  --limit-mm-per-prompt '{"image": 99, "video": 0}' \
  --chat-template-content-format openai \
  --generation-config vllm \
  --max-model-len 131072 \
  --speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'

NuExtract expects prompts to follow a specific structure containing a # Template:, an optional # Examples section, and a # Context: section.

Basic Structure:

# Template:
{template}
# Context:
{text}

With In-Context Examples:

# Template:
{template}
# Examples
## Input:
{input1}
## Output:
{output1}
## Input:
{input2}
## Output:
{output2}
# Context:
{text}

When using images, use image placeholders in the text fields. The model's apply_chat_template method can handle this formatting automatically if you provide the template and examples arguments.