Docling MCP

docling-mcp v2 uses a hybrid architecture allowing for lightweight remote conversion or full offline local conversion.

Remote Mode (Default)

Uses Docling Serve for cloud-based conversion. This is the lightweight option (~50 MB) and requires no model downloads.

• Setup: pip install docling-mcp
• Config: Set DOCLING_SERVICE_URL and optionally DOCLING_SERVICE_API_KEY.

Local Mode (Full/Offline)

Performs conversion locally on your machine.

• Setup: pip install "docling-mcp[local]"
• Config: Set DOCLING_CONVERSION_MODE=local.

Hybrid Mode

Falls back to local conversion if the remote service is unavailable.

• Setup: pip install "docling-mcp[local]"
• Config: Set DOCLING_CONVERSION_MODE=remote (or leave default) and DOCLING_FALLBACK_TO_LOCAL=true.

# Remote mode (lightweight, default)
pip install docling-mcp
export DOCLING_SERVICE_URL=https://your-docling-service.example.com

# Local mode (full, offline)
pip install "docling-mcp[local]"
export DOCLING_CONVERSION_MODE=local

To use Docling MCP tools that leverage LlamaIndex RAG (Retrieval-Augmented Generation) capabilities, you must configure a Llama Stack backend and install specific dependencies.

Requirements

• A Llama Stack backend providing the Responses API.
• The openai-agents library.

Installation

Use uv to create a virtual environment and install the docling-mcp package with the llama-index-rag extra, along with the necessary agent and notebook tools:

uv venv
uv pip install "docling-mcp[llama-index-rag]" openai-agents ipykernel notebook rich

Starting the Server

Run the docling-mcp-server using streamable-http transport. You must explicitly enable the conversion and llama-index-rag tool groups to access these functionalities:

docling-mcp-server --transport streamable-http --port 8000 --host 0.0.0.0 conversion llama-index-rag

Docling MCP can be installed in three different modes depending on your infrastructure needs:

1. Remote Mode (Recommended): Lightweight client that connects to a docling-serve instance. Requires DOCLING_MCP_SERVICE_URL and DOCLING_MCP_SERVICE_API_KEY.
2. Local Mode: Full features including local document conversion. Requires the [local] extra.
3. Hybrid Mode: Installs local support but defaults to remote, with an automatic fallback to local if the remote service is unreachable.

All configuration is handled via environment variables with the DOCLING_MCP_ prefix.

# Remote Mode
pip install docling-mcp
export DOCLING_MCP_SERVICE_URL=https://your-docling-service.example.com
export DOCLING_MCP_SERVICE_API_KEY=your-api-key-here
export DOCLING_MCP_CONVERSION_MODE=remote

# Local Mode
pip install "docling-mcp[local]"
export DOCLING_MCP_CONVERSION_MODE=local

# Hybrid Mode
pip install "docling-mcp[local]"
export DOCLING_MCP_SERVICE_URL=https://your-docling-service.example.com
export DOCLING_MCP_CONVERSION_MODE=remote
export DOCLING_MCP_FALLBACK_TO_LOCAL=true

To connect Llama Stack to watsonx.ai, use LiteLLM as a proxy to serve the watsonx.ai inference endpoint as an OpenAI-compatible API.

1. Create an env.secrets file containing your WATSONX_URL, WATSONX_APIKEY, and WATSONX_PROJECT_ID.
2. Create a litellm_config.yaml to map model names to their watsonx paths.
3. Run LiteLLM in a container.
4. Run the Llama Stack starter distribution, setting VLLM_URL to point to the LiteLLM proxy (default port 4000).
5. Register models and test via llama-stack-client.

# litellm_config.yaml example
model_list:
  - model_name: "granite-3-3-8b-instruct"
    litellm_params:
      model: "watsonx/ibm-granite/granite-3-3-8b-instruct"
  - model_name: "llama-3-3-70b"
    litellm_params:
      model: "watsonx/meta-llama/llama-3-3-70b-instruct"

# 1. Run LiteLLM proxy
podman run \
  -v $(pwd)/litellm_config.yaml:/app/config.yaml \
  --rm \
  -p 4000:4000 \
  --env-file env.secrets \
  ghcr.io/berriai/litellm:v1.74.9-stable \
  --config /app/config.yaml

# 2. Run Llama Stack pointing to LiteLLM
export LLAMA_STACK_PORT=8321

podman run \
    -it \
    --pull always \
    -p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
    -v ~/.llama:/root/.llama \
    llamastack/distribution-starter:0.2.18 \
    --port $LLAMA_STACK_PORT \
    --env VLLM_URL=http://host.containers.internal:4000

# 3. Register and Test
uvx --from llama-stack-client llama-stack-client models register wx/llama-3-3-70b --provider-id vllm --provider-model-id llama-3-3-70b

uvx --from llama-stack-client llama-stack-client --endpoint http://localhost:8321 \
    inference chat-completion \
    --model-id wx/llama-3-3-70b \
    --message "Write a short story about a robot."

This guide demonstrates how to integrate Docling MCP into the Llama Stack framework to create agentic workflows. This requires running an inference model via Ollama, a Llama Stack server (via Podman/Docker), and the Docling MCP server locally.

Prerequisites

• uv
• Ollama
• Podman

1. Prepare the Inference Model

Load a model (e.g., Meta Llama 3.2) using Ollama:

export OLLAMA_INFERENCE_MODEL="llama3.2:3b-instruct-fp16"
ollama run $OLLAMA_INFERENCE_MODEL --keepalive 60m

2. Run Llama Stack Server

Create a local directory for persistence and start the Llama Stack starter distribution using Podman:

mkdir -p ~/.llama

export LLAMA_STACK_PORT=8321

podman run \
  -it \
  --pull always \
  -p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \
  -v ~/.llama:/root/.llama \
  llamastack/distribution-starter \
  --port $LLAMA_STACK_PORT \
  --env OLLAMA_URL=http://host.containers.internal:11434

3. Run Docling MCP Server

Clone the docling-mcp repository, sync dependencies with uv, and start the server using the streamable-http transport:

git clone git@github.com:docling-project/docling-mcp.git
cd docling-mcp
uv sync

uv run docling-mcp-server --transport streamable-http --port 8000 --host 0.0.0.0

4. Register Docling Tools in Llama Stack

Use the llama-stack-client CLI to register the Docling MCP server as a tool group:

uvx --with llama-stack-client llama-stack-client toolgroups register "mcp::docling" \
  --provider-id="model-context-protocol" \
  --mcp-endpoint="http://host.containers.internal:8000/mcp"

To verify registration, list the toolgroups:

uvx --with llama-stack --from llama-stack-client llama-stack-client toolgroups list

# Registering the toolgroup
uvx --with llama-stack-client llama-stack-client toolgroups register "mcp::docling" \
  --provider-id="model-context-protocol" \
  --mcp-endpoint="http://host.containers.internal:8000/mcp"