Docling Serve

When the Ray dispatcher is unavailable or in an invalid state, the docling-serve API follows a specific error contract to ensure clients can handle the failure gracefully:

• HTTP Status Code: 503 Service Unavailable (not 500 Internal Server Error).
• Header: Includes Retry-After: 1 to signal to clients that they should wait before retrying.
• Scope: This applies to both asynchronous Ray submissions and synchronous submissions that internally enqueue work.
• Error Type: The system uses DispatcherUnavailableError to normalize various lower-level Ray exceptions (including process-exit style failures) into this controlled response.

The Docling Serve API provides two distinct endpoints to handle different input types. This separation allows users to send files directly in binary format, which is more efficient than using base64-encoded strings.

• URL endpoint: Used for processing documents via web URLs.
• File endpoint: Used for processing documents sent as direct binary file uploads.

When using the page-slicing fan-out feature, a task might experience partial failures (where some page slices succeed and others fail).

Understanding the status hierarchy is critical:

1. Task-level TaskStatus: This remains binary. If at least one slice succeeds, the overall task status is SUCCESS. If all slices fail, the task status is FAILURE.
2. Document-level ConversionStatus: To identify if a document is incomplete, check the result document's ConversionStatus. If some slices failed but others succeeded, the status will be PARTIAL_SUCCESS. The errors from the failed slices are carried within the result document.

When running with the RQ engine (DOCLING_SERVE_ENG_KIND=rq), Docling Serve provides end-to-end visibility by propagating trace contexts from the API to the workers:

1. API Request: FastAPI creates a trace when a document conversion request arrives.
2. Job Enqueue: The trace context is injected into the RQ job metadata.
3. Worker Execution: The RQ worker extracts the trace context and continues the trace.

This allows you to track document processing latency across the distributed system and identify bottlenecks in the conversion pipeline using tools like Grafana Tempo.

The Ray dispatcher provides durable task status by using Redis as a fallback. If a task is requested via task_status() or get_raw_task() but is not present in the dispatcher's local in-memory self.tasks (which can happen after an API restart), the dispatcher will:

1. Attempt to reconstruct the task from Redis metadata.
2. Repopulate its local self.tasks cache from the Redis record.

This ensures that task status remains visible and prevents false 404 errors following service restarts. Reconstructed tasks preserve status, task_type, timestamps, and error messages.

The response format depends on the input and the target parameter:

• Single File: Returns a JSON document containing document (with md_content, json_content, html_content, text_content, and doctags_content), status, processing_time, timings, and errors.
• Zip Mode: If target is set to zip mode, or if multiple files are processed, the response is a .zip file containing the results.

{
  "document": {
    "md_content": "",
    "json_content": {},
    "html_content": "",
    "text_content": "",
    "doctags_content": ""
  },
  "status": "<success|partial_success|skipped|failure>",
  "processing_time": 0.0,
  "timings": {},
  "errors": []
}

The RayTaskDispatcher is a named detached actor (created with get_if_exists=True) that manages work distribution. It is important to note its specific responsibilities and limitations:

• Fairness: It enforces fairness across different tenants.
• Admission Control: It prevents a tenant from admitting more work than their configured limits allow.
• Queue Management: It moves work from per-tenant Redis queues into Ray Serve in a fair order.
• Not an Execution Queue: It is not the authoritative execution queue for all downstream work. Because Ray Serve can queue work internally, a task counted as "active" by the dispatcher might already be admitted downstream but not yet executing on a Serve replica. Therefore, max_concurrent_tasks acts as an admission-control bound at the dispatcher layer, rather than a strict count of currently executing replicas.

To handle cases where a dispatcher actor dies during an active task, the system implements a specific reconciliation policy to recover or fail tasks based on their durable metadata:

• Auto-Recovery (Started Tasks): If an active task has status=started in its durable metadata but its task:{id}:processing key is missing, the system marks the task as FAILURE, clears its dispatch_state, publishes an update, and releases its capacity.
• Pre-start Tasks: If an active task is still in pending or dispatched states and has no processing key, it remains unresolved.
• Active Processing: If a task has processing state with status=processing, the system leaves it alone (it is assumed to be making progress).
• Capacity Resync: After each tenant reconciliation, the system resyncs counters from the canonical Redis structures.