Zerox OCR Engine

Focused ReAct is an enhanced version of the ReAct (Reason+Act) paradigm designed to improve performance in Question Answering (QA) tasks. It addresses two primary failure modes of the standard ReAct framework:

1. Context Loss: In long reasoning chains, the model often loses track of the original user query as the context window fills with reasoning steps and observations.
2. Action Loops: The model can become stuck in repetitive cycles, performing the same action multiple times without progressing toward a solution.

Focused ReAct introduces two mechanisms to mitigate these issues: Reiterate and Early Stop.

The Reiterate mechanism prevents the original question from being overshadowed by an increasingly long reasoning context.

Implementation: The original question is restated at the beginning of every reasoning step within the ReAct cycle. This constant re-emphasis ensures the model's outputs remain aligned with the user's initial intent, even as the conversation history grows.

Zerox uses LiteLLM to support various vision model providers. You must configure the appropriate environment variables based on your provider.

Common configurations include:

• OpenAI: Set OPENAI_API_KEY and use model names like gpt-4o-mini.
• Azure OpenAI: Set AZURE_API_KEY, AZURE_API_BASE, and AZURE_API_VERSION. Use the format azure/<your_deployment_name>.
• Gemini: Set GEMINI_API_KEY and use the format gemini/<gemini_model>.
• Anthropic: Set ANTHROPIC_API_KEY.
• Vertex AI: Use vertex_ai/<model_name>. You can provide credentials via gcloud auth application-default login or by passing a JSON string of your service account credentials via the vertex_credentials keyword argument in kwargs.

The maintainFormat: true option is useful for documents with complex layouts or tables that span multiple pages. It ensures consistent formatting by passing the Markdown output of the previous page as context for the current page's request.

Note: This requires requests to run synchronously, making it significantly slower than the default concurrent processing.

Workflow:

1. Request #1 $\rightarrow$ page_1_image
2. Request #2 $\rightarrow$ page_1_markdown + page_2_image
3. Request #3 $\rightarrow$ page_2_markdown + page_3_image

The Early Stop mechanism prevents the model from getting trapped in repetitive action loops.

Implementation: The system monitors for duplicate actions. When the program detects that an action is being repeated, it triggers a termination request. This instructs the model to stop performing new actions and instead generate a final answer based on the information already gathered in the context.

To use the Python SDK, you must first install the system dependency poppler and ensure it is available in your system's PATH. You can find platform-specific instructions in the pdf2image documentation.

After installing poppler, install the py-zerox package using pip:

pip install py-zerox

Install the Zerox Node.js SDK via npm:

npm install zerox

System Dependencies: Zerox requires graphicsmagick and ghostscript for converting PDFs to images. While these may be pulled automatically, you may need to install them manually. On Linux, use:

sudo apt-get update
sudo apt-get install -y graphicsmagick

Zerox can extract specific information from documents into a structured format using JSON Schema.

To use this feature:

1. Set extractOnly: true.
2. Provide a schema following the JSON Schema standard.
3. (Optional) Use extractPerPage to perform extraction on a per-page basis.
4. (Optional) Configure extractionModel, extractionModelProvider, and extractionCredentials to use a different model for the extraction step than the initial OCR step. By default, the same model is used for both.

The zerox function converts documents (PDF, DOCX, images, etc.) into Markdown by converting pages to images and processing them with a vision model. You can provide a file URL or a local file path.

Basic Usage (File URL):

import { zerox } from "zerox";

const result = await zerox({
  filePath: "https://omni-demo-data.s3.amazonaws.com/test/cs101.pdf",
  credentials: {
    apiKey: process.env.OPENAI_API_KEY,
  },
});

Basic Usage (Local Path):

import { zerox } from "zerox";
import path from "path";

const result = await zerox({
  filePath: path.resolve(__dirname, "./cs101.pdf"),
  credentials: {
    apiKey: process.env.OPENAI_API_KEY,
  },
});