RepoMaster Documentation

Marker is a pipeline for converting PDFs to Markdown and JSON with high accuracy. It supports both GPU and CPU and includes specialized converters for tables and OCR.

Key Components:

• PdfConverter: General PDF to Markdown/JSON conversion.
• TableConverter: Specialized extraction of table structures.
• OCRConverter: For handling scanned documents or images.
• ConfigParser: Used to customize layout assumptions and output formats.

Advanced Features:

• LLM Integration: You can boost accuracy by using OpenAI or Azure LLM services via the --use_llm flag.
• OCR Force: For complex layouts, use the --force_ocr flag to ensure all elements are processed through OCR.

RepoMaster uses an Intelligent Multi-Agent Orchestration system. Instead of manual selection, an AI Intelligent Dispatcher analyzes your natural language task and routes it to a combination of specialized agents:

• Deep Search Agent: Handles advanced web research, data retrieval, and information synthesis.
• Programming Assistant Agent: Handles intelligent code generation, algorithm implementation, and debugging.
• Repository Exploration Agent: Handles autonomous code exploration, repository understanding, and multi-repo coordination.

The dispatcher automatically combines these agents to deliver a complete solution based on task complexity.

If you intend to contribute to the codebase, follow these steps to set up your development environment:

1. Fork and clone the repository.
2. Install development dependencies using pip install -e ".[dev]".
3. Install pre-commit hooks.
4. Run tests using pytest to ensure the environment is stable.

git clone https://github.com/your-username/RepoMaster.git
cd RepoMaster

# Install dev dependencies
pip install -e ".[dev]"

# Set up pre-commit hooks
pre-commit install

# Run tests
pytest tests/

Docling is a tool for advanced PDF understanding (layout, reading order, tables, formulas, etc.) and converting documents into formats like Markdown or JSON. It works on macOS, Linux, and Windows (x86_64 and arm64).

from docling.document_converter import DocumentConverter

# source can be a local path or a URL
source = "https://arxiv.org/pdf/2508.13167"
converter = DocumentConverter()
result = converter.convert(source)
print(result.document.export_to_markdown())

For development, testing, or specialized workflows, you can bypass the dispatcher and access specific agents directly via the backend mode.

# Deep Search Agent
python launcher.py --mode backend --backend-mode deepsearch

# Programming Assistant Agent
python launcher.py --mode backend --backend-mode general_assistant

# Repository Exploration Agent
python launcher.py --mode backend --backend-mode repository_agent

RepoMaster requires API keys for AI providers to function. You must configure the following environment variables depending on your chosen provider:

Primary AI Provider (Required)

Set the key for your main provider to enable core functionality.

External Services (Required for deep search)

Certain deep search capabilities require specific external service integrations.

Optional AI Providers

You can add support for additional models by setting their respective keys:

• ANTHROPIC_API_KEY: For Anthropic Claude support.
• DEEPSEEK_API_KEY: For DeepSeek integration.
• GEMINI_API_KEY: For Google Gemini support.

# Example environment variable configuration
ANTHROPIC_API_KEY=your_claude_key
DEEPSEEK_API_KEY=your_deepseek_key
GEMINI_API_KEY=your_gemini_key

When tasked with parsing content from arXiv PDF files, several specialized GitHub repositories can be used depending on the required level of complexity and output format:

1. dsdanielpark/arxiv2text: A Python package specifically designed to convert arXiv documents into structured text using their PDF URLs.
2. datalab-to/marker: A pipeline for converting PDFs to Markdown and JSON with high accuracy, supporting structured extraction and high configurability.
3. docling-project/docling: Provides advanced PDF understanding and seamless integration with AI models for diverse format parsing.
4. opendatalab/PDF-Extract-Kit: A comprehensive toolkit supporting high-quality extraction tasks like table recognition and reading order.
5. karpathy/arxiv-sanity-preserver: Offers a web interface for browsing and tools for downloading/parsing arXiv PDFs to text.

You can launch an interactive web dashboard for visual multi-agent interaction. This interface supports multi-user sessions and real-time agent collaboration visualization.

Default Access: http://localhost:8501

Configuring Upload Limits: You can increase the default file upload limit (200MB) using the --max-upload-size flag (value in MB).

# Standard launch
python launcher.py --mode frontend

# Launch with a 500MB upload limit
python launcher.py --mode frontend --max-upload-size 500

To install RepoMaster, clone the repository and install the required Python dependencies using pip.

Prerequisites:

• Python 3.11+
• Git
• Internet connection for repository cloning

git clone https://github.com/QuantaAlpha/RepoMaster.git
cd RepoMaster
pip install -r requirements.txt

RepoMaster requires an environment configuration file for API keys. Copy the example configuration to .env and edit it with your credentials.

Required Keys:

• OPENAI_API_KEY: Your OpenAI API key.
• OPENAI_MODEL: The model to use (e.g., gpt-5).
• SERPER_API_KEY: Required for Google search integration (deep search functionality).
• JINA_API_KEY: Required for web content extraction (deep search functionality).

Optional Keys:

• ANTHROPIC_API_KEY: For Anthropic Claude support.
• DEEPSEEK_API_KEY: For DeepSeek integration.
• GEMINI_API_KEY: For Google Gemini support.

cp configs/env.example configs/.env
nano configs/.env

To use Docling, install it using your package manager:

pip install docling