DOCSEARCHbyAlgolia
Submit Docs

text-extract-api

repository·main·Indexed 25 days ago

https://github.com/catchthetornado/text-extract-api

A high-accuracy document processing API (v0.2.0) that converts images, PDFs, and Office documents (Word, PPTX) into structured JSON or Markdown. It utilizes OCR strategies including EasyOCR, Llama 3.2 Vision, and MiniCPM-V, with Ollama support for LLM-based text improvement and PII removal. The system features asynchronous task processing via Celery and supports multiple storage profiles including Local File System, Google Drive, and Amazon S3.

Tokens
5.6K
Snippets
16
Records
37
Agent score
85%

What's inside text-extract-api

  1. Install and Setup the CLI Tool

    main

    To use the CLI, first set up a virtual environment (especially on macOS), install the main project requirements, and then install the client package.

    # Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install main project requirements
pip install -e .

# Install client requirements
cd client
pip install -e .
    # Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install main project requirements
pip install -e .

# Install client requirements
cd client
pip install -e .

  2. Manage OCR Results via CLI

    main

    Use the following commands to manage processed files:

    • Get result by ID: python client/cli.py result --task_id {task_id}
    • List files in a profile: python client/cli.py list_files (or --storage_profile gdrive)
    • Load a file: python client/cli.py load_file --file_name "path/to/file.md"
    • Delete a file: python client/cli.py delete_file --file_name "path/to/file.md" (optionally specify --storage_profile)
    • Clear cache: python client/cli.py clear_cache
    python client/cli.py list_files
python client/cli.py load_file --file_name "invoices/2024/example-invoice-2024-10-31-16-33.md"
python client/cli.py delete_file --file_name "invoices/2024/example-invoice-2024-10-31-16-33.md"

  3. Manual setup for local development

    main

    If you prefer not to use the Makefile, follow these steps to set up the environment manually. This is recommended for users wanting to utilize Apple GPUs (which are not supported by Docker).

    Prerequisites:

    • Install Ollama.
    • Install Docker.
    • (macOS only) Install system dependencies: brew update && brew install libmagic poppler pkg-config ghostscript ffmpeg automake autoconf.

    Steps:

    1. Configure environment variables by copying .env.localhost.example to .env.localhost.
    2. Create and activate a virtual environment.
    3. Install the package in editable mode.
    4. Run the setup script.
    5. Crucial: You must start a Celery worker in a separate terminal to process tasks.
    # 1. Configure env
cp .env.localhost.example .env.localhost

# 2. Setup Python env
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
chmod +x run.sh
run.sh

# 3. Start Celery worker (Required for task processing)
celery -A text_extract_api.celery_app worker --loglevel=info --pool=solo

  4. Install and run text-extract-api via Makefile

    main

    The easiest way to set up the application is using the provided Makefile. By default, this creates a virtual Python environment in .venv.

    To install and run without creating a virtual environment, use DISABLE_VENV=1.

    Steps:

    1. Clone the repository and enter the directory.
    2. Run make install to set up dependencies.
    3. Run make run to start the application.
    git clone https://github.com/CatchTheTornado/text-extract-api.git
cd text-extract-api

# Standard setup
make install
make run

# Setup without virtual environment
DISABLE_VENV=1 make install
DISABLE_VENV=1 make run

  5. Pull LLM Models via CLI

    main

    Most features require LLM models. Use the llm_pull command to download models (e.g., Llama 3.1 or Llama 3.2-vision) via the CLI before performing OCR tasks that require LLM processing.

    python client/cli.py llm_pull --model llama3.1
python client/cli.py llm_pull --model llama3.2-vision

  6. Run text-extract-api using Docker Compose

    main

    Use Docker Compose to run the full stack (FastAPI, Celery, Redis, and Ollama) in a containerized environment.

    Steps:

    1. Clone the repository.
    2. Create a .env file from .env.example or .env.example.localhost.
    3. Build and run the containers.

    Note: For GPU support, use the specific GPU docker-compose file. Note that Apple Silicon GPUs are currently not supported via Docker in this project; use the manual setup for Mac users.

    # Standard Docker setup
cp .env.example .env
docker-compose up --build

# GPU enabled setup
docker-compose -f docker-compose.gpu.yml -p text-extract-api-gpu up --build

  7. Scale parallel processing with Celery workers

    main

    To handle multiple OCR tasks concurrently, you must run multiple Celery worker processes. Each worker process handles one task at a time when using the --pool=solo flag.

    Run the following command multiple times to increase the number of concurrent processes:

    celery -A text_extract_api.tasks worker --loglevel=info --pool=solo &

  8. Configure the remote OCR strategy (Marker)

    main

    The remote strategy allows using marker-pdf as an external OCR engine. This is useful for high-accuracy multi-language support.

    Setup Steps:

    1. Install marker-pdf in a separate directory (outside text-extract-api).
    2. Start the marker_server.
    3. Set the REMOTE_API_URL environment variable in the text-extract-api environment to point to your server.

    Example usage via curl:

    curl -X POST -H "Content-Type: multipart/form-data" -F "file=@examples/example-mri.pdf" -F "strategy=remote" -F "ocr_cache=true" -F "prompt=" -F "model=" "http://localhost:8000/ocr/upload"
    # 1. Setup marker-pdf server
mkdir marker-distribution
cd marker-distribution
pip install marker-pdf
pip install -U uvicorn fastapi python-multipart
marker_server --port 8002

# 2. Set environment variable for text-extract-api
export REMOTE_API_URL=http://localhost:8002/marker/upload

# 3. Run text-extract-api
make run

  9. Configure Storage Profiles

    main

    Storage profiles are configured via YAML files in the /storage_profiles directory. Supported strategies include:

    Local File System

    strategy: local_filesystem
settings:
  root_path: /storage
  subfolder_names_format: ""
  create_subfolders: true

    Google Drive

    Requires a Google Service Account JSON file.

    strategy: google_drive
settings:
  service_account_file: /path/to/your/service_account.json
  folder_id: YOUR_FOLDER_ID

    Amazon S3

    Requires environment variables for credentials.

    strategy: aws_s3
settings:
  bucket_name: ${AWS_S3_BUCKET_NAME}
  region: ${AWS_REGION}
  access_key: ${AWS_ACCESS_KEY_ID}
  secret_access_key: ${AWS_SECRET_ACCESS_KEY}
    strategy: aws_s3
settings:
  bucket_name: ${AWS_S3_BUCKET_NAME}
  region: ${AWS_REGION}
  access_key: ${AWS_ACCESS_KEY_ID}
  secret_access_key: ${AWS_SECRET_ACCESS_KEY}

  12. Perform OCR via CLI (Multipart Upload)

    main

    Upload a file for OCR (converting it to Markdown) using the ocr_upload command. This uses multipart/form-data.

    Use --ocr_cache to cache results. You can also provide a --prompt_file for LLM-based transformations and a --language flag (e.g., en,de,pl) to load specific language weights.

    python client/cli.py ocr_upload --file examples/example-mri.pdf --ocr_cache