DOCSEARCHbyAlgolia
Submit Docs

Nougat

repository·main·Indexed 27 days ago

https://github.com/facebookresearch/nougat

A neural optical understanding tool designed to parse academic PDF documents into Markdown, with specific support for LaTeX math and tables. Nougat provides a command-line interface for converting PDFs to .mmd files, a FastAPI-based API server for remote predictions, and tools for dataset generation, model training, and evaluation.

Tokens
2.6K
Snippets
7
Records
16
Agent score
94%

What's inside Nougat

  1. Understand the Creative Commons Attribution-NonCommercial 4.0 License for Nougat Models

    main

    The models in this repository are licensed under the Creative Commons Attribution-NonCommercial 4.0 International Public License.

    Key Usage Terms:

    • NonCommercial Use Only: You may reproduce, share, or create adapted material (derivatives) only for non-commercial purposes. "NonCommercial" is defined as not being primarily intended for or directed towards commercial advantage or monetary compensation.
    • Attribution Required: If you share the material or adapted versions, you must:
      • Retain identification of the creator(s) and any copyright notices.
      • Provide a notice referring to this Public License and the disclaimer of warranties.
      • Provide a URI or hyperlink to the Licensed Material where reasonably practicable.
      • Indicate if you modified the material and retain indications of previous modifications.
    • No Endorsement: Use of the material does not imply endorsement by the Licensor.
    • No Warranty: The material is provided "as-is" without warranties of any kind. The Licensor is not liable for any damages arising from the use of the material.

  2. Install Nougat

    main

    You can install Nougat via pip from PyPI or directly from the GitHub repository.

    Note for Windows users: If you intend to use a GPU, ensure you have installed the correct PyTorch version following the official PyTorch instructions.

    To include extra dependencies for using the model via an API or for dataset generation, use the optional installation extras.

  3. Generate a dataset for training

    main

    To generate a dataset, you need a directory of PDFs, a directory of .html files (processed .tex files via LaTeXML) with matching structures, and a pdffigures2 binary with the PDFFIGURES_PATH environment variable set.

    1. Split HTMLs into pages.
    2. Create a JSONL index.
    3. Generate seek maps for faster loading.

  4. Run the Nougat Docker container

    main

    Run the container with GPU support enabled. You must map a host port to the container's port 8503 to access the API server. Use the --gpus all flag to allow the container to access your NVIDIA hardware.

    docker run -it -d -p <your-port>:8503 --gpus all <image-name>

  5. Prerequisites for Docker deployment

    main

    Before deploying Nougat via Docker, ensure your machine has the following installed:

    • Docker
    • NVIDIA CUDA and CuDNN

    You must verify your CUDA version using nvcc -V and ensure the base image name and PyTorch version in the Docker configuration are compatible with your specific CUDA version.

    nvcc -V

  6. Build the Nougat Docker image

    main

    Clone the repository and navigate to the nougat/docker directory. Build the image using the docker build command. Note that the image size is approximately 17GB and the process may take a significant amount of time as it pulls CUDA-capable images and installs libraries.

    docker build -t <image-name> .

  7. Train and Evaluate Nougat models

    main

    Use the provided scripts to train/fine-tune models or evaluate existing checkpoints.

    Training: Run train.py with a configuration YAML file.

    Evaluation:

    1. Run test.py to generate results in a JSON file.
    2. Run nougat.metrics to calculate results for different text modalities.
    # Training
python train.py --config config/train_nougat.yaml

# Evaluation
python test.py --checkpoint path/to/checkpoint --dataset path/to/test.jsonl --save_path path/to/results.json

# Calculate metrics
python -m nougat.metrics path/to/results.json

  9. Test the Nougat API server connection

    main
    After starting the container, verify the API server is reachable. Note that the server may take a while to respond initially because it must download the Nougat model at startup. A successful connection returns a JSON object with status-code: 200.

  10. Use the Nougat API

    main

    After installing with pip install "nougat-ocr[api]", you can start a local API server using the nougat_api command.

    Predictions are obtained by making a POST request to http://127.0.0.1:8503/predict/. The response is a string containing the markdown text of the document. You can limit processing to specific pages using the start and stop query parameters (boundaries are included).

    # Start the API server
nougat_api

# Send a POST request to predict a PDF
curl -X 'POST' \
  'http://127.0.0.1:8503/predict/' \
  -H 'accept: application/json' \
  -H 'Content-Type: multipart/form-data' \
  -F 'file=@<PDFFILE.pdf>;type=application/pdf'

# Predict specific pages (e.g., pages 1 to 5)
curl -X 'POST' \
  'http://127.0.0.1:8503/predict/?start=1&stop=5' \
  -H 'accept: application/json' \
  -H 'Content-Type: multipart/form-data' \
  -F 'file=@<PDFFILE.pdf>;type=application/pdf'

  11. Get PDF predictions via the API server

    main

    To convert a PDF to markdown, send a POST request to the /predict/ endpoint.

    Endpoint: http://127.0.0.1:<your-port>/predict/

    Parameters:

    • start (query param): The starting page number (inclusive).
    • stop (query param): The ending page number (inclusive).

    Response: A string containing the markdown text of the document.

    Example: Full document conversion

    curl -X 'POST' \
  'http://127.0.0.1:<your-port>/predict/' \
  -H 'accept: application/json' \
  -H 'Content-Type: multipart/form-data' \
  -F 'file=@<PDFFILE.pdf>;type=application/pdf'

    Example: Conversion for specific pages (e.g., pages 1 to 5)

    curl -X 'POST' \
  'http://127.0.0.1:<your-port>/predict/?start=1&stop=5' \
  -H 'accept: application/json' \
  -H 'Content-Type: multipart/form-data' \
  -F 'file=@<PDFFILE.pdf>;type=application/pdf'

  12. Configure Nougat API via environment variables

    main

    The FastAPI application uses the following environment variables for configuration:

    • NOUGAT_CHECKPOINT: Required. The file path to the model checkpoint. If not set, the application will exit.
    • NOUGAT_BATCHSIZE: The batch size used during model inference. Defaults to the value returned by nougat.utils.device.default_batch_size().