arxiv2md

repository·main·Indexed 41 days ago

https://github.com/timf34/arxiv2md

A tool to convert arXiv papers into clean Markdown by parsing structured HTML. It provides a Python library (version 0.1.0), a CLI, a REST API, and a Web App, making it suitable for researchers, AI agents, and LLM prompting workflows.

Tokens
7.4K
Snippets
30
Records
36
Agent score
52%

What's inside arxiv2md

  1. Install and use the arxiv2md CLI

    main

    Install the CLI tool via pip:

    pip install arxiv2markdown

    Common CLI commands:

    • Output to file: arxiv2md <ID> -o <filename>.md
    • Output to stdout: arxiv2md <ID> -o -
    • Include YAML frontmatter: arxiv2md <ID> --frontmatter -o <filename>.md
    • Strip references and TOC: arxiv2md <ID> --remove-refs --remove-toc -o -
    • Filter specific sections: Use --section-filter-mode (e.g., include) and --sections (comma-separated list).
    arxiv2md 2501.11120v1 --frontmatter -o paper.md
  2. Get raw Markdown via REST API

    main

    Use the /api/markdown endpoint to retrieve a paper's content as raw Markdown text. This is useful for LLM processing or direct file conversion.

    curl "https://arxiv2md.org/api/markdown?url=2501.11120"
  3. Get JSON metadata and content via REST API

    main

    Use the /api/json endpoint to retrieve a structured JSON object containing the paper's metadata and content.

    Response Schema:

    {
      "arxiv_id": "...",
      "title": "...",
      "source_url": "...",
      "content": "..."
    }
    curl "https://arxiv2md.org/api/json?url=2501.11120"
  4. Use the arxiv2md Python library

    main

    You can integrate arXiv to Markdown conversion directly into your Python code using either synchronous or asynchronous functions. Both functions accept the same keyword arguments for filtering and metadata.

    Keyword Arguments:

    ArgumentDefaultDescription
    remove_refsTrueRemove bibliography/references sections
    remove_tocTrueRemove table of contents
    remove_inline_citationsTrueRemove inline citation text
    section_filter_mode"exclude""include" or "exclude" for section filtering
    sectionsNone (all)List of section titles to include/exclude
    include_frontmatterFalsePrepend YAML frontmatter with paper metadata
    from arxiv2md import ingest_paper_sync
    
    # Synchronous usage
    result = ingest_paper_sync("2501.11120v1")
    print(result.content)
    
    # or use the async version
    from arxiv2md import ingest_paper
    
    result = await ingest_paper("2501.11120v1")
  5. Use the arxiv2md REST API

    main

    The REST API provides two GET endpoints that require no authentication. It is suitable for AI agents and LLM workflows.

    Rate limit: 30 requests/minute per IP.

    Query Parameters:

    ParamDefaultDescription
    urlrequiredarXiv URL or ID
    remove_refstrueRemove references
    remove_toctrueRemove table of contents
    remove_citationstrueRemove inline citations
    frontmatterfalsePrepend YAML frontmatter (/api/markdown only)

    Endpoints:

    • https://arxiv2md.org/api/json?url={url}: Returns a JSON response containing the paper content and metadata.
    • https://arxiv2md.org/api/markdown?url={url}: Returns the raw Markdown content.
    # JSON response (with metadata)
    curl "https://arxiv2md.org/api/json?url=2312.00752"
    
    # Raw markdown
    curl "https://arxiv2md.org/api/markdown?url=2312.00752"
  6. Configure arxiv2md via Docker Compose

    main

    You can deploy arxiv2md using Docker Compose. By default, the service maps host port 8001 to container port 8000.

    Environment Variables

    Use the following environment variables to configure the service behavior:

    • HOST: The interface to bind to (default 0.0.0.0).
    • PORT: The internal port the application listens on (default 8000).
    • ARXIV2MD_CACHE_PATH: The directory path where the cache is stored (default /app/.arxiv2md_cache).
    • ARXIV2MD_CACHE_TTL_SECONDS: The Time-To-Live for cached items in seconds (default 86400).
    • APP_VERSION: The version string for the application.
    • APP_REPOSITORY: The URL of the application repository.

    Volumes and Persistence

    • Persistent Cache: A named volume arxiv2md_cache is used to persist the cache across container restarts.
    • Development Mode: To use the service for development, you can uncomment the volume mount ./src:/app/src to sync your local source code with the container.
    services:
      arxiv2md:
        build:
          context: .
          dockerfile: Dockerfile
        container_name: arxiv2md
        restart: unless-stopped
        ports:
          - "8001:8000"
        environment:
          - HOST=0.0.0.0
          - PORT=8000
          - ARXIV2MD_CACHE_PATH=/app/.arxiv2md_cache
          - ARXIV2MD_CACHE_TTL_SECONDS=86400
          - APP_VERSION=0.1.0
          - APP_REPOSITORY=https://github.com/timf34/arxiv2md
        volumes:
          - arxiv2md_cache:/app/.arxiv2md_cache
          # - ./src:/app/src
        networks:
          - arxiv2md_network
  7. Configure arxiv2md REST API parameters

    main

    Both /api/markdown and /api/json endpoints support the following optional query parameters:

    ParamDefaultDescription
    urlrequiredarXiv URL or ID (e.g. 2501.11120v1 or https://arxiv.org/abs/2501.11120)
    remove_refstrueRemove references section
    remove_toctrueRemove table of contents
    remove_citationstrueRemove inline citations
    frontmatterfalsePrepend YAML metadata (/api/markdown only)
  8. Use the arxiv2md CLI

    main

    The CLI allows you to convert arXiv papers to Markdown directly from your terminal. You can provide an arXiv ID or a full URL. Use the -o flag to specify an output file, or -o - to output to stdout.

    # Basic usage
    arxiv2md 2501.11120v1 -o paper.md
    
    # Only extract specific sections
    arxiv2md 2501.11120v1 --section-filter-mode include --sections "Abstract,Introduction" -o -
    
    # Strip references and TOC
    arxiv2md 2501.11120v1 --remove-refs --remove-toc -o -
    
    # Include YAML frontmatter with paper metadata
    arxiv2md 2501.11120v1 --frontmatter -o paper.md
  9. Ingest arXiv papers asynchronously with ingest_paper()

    main

    Use ingest_paper to fetch, parse, and serialize an arXiv paper into Markdown within an async context.

    Parameters:

    • arxiv_id (str): The arXiv ID (e.g., '2501.11120v1') or URL (e.g., 'https://arxiv.org/abs/2501.11120').
    • remove_refs (bool): If True, removes the bibliography/references sections. Defaults to True.
    • remove_toc (bool): If True, removes the table of contents. Defaults to True.
    • remove_inline_citations (bool): If True, removes inline citation text. Defaults to True.
    • section_filter_mode (Literal['include', 'exclude']): Determines how the sections list is applied. Defaults to 'exclude'.
    • sections (list[str] | None): A list of section titles. If section_filter_mode is 'include', only these sections are kept. If 'exclude', these sections are removed. Defaults to None (all sections).
    • include_frontmatter (bool): If True, prepends YAML frontmatter containing paper metadata. Defaults to False.

    Returns: An IngestionResult object containing .content (Markdown), .summary, .sections_tree, and .frontmatter.

    import asyncio
    from arxiv2md import ingest_paper
    
    async def main():
        result = await ingest_paper(
            "2501.11120",
            remove_refs=True,
            include_frontmatter=True,
            section_filter_mode="include",
            sections=["Introduction", "Conclusion"]
        )
        print(result.content)
    
    asyncio.run(main())
  10. Ingest arXiv papers synchronously with ingest_paper_sync()

    main

    Use ingest_paper_sync when you need to ingest a paper in a standard synchronous Python environment where an event loop is not already running. It behaves identically to ingest_paper but handles the underlying asyncio execution for you.

    Note: Calling this function from within an existing running event loop will raise a RuntimeError. In such cases, you must use await ingest_paper(...) instead.

    from arxiv2md import ingest_paper_sync
    
    result = ingest_paper_sync(
        "2501.11120",
        include_frontmatter=True
    )
    print(result.content)
  11. Configure the /api/ingest endpoint request

    main

    The /api/ingest endpoint accepts an IngestRequest JSON body. This allows you to specify an arXiv URL or ID and apply various transformations to the resulting markdown content.

    Request Fields

    • input_text (string, required): The arXiv URL or ID to ingest.
    • remove_refs (boolean): If true, removes references from the output.
    • remove_toc (boolean): If true, removes the table of contents.
    • remove_inline_citations (boolean): If true, removes inline citations and internal paper links.
    • include_frontmatter (boolean): If true, prepends YAML frontmatter containing paper metadata.
    • section_filter_mode (string): Determines how the sections list is applied. Use include or exclude.
    • sections (array of strings): A list of section titles to filter. If provided as a comma-separated string, it will be automatically normalized into a list.

    Section Filter Modes

    • include: Only include the specified sections.
    • exclude: Exclude the specified sections (default).
    {
      "input_text": "https://arxiv.org/abs/2301.00001",
      "remove_refs": true,
      "remove_toc": true,
      "section_filter_mode": "exclude",
      "sections": ["Introduction", "Conclusion"]
    }