DOCSEARCHbyAlgolia
Submit Docs

Jina Reader

repository·main·Indexed 9 days ago

https://github.com/jina-ai/reader

A tool that converts web content—including URLs, PDFs, MS Office documents, and images—into LLM-friendly markdown. It provides a read interface via r.jina.ai and a search-to-markdown interface via s.jina.ai to give LLMs access to real-time web knowledge. Version 0.5.0 supports headless Chrome rendering, VLM image captioning, and customizable output formats via HTTP headers.

Tokens
16.8K
Snippets
62
Records
79
Agent score
95%

What's inside Jina Reader

  1. Overview of Reader capabilities

    main

    Reader is a tool designed to optimize web content for Large Language Models (LLMs). It offers two primary functions:

    1. Read: Converts a specific URL to markdown using https://r.jina.ai/.
    2. Search: Searches the web for a query and returns markdown results using https://s.jina.ai/.

    The service is designed to be production-ready, stable, and scalable. The open-source version of this repository runs in a stateless mode, whereas the SaaS version uses a MongoDB-backed storage layer.

  2. Fetch Single Page Applications (SPAs)

    main

    Reader supports SPAs via headless Chrome, but specific routing or loading patterns may require different approaches:

    Hash-based routing: If the URL contains a # (which is not sent to the server), use a POST request with the url parameter in the body:

    curl -X POST 'https://r.jina.ai/' -d 'url=https://example.com/#/route'

    Preloading/Dynamic content: If the page shows preload content before the main content loads, use x-timeout or x-wait-for-selector to ensure the dynamic content is captured:

    # Wait for network idle or until timeout
curl 'https://r.jina.ai/https://example.com/' -H 'x-timeout: 10'

# Wait for a specific element
curl 'https://r.jina.ai/https://example.com/' -H 'x-wait-for-selector: #content'
    curl -X POST 'https://r.jina.ai/' -d 'url=https://example.com/#/route'

  3. Search the web for LLM input with s.jina.ai

    main

    Reader includes a search capability that allows LLMs to access the latest world knowledge. By using the s.jina.ai endpoint, you can perform a web search and receive the results formatted as markdown. To use it, append your search query to the base URL.

    Example usage: https://s.jina.ai/your+query

    https://s.jina.ai/Who%20will%20win%202024%20US%20presidential%20election%3F

  4. Perform geo- and locale-sensitive scraping

    main

    To bypass regional restrictions or scrape localized content, you can pin geography and language using these headers (requires a premium key):

    • x-proxy: Sets the exit IP (e.g., x-proxy: de for Germany).
    • x-locale: Sets navigator.language and Accept-Language (e.g., x-locale: de-DE).
    • x-set-cookie: Sets specific cookies to bypass gating (e.g., x-set-cookie: country=DE; Path=/).

    Warning: Any request using x-set-cookie will skip the cache.

    curl https://r.jina.ai/https://shop.example.com/product/123 \
  -H 'x-proxy: de' \
  -H 'x-locale: de-DE' \
  -H 'x-set-cookie: country=DE; Path=/'

  5. Inject JavaScript to reveal hidden content

    main

    To interact with pages that hide content behind buttons (e.g., "Show transcript"), use injectPageScript.

    Reader provides a window.waitForSelector(selector) helper that resolves when the element appears. You can use this to click elements before extraction begins. After the click, Reader's mutationIdle logic waits for the DOM to settle before serializing.

    • Multiple steps: Pass multiple -F 'injectPageScript=...' flags to run a sequence of scripts.
    • Iframes: To interact with content inside an iframe (like a Twitter embed), use injectFrameScript instead.
    • Note: Injecting scripts disables early-return optimizations; use x-timeout for slow async actions.
    curl -F 'url=https://www.youtube.com/watch?v=dQw4w9WgXcQ' \
     -F "injectPageScript=waitForSelector('ytd-video-description-transcript-section-renderer button').then((el) => el.click())" \
     -H 'Accept: application/json' \
     https://r.jina.ai/

  6. Convert URLs to LLM-friendly markdown with r.jina.ai

    main

    Reader provides a way to convert any URL into high-quality, LLM-friendly markdown. This is useful for improving the performance of RAG (Retrieval-Augmented Generation) systems and AI agents. To use it, prepend https://r.jina.ai/ to the target URL.

    Reader supports:

    • Web pages: Rendered via headless Chrome or lightweight fetching.
    • PDFs: Parsed using PDF.js.
    • MS Office documents: Word, Excel, and PowerPoint files are converted via LibreOffice.
    • Images: Captioned by a vision-language model to provide text descriptions for text-only LLMs.
    https://r.jina.ai/https://your.url

  7. Upload raw HTML content

    main

    You can send existing HTML directly to Reader using the html body field. This bypasses the fetching stage and sends the content directly through the conversion pipeline.

    It is highly recommended to include the url field when sending raw HTML. Reader uses this URL as the base to resolve any relative links or images contained within the HTML snippet.

    curl -X POST 'https://r.jina.ai/' \
  -H 'Content-Type: application/json' \
  -d '{"html": "<html>...</html>", "url": "https://example.com/source"}'

  8. Upload PDF and MS Office files to Reader

    main

    Reader can ingest local files including .pdf, .docx, .xlsx, and .pptx without hosting them first. Use the file body field in a multipart/form-data request. Reader automatically detects the MIME type from the bytes.

    Key details:

    • Caching: File requests are cached by the sha256 hash of the bytes; re-uploading the same file results in a cache hit.
    • Office Fidelity: Office files are processed via LibreOffice. For high-fidelity requirements (e.g., complex Excel layouts or custom PowerPoint masters), it is recommended to export to PDF locally before uploading.
    • Options: All web page options like x-retain-images, x-markdown-chunking, and Accept: application/json are supported for file uploads.
    curl -X POST 'https://r.jina.ai/' \
  -F 'file=@./report.pdf' \
  -H 'Accept: application/json' \
  -H 'x-markdown-chunking: s3'

  9. Set up Local Development Environment

    main

    To develop on Reader locally, ensure you have nvm installed. You may also want Docker if you plan to test with a local MinIO bucket cache.

    1. Clone the repository.
    2. Install dependencies via npm install.
    3. (Optional) Start a local cache using docker compose up -d.
    4. Run the development server using npm run dev (after setting up required environment variables) or use the VSCode debugger.
    git clone git@github.com:jina-ai/reader.git
cd reader
npm install

# Optional: for bucket-cached mode
docker compose up -d

# Start dev server
npm run dev

  10. Use JSON mode for Reader API

    main

    To receive the response in JSON format, set the Accept header to application/json.

    Example:

    curl -H "Accept: application/json" https://r.jina.ai/https://en.m.wikipedia.org/wiki/Main_Page
    curl -H "Accept: application/json" https://r.jina.ai/https://en.m.wikipedia.org/wiki/Main_Page

  11. Perform in-site search with s.jina.ai

    main

    Restrict search results to specific domains by providing the site parameter in the query string. You can specify multiple sites by repeating the parameter.

    Example:

    curl 'https://s.jina.ai/When%20was%20Jina%20AI%20founded%3F?site=jina.ai&site=github.com'
    curl 'https://s.jina.ai/When%20was%20Jina%20AI%20founded%3F?site=jina.ai&site=github.com'

  12. Extract content from Iframes and Shadow DOM

    main

    By default, Reader skips iframes and shadow roots. To include them in the main document extraction:

    • x-with-iframe: true: Includes iframe contents.
    • x-with-iframe: quoted: Wraps iframe contents in a markdown blockquote to distinguish them from the main body.
    • x-with-shadow-dom: true: Includes content residing inside shadow roots (common in modern web component sites).

    Note: Enabling these options forces network-idle timing, making requests slower. Use x-timeout (up to 180s) to manage the wait time.

    curl 'https://r.jina.ai/https://example.com/docs-page' \
  -H 'x-with-iframe: true' \
  -H 'x-with-shadow-dom: true'