DOCSEARCHbyAlgolia
Submit Docs

llm-scraper

repository·main·Indexed 27 days ago

https://github.com/mishushakov/llm-scraper

A TypeScript library that uses LLMs and Playwright to extract structured data from webpages using Zod or JSON schemas. It supports multiple LLM providers via the Vercel AI SDK, including OpenAI, Anthropic, Google, Groq, and Ollama. Key features include data extraction via run(), real-time streaming with stream(), and the ability to generate reusable Playwright scraping scripts using generate().

Tokens
2.6K
Snippets
5
Records
17
Agent score
89%

What's inside llm-scraper

  1. Initialize LLM providers

    main

    LLM Scraper uses the Vercel AI SDK. You must install the specific provider package for the model you intend to use and initialize the LLM instance.

    OpenAI

    Install: npm i @ai-sdk/openai

    import { openai } from '@ai-sdk/openai'
const llm = openai('gpt-4o')

    Anthropic

    Install: npm i @ai-sdk/anthropic

    import { anthropic } from '@ai-sdk/anthropic'
const llm = anthropic('claude-3-5-sonnet-20240620')

    Google

    Install: npm i @ai-sdk/google

    import { google } from '@ai-sdk/google'
const llm = google('gemini-1.5-flash')

    Groq

    Install: npm i @ai-sdk/openai

    import { createOpenAI } from '@ai-sdk/openai'
const groq = createOpenAI({
  baseURL: 'https://api.groq.com/openai/v1',
  apiKey: process.env.GROQ_API_KEY,
})
const llm = groq('llama3-8b-8192')

    Ollama

    Install: npm i ollama-ai-provider-v2

    import { ollama } from 'ollama-ai-provider-v2'
const llm = ollama('llama3')

  3. Extract structured data with scraper.run()

    main

    Use the run method to extract data from a Playwright page object based on a Zod or JSON Schema. You can specify a format to control how the page content is loaded into the LLM.

    Available formatting modes:

    • html: loading pre-processed HTML
    • raw_html: loading raw HTML (no processing)
    • markdown: loading markdown
    • text: loading extracted text (using Readability.js)
    • image: loading a screenshot (multi-modal only)
    • custom: loading custom content (using a custom function)
    import { chromium } from 'playwright'
import { z } from 'zod'
import { Output } from 'ai'
import { openai } from '@ai-sdk/openai'
import LLMScraper from 'llm-scraper'

const browser = await chromium.launch()
const llm = openai('gpt-4o')
const scraper = new LLMScraper(llm)
const page = await browser.newPage()
await page.goto('https://news.ycombinator.com')

const schema = z.object({
  top: z.array(z.object({
    title: z.string(),
    points: z.number(),
    by: z.string(),
    commentsURL: z.string(),
  })).length(5).describe('Top 5 stories on Hacker News'),
})

const { data } = await scraper.run(page, Output.object({ schema }), {
  format: 'html',
})

console.log(data.top)

await page.close()
await browser.close()
    import { chromium } from 'playwright'
import { z } from 'zod'
import { Output } from 'ai'
import { openai } from '@ai-sdk/openai'
import LLMScraper from 'llm-scraper'

// Launch a browser instance
const browser = await chromium.launch()

// Initialize LLM provider
const llm = openai('gpt-4o')

// Create a new LLMScraper
const scraper = new LLMScraper(llm)

// Open new page
const page = await browser.newPage()
await page.goto('https://news.ycombinator.com')

// Define schema to extract contents into
const schema = z.object({
  top: z
    .array(
      z.object({
        title: z.string(),
        points: z.number(),
        by: z.string(),
        commentsURL: z.string(),
      })
    )
    .length(5)
    .describe('Top 5 stories on Hacker News'),
})

// Run the scraper
const { data } = await scraper.run(page, Output.object({ schema }), {
  format: 'html',
})

// Show the result from LLM
console.log(data.top)

await page.close()
await browser.close()

  4. Generate reusable Playwright scripts with scraper.generate()

    main

    The generate function creates a reusable Playwright script that can scrape content according to a specific schema. This is useful for automating repetitive scraping tasks without calling the LLM every time.

    // Generate code and run it on the page
const { code } = await scraper.generate(page, Output.object({ schema }))
const result = await page.evaluate(code)
const data = schema.parse(result)

// Show the parsed result
console.log(data.top)

  5. Stream structured data with scraper.stream()

    main

    To receive a partial object stream instead of waiting for the full extraction, use the stream method. This returns an object containing an async iterator.

    // Run the scraper in streaming mode
const { stream } = await scraper.stream(page, Output.object({ schema }))

// Stream the result from LLM
for await (const data of stream) {
  console.log(data.top)
}

  6. Create an LLMScraper instance

    main

    Once you have initialized your LLM provider, pass it to the LLMScraper constructor.

    import LLMScraper from 'llm-scraper'
const scraper = new LLMScraper(llm)
    import LLMScraper from 'llm-scraper'

const scraper = new LLMScraper(llm)

  7. Configure ScraperGenerateOptions

    main

    When using the generate method to create scraping code, you can pass ScraperGenerateOptions. This type omits the mode property from ScraperLLMOptions and adds formatting controls.

    Keys:

    • format: Specifies the output format. Supported values: 'html' | 'raw_html'.
    • system: System prompt for code generation.
    • messages: Custom message history.

  8. Configure ScraperLLMOptions

    main

    When calling run, stream, or generate, you can provide ScraperLLMOptions to control the LLM behavior. This type extends CallSettings from the ai SDK.

    Keys:

    • system: A string representing the system prompt.
    • messages: An array of ModelMessage[] for multi-turn or specific message structuring.
    • All standard CallSettings properties (e.g., temperature, maxTokens).

  9. Generate completions with generateAISDKCompletions

    main

    Use generateAISDKCompletions to perform a single-shot extraction of data from a pre-processed webpage using the AI SDK. It accepts a LanguageModel, a PreProcessResult (containing the page content and URL), an output object (defining the schema), and optional ScraperLLMOptions.

    Returns an object containing the extracted data and the source url.

  11. Run data extraction with run()

    main

    The run method extracts structured data from a Playwright Page. It first preprocesses the page content and then uses the LLM to generate completions based on the provided output schema.

    Parameters:

    • page: A Playwright Page instance.
    • output: An Output object defining the desired schema.
    • options: ScraperRunOptions (includes ScraperLLMOptions and PreProcessOptions).

  12. Stream data extraction with stream()

    main

    The stream method performs the same task as run but returns a stream of completions instead of a single completed object. This is useful for real-time UI updates or handling large extractions.

    Parameters:

    • page: A Playwright Page instance.
    • output: An Output object defining the desired schema.
    • options: ScraperRunOptions.