DOCSEARCHbyAlgolia
Submit Docs

llmstxt-generator

repository·main·Indexed 19 days ago

https://github.com/firecrawl/llmstxt-generator

A tool that crawls websites using Firecrawl and processes content with GPT-4-mini to create consolidated llms.txt and llms-full.txt files optimized for LLM training and inference. Available via API endpoint, web interface, or local development setup. Note: This API is deprecated and will not be maintained after June 30, 2025.

Tokens
359
Snippets
2
Records
4
Agent score
18%

What's inside llmstxt-generator

  2. Set up llms.txt Generator for local development

    main

    To run the generator locally, follow these steps:

    1. Configure Environment Variables

    Create a .env file in the root directory and populate the following keys:

    • FIRECRAWL_API_KEY
    • SUPABASE_URL
    • SUPABASE_KEY
    • OPENAI_API_KEY

    2. Install and Run

    Execute the following commands to install dependencies and start the development server:

    npm install
npm run dev

  4. Use the llms.txt Generator API

    main

    You can generate consolidated text files from websites for LLM training and inference by calling the API endpoint. The generator crawls websites and produces both standard (llms.txt) and full (llms-full.txt) versions. Note that processing can take several minutes because it involves web crawling and LLM-based text processing.

    Endpoint Pattern: GET https://llmstxt.firecrawl.dev/[YOUR_URL_HERE]

    GET https://llmstxt.firecrawl.dev/https://example.com