DOCSEARCHbyAlgolia
Submit Docs

llms-txt

repository·main·Indexed 25 days ago

https://github.com/answerdotai/llms-txt

A Python module and CLI tool for parsing and converting llms.txt files into structured data, HTML, or XML formats. It helps website owners provide documentation as context for Large Language Models by extracting project titles, summaries, and linked resources into formats suitable for LLM systems like Claude.

Tokens
2.8K
Snippets
5
Records
21
Agent score
81%

What's inside llms-txt

  1. Conceptual workflow for LLM-enhanced editors using llms.txt

    main

    While the llms.txt proposal is intended for various editors and IDEs (like VS Code, Cursor, Vim, or Emacs), the conceptual workflow for interacting with an llms.txt file follows a specific pattern:

    1. Load Context: The editor retrieves the /llms.txt file from a specified URL, parses it, and fetches the relevant URLs defined in the file's sections (e.g., 'Docs' or 'Examples').
    2. Create Context: The editor converts the fetched information into a structured format (such as XML) suitable for an LLM (e.g., Claude).
    3. Execute Prompt: The user provides a natural language instruction (a prompt). The editor combines this prompt with the loaded context to generate code or content directly into the editor's buffer.
    4. Review and Save: The user reviews the generated output in the buffer and saves it to a file.

  2. Best practices for constructing llms.txt

    main

    When creating an llms.txt file for your domain, follow these guidelines to ensure LLMs can interface with your site effectively:

    • Use concise, clear language.
    • Provide informative descriptions: When linking to resources, include brief descriptions of what the link contains.
    • Avoid ambiguity: Do not use ambiguous terms or unexplained jargon.
    • Test your implementation: Use an LLM to query your llms.txt to see if it can answer questions about your site accurately.

  3. Test your llms.txt with Anthropic's Claude

    main

    To verify if your llms.txt provides sufficient context for an LLM, you can use a Python script that fetches your file, converts it into a context object using llms-txt, and starts an interactive chat session with Claude.

    This script uses the uv script syntax. If you have uv installed, you can run it directly with uv run <filename>.py to automatically manage dependencies in an isolated environment. Otherwise, install the dependencies manually and run it with python <filename>.py.

    # /// script
# requires-python = ">=3.8"
# dependencies = [
#     "claudette",
#     "llms-txt",
#     "requests",
# ]
# ///
from claudette import *
from llms_txt import create_ctx

import requests

model = models[1] # Sonnet 3.5
chat = Chat(model, sp="""You are a helpful and concise assistant."
")

url = 'your_url/llms.txt'
text = requests.get(url).text
llm_ctx = create_ctx(text)
chat(llm_ctx + '\n\nThe above is necessary context for the conversation.')

while True:
    msg = input('Your question about the site: ')
    res = chat(msg)
    print('From Claude:', contents(res))

  4. Understand the llms.txt file specification

    main

    An llms.txt file is a markdown file located at the root (or a subpath) of a website designed to provide context to LLMs. It must follow a specific structure:

    1. H1 Header: The name of the project or site (Required).
    2. Blockquote: A short summary of the project (Required).
    3. Information Sections: Zero or more markdown sections (paragraphs, lists, etc.) providing detailed project info. These must not contain headings.
    4. File Lists (H2 Sections): Zero or more sections delimited by ## headers. Each section contains a markdown list of hyperlinks in the format [name](url): optional notes.

  6. Example llms.txt for a Restaurant domain

    main

    This example demonstrates how a restaurant can structure its llms.txt to provide essential information like business descriptions, operating hours, and links to menus with descriptions.

    # Nate the Great's Grill

> Nate the Great's Grill is a popular destination off of Sesame Street that has been serving the community for over 20 years. We offer the best BBQ for a great price.

Here are our weekly hours:

- Monday - Friday: 9am - 9pm
- Saturday: 11am - 9pm
- Sunday: Closed

## Menus

- [Lunch Menu](https://host/lunch.html.md): Our lunch menu served from 11am to 4pm every day.
- [Dinner Menu](https://host/dinner.html.md): Our dinner menu served from 4pm to 9pm every day.

## Optional

- [Dessert Menu](https://host/dessert.md): A subset of the Starlette docs

  8. Convert llms.txt to XML context

    main

    To provide context to LLMs that prefer XML (like Claude), use the create_ctx function. This function parses the llms.txt content, fetches the content of all linked URLs in parallel, and wraps them in a Project structure containing Section and Doc elements.

    Parameters:

    • txt: The raw text of the llms.txt file.
    • optional (bool): If False, the 'Optional' section (if present in the file) will be included. Defaults to True.
    • n_workers (int): The number of threads to use for parallel downloading of the linked documents.

  9. Convert llms.txt to XML context with create_ctx

    main

    The create_ctx(txt, optional=False, n_workers=None) function parses an llms.txt string and returns a serialized XML representation of a Project.

    • txt: The raw string content of the llms.txt file.
    • optional: If False, the 'Optional' section (H2 header) will be included in the output.
    • n_workers: The number of threads to use for parallel downloading of the linked documentation content.

    This is useful for generating structured context for LLMs from a standard llms.txt file.

  12. Parse an llms.txt file with `parse_llms_file`

    main

    Use parse_llms_file(txt) to convert the raw text of an llms.txt file into a structured AttrDict object. This object allows you to access the project title, summary, info, and the parsed sections (which contain lists of link dictionaries) using attribute access.

    Example of accessing parsed data:

    llmsd = parse_llms_file(txt_content)
print(llmsd.title)
print(llmsd.summary)
print(llmsd.sections.Examples) # Accessing a specific H2 section
    llmsd = parse_llms_file(samp)
llmsd.summary
llmsd.sections.Examples