GPT Crawler

Before running the crawler, you must configure the target site by editing config.ts. You need to specify the starting url, a match pattern for subsequent links, and a CSS selector to extract the relevant text content.

Example configuration for crawling documentation:

export const defaultConfig: Config = {
  url: "https://www.builder.io/c/docs/developers",
  match: "https://www.builder.io/c/docs/**",
  selector: `.docs-builder-container`,
  maxPagesToCrawl: 50,
  outputFileName: "output.json",
};

To run the crawler in a containerized environment:

1. Navigate to the containerapp directory.
2. Modify the config.ts within that directory to set your desired crawl parameters.
3. The resulting output.json will be generated in the data folder.

You can run the crawler as an Express JS API server.

1. Install dependencies: npm install
2. Start the server: npm run start:server. The server runs on port 3000 by default.
3. Configure environment: Copy .env.example to .env to override variables like the port.
4. Use the endpoint: Send a POST request to /crawl with a JSON body containing your Config object.
5. API Documentation: Swagger docs are available at the /api-docs endpoint.

Once the crawl is complete, a file named output.json is generated at the project root. You can upload this file to OpenAI to build custom AI tools.

Create a custom GPT (UI-based)

Best for sharing knowledge via the ChatGPT interface.

1. Go to ChatGPT and select "My GPTs" -> "Create a GPT".
2. Go to the "Configure" tab.
3. Under "Knowledge", select "Upload a file" and upload your output.json.
4. Tip: If the file is too large, use maxFileSize or maxTokens in your config.ts to reduce size before re-crawling.

Create a custom Assistant (API-based)

Best for integrating knowledge into your own applications via the OpenAI API.

1. Go to the OpenAI Assistants dashboard.
2. Click "+ Create".
3. Select "upload" and upload your output.json file.

You can run the GPT Crawler as a containerized application. This method uses a Docker image where all dependencies are pre-configured in the Dockerfile.

Prerequisites

• Docker must be installed on your system.

Execution Steps

1. Navigate to the container application directory:

   cd gpt-crawler/containerapp

2. Execute the provided shell script to build and run the container:

   . ./run.sh

To run the crawler on your local machine, ensure you have Node.js >= 16 installed. Follow these steps:

1. Clone the repository

   git clone https://github.com/builderio/gpt-crawler

2. Install dependencies

   npm i

3. Run the crawler

   npm start

The GPT Crawler CLI allows you to crawl a website based on specific URL patterns and CSS selectors to generate data for LLMs.

You can run the crawler in two ways:

1. Interactive Mode: Run the command without arguments. The CLI will prompt you for the required configuration (URL, match pattern, selector, etc.) using inquirer.
2. Flag Mode: Provide configuration directly via command-line flags to bypass interactive prompts.

Once the crawling process is complete, the CLI automatically writes the results to the specified output file.

# Example using flags
# -u: starting URL
# -m: URL pattern to match
# -s: CSS selector for content
# -p: max pages to crawl
# -o: output filename

gpt-crawler --url https://example.com --match /.*\/blog\/.* --selector article --maxPagesToCrawl 10 --outputFileName my_data.json

The API server's network settings can be controlled using the following environment variables:

• API_PORT: The port on which the server will listen (defaults to 3000).
• API_HOST: The hostname the server will bind to (defaults to localhost).

The application entrypoint provides a way to execute the full crawling and writing lifecycle using the defaultConfig. This involves two sequential steps:

1. crawl(config): Scrapes the target website based on the provided configuration.
2. write(config): Persists the crawled data to the specified output format (e.g., JSON files).

Note that these functions are intended to be called with a configuration object that follows the project's schema.

import { defaultConfig } from "../config.js";
import { crawl, write } from "./core.js";

await crawl(defaultConfig);
await write(defaultConfig);

The API server exposes a /crawl endpoint that accepts a crawler configuration object in the request body. When called, the server validates the configuration, executes the crawling process using GPTCrawlerCore, writes the results to a file, and returns the resulting JSON content in the response.

Request Body: A JSON object matching the Config schema. Response: A JSON object containing the crawled data. Error Handling: Returns a 500 status code with a JSON error message if validation fails or the crawling process encounters an error.

curl -X POST http://localhost:3000/crawl \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "selector": "main",
    "maxDepth": 2
  }'