LibreCrawl Documentation

LibreCrawl operates in two primary modes depending on your deployment needs:

Standard Mode

• Use Case: Public-facing demos or shared hosting.
• Features: Full authentication (login/register) and a tier-based access system (Guest, User, Extra, Admin).
• Restrictions: Guest users are limited to 3 crawls per 24 hours based on IP.

Local Mode

• Use Case: Personal use, single-user self-hosting, or local development.
• Features: All users are automatically granted Admin tier access.
• Restrictions: No rate limits or tier restrictions applied.

LibreCrawl plugins are standalone .js files dropped into the plugins folder that register themselves via the LibreCrawlPlugin API. Each plugin creates a new tab in the interface.

Plugins operate based on Lifecycle Hooks that respond to user actions (like activating a tab) or crawl events (like data updates or crawl completion). They receive a data object containing the results of the current crawl, allowing for custom analysis, visualization, or reporting.

LibreCrawlPlugin.register({
  id: 'my-plugin',
  name: 'My Plugin',
  tab: {
    label: 'My Tab',
    icon: '🔥',
  },
  onTabActivate(container, data) {
    container.innerHTML = `<div>${data.urls.length} URLs found</div>`;
  }
});

For controlled environments, use Docker and Docker Compose.

1. Clone the repository.
2. Create a .env file from the example.
3. Run the container.

Note on Modes:

• Local Mode: Set LOCAL_MODE=true in .env for easy personal use (no authentication required).
• Production Mode: Set LOCAL_MODE=false, configure HOST_BINDING=0.0.0.0, and provide a SECRET_KEY (a long random string generated via python -c "import secrets; print(secrets.token_hex(32))").

# Clone the repository
git clone https://github.com/PhialsBasement/LibreCrawl.git
cd LibreCrawl

# Copy environment file
cp .env.example .env

# Start LibreCrawl
docker compose up -d

To add a custom plugin to LibreCrawl, follow these steps:

1. Create a new .js file in the web/static/plugins/ directory (e.g., my-plugin.js).
2. Use the LibreCrawlPlugin.register() method to define your plugin's configuration and lifecycle hooks.
3. Refresh the LibreCrawl application. Your plugin will automatically appear as a new tab in the UI.

LibreCrawlPlugin.register({
  id: 'my-plugin',
  name: 'My Plugin',
  tab: {
    label: 'My Tab',
    icon: '🔥',
  },
  onTabActivate(container, data) {
    // Implementation
  }
});

The easiest way to run LibreCrawl is using the provided startup scripts. These scripts automatically handle Docker detection, dependency installation via pip, Playwright browser installation for JavaScript rendering, and starting the application in local mode.

Windows:

start-librecrawl.bat

Linux/Mac:

chmod +x start-librecrawl.sh
./start-librecrawl.sh

Once finished, the application will be available at http://localhost:5000.

# Linux/Mac
chmod +x start-librecrawl.sh
./start-librecrawl.sh

To run LibreCrawl directly using Python (requires Python 3.8+):

1. Clone or download the repository.
2. Install dependencies: pip install -r requirements.txt.
3. (Optional) For JavaScript rendering support: playwright install chromium.
4. Run the application:
   • Standard Mode (with authentication and tier system): python main.py
   • Local Mode (admin tier for all, no rate limits): python main.py --local or python main.py -l.

Access the app at http://localhost:5000 (Local) or http://<your-ip>:5000 (Network).

LibreCrawl uses SQLite for authentication and user data.

• Storage Path: The database file is located at data/users.db relative to the project root. This directory is intended for Docker volume persistence.
• Core Tables:
  • users: Stores credentials, verification status, and tier.
  • user_settings: Stores JSON-serialized user preferences.
  • crawl_history: Tracks user-initiated crawl sessions.
  • guest_crawls: Tracks IP-based crawl activity for unauthenticated users.
  • verification_tokens: Manages email verification lifecycle.

Settings can be adjusted via the UI to customize crawling behavior:

• Crawler settings: Depth (up to 5M URLs), delays, and external link handling.
• Request settings: User agent, timeouts, proxy, and robots.txt compliance. Add a Google API key here to increase PageSpeed Insights rate limits.
• JavaScript rendering: Configure the browser engine, wait times, and viewport size.
• Filters: Define file types and URL patterns to include or exclude.
• Export options: Choose between CSV, JSON, or XML formats.
• Custom CSS: Apply custom themes to the UI.
• Issue exclusion: Define patterns to ignore during SEO issue detection.

The crawler automatically performs HEAD requests on image URLs found on a page to detect broken links. This is done in batches (up to 50 images per page) using a thread pool to avoid blocking the main crawl loop.

If broken images are found, they are added to the result object under the broken_images key:

"broken_images": [
    {"url": "https://example.com/broken.png", "status": 404}
]