DOCSEARCHbyAlgolia
Submit Docs

ai.robots.txt

repository·main·Indexed 26 days ago

https://github.com/ai-robots-txt/ai.robots.txt

A centralized repository of AI-related crawlers providing standardized configuration files to block AI bots. It includes implementation guides for Apache, Nginx, Caddy, HAProxy, Lighttpd, and the Robots Exclusion Protocol, as well as specific instructions for opting out of Bing AI training and setting up central robots.txt services using Traefik.

Tokens
1.6K
Snippets
5
Records
11
Agent score
88%

What's inside ai.robots.txt

  1. Setup a central robots.txt service using Traefik and NGINX

    main

    If you use Traefik as a reverse proxy in Docker, you can centrally serve a single /robots.txt file for all your services. This is achieved by running a lightweight service (like nginx:alpine) to host the static file and configuring a high-priority Traefik HTTP Router rule that matches the path /robots.txt regardless of the hostname.

    services:
  robots:
    image: nginx:alpine
    container_name: robots-server
    volumes:
      - ./static/:/usr/share/nginx/html/:ro
    labels:
      - "traefik.enable=true"
      # Router for all /robots.txt requests
      - "traefik.http.routers.robots.rule=Path(`/robots.txt`)"
      - "traefik.http.routers.robots.entrypoints=web,websecure"
      - "traefik.http.routers.robots.priority=3000"
      - "traefik.http.routers.robots.service=robots"
      - "traefik.http.routers.robots.tls.certresolver=letsencrypt"
      - "traefik.http.services.robots.loadbalancer.server.port=80"
    networks:
      - external_network

networks:
  external_network:
     name: traefik_external_network
     external: true

  2. Opt-out of Bing AI training using HTML metatags

    main

    To prevent Bing from using your website content to train Microsoft's generative AI foundation models, add the noarchive robots metatag to the <head> section of every page on your website. This method is safe and does not impact your site's search engine visibility, as major search engines like Google have retired the use of noarchive for traditional search purposes.

    <meta name="robots" content="noarchive">

  5. Configure HAProxy to block AI bots

    main

    To block AI bots using HAProxy, follow these steps:

    1. Add the haproxy-block-ai-bots.txt file to your HAProxy configuration directory.
    2. Add the following lines to your frontend section (ensure the path to the text file matches your environment):
    acl ai_robot hdr_sub(user-agent) -i -f /etc/haproxy/haproxy-block-ai-bots.txt
http-request deny if ai_robot
    acl ai_robot hdr_sub(user-agent) -i -f /etc/haproxy/haproxy-block-ai-bots.txt
http-request deny if ai_robot

  6. Block crawlers using User Agent strings

    main
    You can block crawlers based on their User Agent strings if the crawlers identify themselves and your application or hosting provider supports User Agent filtering. Note that some crawlers (e.g., Perplexity) may not identify themselves via User Agent strings, making them difficult to block using this method.

  8. Opt-out of Bing AI training using HTTP headers

    main

    To prevent Bing from using your website content to train Microsoft's generative AI foundation models, set the X-Robots-Tag HTTP header to noarchive in your server's response. This must be applied to every page or response on your website. This approach is non-disruptive to search engine rankings.

    X-Robots-Tag: noarchive

  9. Implement AI crawler blocking via web server configurations

    main

    The ai.robots.txt repository provides several configuration files to block AI-related crawlers across different web server environments. You can choose the file corresponding to your server type:

    • Apache: Use .htaccess to return error pages when listed AI crawlers make requests.
    • Nginx: Include the nginx-block-ai-bots.conf snippet into your virtual host server {} block using the include directive.
    • Caddy: Copy or import the Caddyfile Header Regex matcher group, then handle rejection using abort @aibots.
    • HAProxy: Use haproxy-block-ai-bots.txt to configure ACLs.
    • Lighttpd: Include lighttpd-block-ai-bots.conf globally or in a conditional section using include "fragments/lighttpd-block-ai-bots.conf".
    • Robots Exclusion Protocol: Use the provided robots.txt to implement standard exclusion rules.

  10. Troubleshoot bots that do not respect robots.txt

    main

    Since robots.txt compliance is voluntary and lacks an enforcement mechanism, you may need to implement blocking at the infrastructure or server level if a bot ignores your instructions. Common methods include:

    • Nginx: Use Nginx configuration to block specific bots.
    • Apache httpd: Use mod_setenvif (recommended for lower resource usage) or mod_rewrite (use as a last resort). It is also recommended to configure these in httpd.conf rather than .htaccess if possible.
    • Netlify: Use Netlify Edge Functions to block bots.
    • Cloudflare: Use Cloudflare's built-in features to block AI bots, scrapers, and crawlers.
    • Vercel: Use Vercel Firewall Rules to block AI bots.

  11. Configure Traefik Router rules for global robots.txt

    main

    To ensure the central robots.txt service intercepts requests before other service routers, use the following Traefik label configurations:

    • traefik.http.routers.robots.rule=Path("/robots.txt"): Matches the specific path without a Hostname requirement.
    • traefik.http.routers.robots.priority=3000: Sets a high priority to ensure this rule is evaluated first.
    • traefik.http.routers.robots.entrypoints=web,websecure: Enables the router on both HTTP and HTTPS entrypoints.

    Note: Because the rule does not contain a Hostname, Traefik may print a warning during TLS setup, but the configuration will function correctly.