django-robots

The application uses two primary abstractions to manage robot instructions:

Rule

An abstract Rule defines how to respond to specific web robots (user agents). Rules can link multiple URL patterns to allow or disallow actions.

• Crawl Delay: Supports setting a delay (in seconds) between successive crawler accesses to reduce server load.
• Multi-site support: Uses the Django sites framework to enable different robots.txt rules per Django instance.

Url

An Url defines the specific pattern to match.

• Matching: Patterns are case-sensitive and exact.
• Trailing Slashes: A missing trailing slash in a pattern will also match files starting with that name (e.g., '/admin' matches '/admin.html').
• Wildcards: Some search engines support * (wildcard for any sequence) and $ (end of URL). Example: '/*.jpg$' matches all JPEG files.

Install the package using pip and configure your Django settings to enable it.

Installation Steps

1. Install via PyPI:

   pip install django-robots

2. Add 'robots' to your INSTALLED_APPS in settings.py.
3. Ensure 'django.template.loaders.app_directories.Loader' is present in your TEMPLATES setting.
4. Install and configure the Django sites framework.
5. Run the database migrations:

   python manage.py migrate

To enable the generation of robots.txt at the /robots.txt endpoint, you must include the robots.urls in your project's main urls.py file using re_path.

from django.urls import re_path, include

urlpatterns = [
    ...
    re_path(r'^robots\.txt', include('robots.urls')),
    ...
]

By default, a Host statement is automatically added to the robots.txt to help select the main website and avoid mirrors.

Configuration Options

• Disable Host directive: Set ROBOTS_USE_HOST = False.
• Include protocol in Host directive: To prefix the domain with the current request protocol (e.g., Host: https://www.mysite.com), set ROBOTS_USE_SCHEME_IN_HOST = True.

You can cache the generated robots.txt output to improve performance. Use the ROBOTS_CACHE_TIMEOUT setting to specify the cache duration in seconds.

Example (24-hour cache):

ROBOTS_CACHE_TIMEOUT = 60*60*24

If set to None (the default), no caching is applied.

ROBOTS_CACHE_TIMEOUT = 86400

To use django-robots in a Django project, you may need to add its configuration class to your INSTALLED_APPS setting. The configuration class is robots.apps.RobotsConfig.

INSTALLED_APPS = [
    ...
    'robots.apps.RobotsConfig',
    ...
]

The django-robots package provides built-in Django Admin integration to manage Rule and Url objects.

• Rules (RuleAdmin): Allows you to define which robots (e.g., Googlebot) are subject to specific rules. It includes fieldsets for robot identification, site association, URL patterns (allowed/disallowed), and advanced options like crawl_delay.
• URL Patterns (Url): Individual URL patterns associated with rules can be managed directly.

To enable this in your Django project, ensure robots is included in your INSTALLED_APPS and that you have configured the Django admin site.

The RuleList view is the primary entry point for serving a generated robots.txt file. It returns the file with a text/plain MIME type and automatically handles sitemap URL inclusion based on your project's configuration.

To use it, you should point a URL pattern to rules_list (which is the as_view() instance of RuleList).

from django.urls import path
from robots.views import rules_list

urlpatterns = [
    path('robots.txt', rules_list),
]

The Rule model provides helper methods to get a human-readable list of associated URL patterns.

• allowed_urls(): Returns a string containing a list of all URLs in the allowed relationship.
• disallowed_urls(): Returns a string containing a list of all URLs in the disallowed relationship.

# Assuming 'rule' is an instance of Rule
print(rule.allowed_urls())
print(rule.disallowed_urls())