DOCSEARCHbyAlgolia
Submit Docs

Laravel Robots Middleware by Spatie

repository·main·Indexed 43 days ago

https://github.com/spatie/laravel-robots-middleware

A Laravel package that manages search engine indexing by automatically adding x-robots-tag headers to HTTP responses. Developers can implement custom indexing logic by extending the RobotsMiddleware class and overriding the shouldIndex method.

Tokens
792
Snippets
4
Records
4
Agent score
6%

What's inside Laravel Robots Middleware

  1. Configure indexing rules by extending RobotsMiddleware

    main

    By default, the middleware enables indexing on all pages. To implement custom logic, create a new middleware class that extends Spatie\RobotsMiddleware\RobotsMiddleware and implement the shouldIndex method.

    The shouldIndex method must return either a string or a bool. If it returns true (or a truthy value), the response will include an x-robots-tag: all header. If it returns false (or a falsy value), the response will include an x-robots-tag: none header.

    // app/Http/Middleware/MyRobotsMiddleware.php
namespace App\
Http\\Middleware;

use Illuminate\\Http\\Request;
use Spatie\\RobotsMiddleware\\RobotsMiddleware;

class MyRobotsMiddleware extends RobotsMiddleware
{
    /**
     * @return string|bool
     */
    protected function shouldIndex(Request $request)
    {
        // Example: Do not index any routes starting with 'admin'
        return $request->segment(1) !== 'admin';
    }
}

  3. Register your custom Robots middleware

    main

    After creating your custom middleware class, register it in your application's middleware stack (typically in app/Http/Kernel.php) to ensure the x-robots-tag header is applied to your responses.

    // app/Http/Kernel.php

class Kernel extends HttpKernel
{
    protected $middleware = [
        // ...
        \App\Http\Middleware\MyRobotsMiddleware::class,
    ];
    
    // ...
}

  4. Use RobotsMiddleware to control indexing via HTTP headers

    main

    The RobotsMiddleware is a Laravel middleware designed to control how search engine crawlers index your application by setting the x-robots-tag HTTP header.

    When applied to a route, the middleware evaluates whether the page should be indexed. It can return:

    • 'all': Allows indexing.
    • 'none': Prevents indexing.
    • Any other valid robots string (e.g., 'noindex, nofollow').

    Note: The base implementation of shouldIndex() always returns true, meaning by default it will set the header to all. To implement custom logic, you must extend this class.

    // Example of how the middleware is typically applied in a Laravel route file
Route::get('/private-page', function () {
    return view('private');
})->middleware(RobotsMiddleware::class);