CrawlerDetect

By configuring the Rack::CrawlerDetect middleware, you can extend the request object with is_crawler? and crawler_name methods. This approach is more robust than manual detection because it automatically checks 10 different HTTP headers in addition to the HTTP_USER_AGENT.

### Rails
```ruby
class Application < Rails::Application
  # ...
  config.middleware.use Rack::CrawlerDetect
end

Rack

use Rack::CrawlerDetect

To use CrawlerDetect in your Ruby application, add the gem to your Gemfile.

gem 'crawler_detect'

You can override the default crawler data by providing your own JSON files for headers, crawlers, and exclusions using the CrawlerDetect.setup! block. This is useful for implementing custom white-lists or black-lists.

CrawlerDetect.setup! do |config|
  config.raw_headers_path    = File.expand_path("crawlers/MyHeaders.json", __dir__)
  config.raw_crawlers_path   = File.expand_path("crawlers/MyCrawlers.json", __dir__)
  config.raw_exclusions_path = File.expand_path("crawlers/MyExclusions.json", __dir__)
end

Integrate Rack::CrawlerDetect into your Rack-based application stack to automatically detect crawlers based on User-Agent headers. When the middleware processes a request, it populates the env hash with detection results, which are then accessible via helper methods on the Rack::Request object.

# Example Rack middleware integration
use Rack::CrawlerDetect

You can perform simple bot detection using the class methods or by instantiating a new detector object. Use CrawlerDetect.is_crawler? for a boolean check or CrawlerDetect.new to retrieve the specific crawler name.

# Simple boolean check
CrawlerDetect.is_crawler?("Bot user agent")
# => true

# Detailed detection with crawler name
detector = CrawlerDetect.new("Googlebot/2.1 (http://www.google.com/bot.html)")
detector.is_crawler?
# => true
detector.crawler_name
# => "Googlebot"

CrawlerDetect uses three primary JSON files to define its detection logic: crawlers, exclusions, and headers. While these are pre-configured to point to the library's internal library/raw directory, you can override these paths in the CrawlerDetect::Config object if you wish to use custom JSON data files.

Available configuration settings:

• raw_crawlers_path: The file path to the crawlers raw JSON file.
• raw_exclusions_path: The file path to the exclusions raw JSON file.
• raw_headers_path: The file path to the headers raw JSON file.

All settings must be valid strings.

# Example of overriding default paths
CrawlerDetect.configure do |config|
  config.raw_crawlers_path = '/path/to/your/custom_crawlers.json'
  config.raw_exclusions_path = '/path/to/your/custom_exclusions.json'
  config.raw_headers_path = '/path/to/your/custom_headers.json'
end

The CrawlerDetect::Detector class is used to identify if a given User-agent string belongs to a known crawler. You initialize the detector with a User-agent string and then use is_crawler? to check for a match and crawler_name to retrieve the specific name of the detected crawler.

Note that crawler_name will only return a value if is_crawler? is true.

# Initialize with a User-agent string
detector = CrawlerDetect::Detector.new("Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)")

if detector.is_crawler?
  puts "Detected: #{detector.crawler_name}"
else
  puts "Not a crawler"
end
# Output: Detected: Googlebot

Use CrawlerDetect.is_crawler?(user_agent) to perform a quick check on whether a specific User-Agent string belongs to a known crawler. It returns true if a crawler is detected and false otherwise.

CrawlerDetect.is_crawler?("Googlebot") # => true
CrawlerDetect.is_crawler?("Mozilla/5.0") # => false

The CrawlerDetect::Library::Loader module provides utility methods for handling the raw JSON data used by the library.

• load_raw(path): Reads and parses a JSON file from the specified path into a Ruby hash.
• reload_data: Clears the internal @data instance variable if it exists, forcing the library to reload data from the source on the next access.

# Example of loading a raw JSON file
data = CrawlerDetect::Library::Loader.load_raw('path/to/data.json')

# Example of clearing cached data to force a reload
CrawlerDetect::Library::Loader.reload_data

Once Rack::CrawlerDetect is added to your middleware stack, it extends Rack::Request::Helpers to provide two convenient methods for inspecting the current request:

• is_crawler?: Returns true if the request is identified as a crawler, otherwise false.
• crawler_name: Returns the name of the detected crawler (e.g., "Googlebot") or nil if no crawler is detected.

These methods retrieve data from the env["rack.crawler_detect"] hash populated by the middleware.

# Inside a Rack application or Rails controller
if request.is_crawler?
  puts "Detected: #{request.crawler_name}"
end

You can access the current configuration object via CrawlerDetect.config. If setup! has not been called yet, this will return a new default CrawlerDetect::Config instance.

config = CrawlerDetect.config
# Access configuration properties

Use CrawlerDetect.new(user_agent) to create a new instance of CrawlerDetect::Detector for a specific User-Agent. This allows for more granular interaction with the detector instance.

detector = CrawlerDetect.new("Googlebot")
# Use the detector instance as needed