recipe-scrapers

The recipe-scrapers library is a neutral tool for transforming unstructured recipe data into structured formats. It does not store, host, or redistribute content, nor does it circumvent technical access controls.

End User Responsibilities: Users are solely responsible for the legal implications of their scraping activities, including:

• Ensuring compliance with applicable laws.
• Respecting robots.txt and Terms of Service.
• Obtaining necessary permissions for content use.
• Managing copyright and licensing requirements.

Fair Use: While uses like personal collection, academic research, and non-commercial educational purposes may fall under fair use, users must conduct their own legal analysis for their specific jurisdiction and use case.

The recipe-scrapers library is designed to focus exclusively on HTML parsing. It does not handle network requests or fetching content from the internet.

To use the library, you must implement your own solution for fetching the recipe HTML (using libraries like requests, httpx, aiohttp, or urllib.request) and then pass that HTML content to the scraper. For optimal parsing results, you should provide both the raw HTML content and the original source URL (org_url).

To scrape a recipe, fetch the HTML content of the target URL using an HTTP client, then use scrape_html from recipe_scrapers.

Pass the HTML string and the org_url (the original URL of the recipe) to scrape_html. This returns a scraper object that provides various methods to extract specific recipe data.

Install the package using pip to begin extracting recipe data from cooking websites.

pip install recipe-scrapers

The simplest way to parse a recipe is by using the scrape_me function. Pass a recipe URL directly to scrape_me(), which returns a scraper object containing the parsed data.

Note: This library focuses exclusively on HTML parsing. You are responsible for fetching the HTML and managing network requests. The library works best when you provide the URL so it can identify the correct scraper for the domain.

When using recipe-scrapers to extract data, follow these best practices to ensure ethical and responsible usage:

• Minimize server load: Cache scraped content appropriately to avoid excessive requests.
• Respect website rules: Always follow robots.txt directives and consider the website's Terms of Service.
• Rate limiting: Implement respect for rate limits to avoid overwhelming target servers.
• Attribution: Include proper attribution when displaying scraped content.
• Error handling: Implement robust error handling and fallbacks in your scraping logic.

You can programmatically retrieve the list of all currently supported recipe website hosts by accessing the keys of the SCRAPERS object. This allows you to dynamically check which sites are available for scraping in your application.

In addition to the explicitly supported sites, the library provides a wild_mode option which enables scraping of many more sites that follow common web patterns, significantly expanding the library's coverage.

When calling scrape_html, you can control the scraping logic using several keyword arguments:

By default, the library enables higher-quality image detection when multiple options are available. You can control this behavior in two ways:

1. Per-call basis: Pass best_image=False to scrape_html to keep the first image returned by the site.
2. Globally: Update the BEST_IMAGE_SELECTION setting in recipe_scrapers.settings.settings.

# Per-call basis
scraper = scrape_html(html, url, best_image=False)
image_url = scraper.image()

# Global setting
from recipe_scrapers.settings import settings
settings.BEST_IMAGE_SELECTION = False

To create a new scraper for a specific website, you must subclass AbstractScraper and implement the methods that raise NotImplementedError. The base class handles HTML parsing via BeautifulSoup, OpenGraph, and SchemaOrg metadata extraction, as well as plugin application.

Required methods to implement typically include:

• host(): Returns the host domain of the recipe URL.
• site_name(): Returns the name of the website.
• title(): Returns the recipe title.
• ingredients(): Returns the list of ingredients.
• instructions(): Returns the instructions as a single string.
• image(): Returns an image URL for the recipe.

Other useful methods provided by the base class include canonical_url(), language(), instructions_list(), and to_json().