scrapy-playwright

The PLAYWRIGHT_BROWSER_PROVIDER setting is an extension point that allows you to replace the default browser lifecycle management. By providing a custom class (or an import path string), you can integrate third-party drivers that are Playwright-compatible (e.g., patchright or camoufox) without modifying the Scrapy handler.

PLAYWRIGHT_BROWSER_PROVIDER = "myproject.providers.CustomBrowserProvider"

You can replace the default Playwright browser startup with third-party projects like patchright, camoufox, or invisible_playwright. These projects provide drop-in replacements that maintain standard Browser, BrowserContext, and Page objects, meaning your existing routing and page code remains unchanged.

To use a custom provider, set the PLAYWRIGHT_BROWSER_PROVIDER setting to the import path of your provider class or the class itself.

# settings.py
PLAYWRIGHT_BROWSER_PROVIDER = "myproject.providers.CustomBrowserProvider"

Install the package via pip. Note that playwright is a dependency and will be installed automatically. You must also manually install the browser binaries (e.g., chromium, firefox) using the Playwright CLI.

pip install scrapy-playwright

# Install specific browsers
playwright install firefox chromium

When reporting issues, provide a self-contained spider that can be run with scrapy runspider. Include the necessary custom_settings to ensure the Twisted reactor and Download Handlers are correctly configured for Playwright.

import scrapy

class ExampleSpider(scrapy.Spider):
    name = "example"
    custom_settings = {
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "DOWNLOAD_HANDLERS": {
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        },
    }

    async def start(self):
        yield scrapy.Request(
            url="https://example.org",
            meta={"playwright": True},
        )

When using dynamic contexts, you must manually close them to avoid memory leaks and hitting the PLAYWRIGHT_MAX_CONTEXTS limit.

Crucial Order: Always close the Page object before closing its corresponding context.

Access the context via response.meta['playwright_page'].context and use await context.close().

async def parse_in_new_context(self, response):
    page = response.meta["playwright_page"]
    title = await page.title()
    await page.close()  # Close page first
    await page.context.close()  # Then close context
    return {"title": title}

async def close_context_on_error(self, failure):
    page = failure.request.meta["playwright_page"]
    await page.close()
    await page.context.close()

To use Playwright with a CrawlSpider, you must ensure that requests generated by your crawling rules have the playwright key set to True in their meta attribute. This is achieved by providing a process_request method to your Rule objects that modifies the request in-place.

def set_playwright_true(request, response):
    request.meta["playwright"] = True
    return request

class MyCrawlSpider(CrawlSpider):
    ...
    rules = (
        Rule(
            link_extractor=LinkExtractor(...),
            callback="parse_item",
            follow=False,
            process_request=set_playwright_true,
        ),
    )

The default Scrapy MemoryUsage extension does not account for memory used by Playwright because browsers run as separate processes. To track total memory usage including Playwright, replace the built-in extension with scrapy_playwright.memusage.ScrapyPlaywrightMemoryUsageExtension.

Requirements:

• You must install the psutil package.
• Note: This extension does not work on Windows because it relies on the Python resource module.

Update your settings.py as follows:

# settings.py
EXTENSIONS = {
    "scrapy.extensions.memusage.MemoryUsage": None,
    "scrapy_playwright.memusage.ScrapyPlaywrightMemoryUsageExtension": 0,
}

If you want every request in your spider to be processed by Playwright without manually setting meta['playwright'] = True for each one, you can use a Scrapy middleware to inject the required metadata.

Depending on your needs, you can use either a Spider Middleware or a Downloader Middleware.

# Spider middleware example
class PlaywrightSpiderMiddleware:
    def process_spider_output(self, response, result, spider):
        for obj in result:
            if isinstance(obj, scrapy.Request):
                obj.meta.setdefault("playwright", True)
            yield obj

# Downloader middleware example
class PlaywrightDownloaderMiddleware:
    def process_request(self, request, spider):
        request.meta.setdefault("playwright", True)
        return None