Scrapling Python Library

Scrapling is optimized for high-speed parsing and element discovery. Benchmarks comparing text extraction speed (using 5000 nested elements) and element similarity/text search performance show Scrapling outperforming several popular libraries:

Text Extraction Speed (5000 nested elements):

• Scrapling: 1.99 ms (Baseline)
• Parsel/Scrapy: 2.06 ms
• Raw Lxml: 2.56 ms
• PyQuery: 23.98 ms
• Selectolax: 197.02 ms
• MechanicalSoup: 1545.15 ms
• BS4 with Lxml: 1562.1 ms
• BS4 with html5lib: 3412.73 ms

Element Similarity & Text Search Performance:

• Scrapling: 2.3 ms (Baseline)
• AutoScraper: 12.58 ms

Note: All benchmarks represent averages of 100+ runs.

| # |      Library      | Time (ms) | vs Scrapling | 
|---|:-----------------:|:---------:|:------------:|
| 1 |     Scrapling     |   1.99    |     1.0x     | 
| 2 |   Parsel/Scrapy   |   2.06    |    1.035     | 
| 3 |     Raw Lxml      |   2.56    |    1.286     | 
| 4 |      PyQuery      |   23.98   |     ~12x     | 
| 5 |    Selectolax     |  197.02   |     ~99x     | 
| 6 |  MechanicalSoup   |  1545.15   |   ~776.5x    | 
| 7 |    BS4 with Lxml  |  1562.1    |   ~785.0x    | 
| 8 | BS4 with html5lib |  3412.73   |   ~1714.9x   |

Scrapling provides multiple ways to fetch web content depending on the target's complexity:

• Fetcher: Fast, stealthy HTTP requests with browser impersonation (TLS fingerprint, headers, HTTP/3).
• DynamicFetcher: Full browser automation using Playwright (Chromium) or Google Chrome.
• StealthyFetcher: Advanced anti-bot bypass and fingerprint spoofing (e.g., Cloudflare Turnstile/Interstitial).
• Session Management: Use FetcherSession, StealthySession, or DynamicSession to maintain cookies and state across requests.
• Proxy & Network: Built-in ProxyRotator (cyclic or custom), DNS-over-HTTPS (DoH) to prevent leaks, and domain/ad blocking.
• Remote Browsers: Connect to existing browsers via cdp_url or specify a custom executable_path.
• Background API Capture: Use capture_xhr to automatically collect all matching XHR/fetch responses into response.captured_xhr.

In Scrapling, you can configure arguments globally for a session or individually for each request.

Global Session Configuration: Arguments set in the session class apply to all requests made within that session.

Individual Request Configuration: You can override session settings by passing specific arguments during a request. The following arguments can be configured at the browser tab/request level:

• google_search
• timeout
• wait
• page_action
• page_setup
• extra_headers
• disable_resources
• wait_selector
• wait_selector_state
• network_idle
• load_dom
• blocked_domains
• proxy
• selector_config

|      Argument       | Description                                                                                                                                                                                                                         | Optional |
|:-------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
|         url         | Target url                                                                                                                                                                                                                          |    ❌     |
|      headless       | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode.                                                                                                 |    ✔️    |
|  disable_resources  | Drop requests for unnecessary resources for a speed boost. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.                         |    ✔️    |
|       cookies       | Set cookies for the next request.                                                                                                                                                                                                   |    ✔️    |
|      useragent      | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser and version.**                                                                               |    ✔️    |
|    network_idle     | Wait for the page until there are no network connections for at least 500 ms.                                                                                                                        |    ✔️    |
|      load_dom       | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state).                                                                                             |    ✔️    |
|       timeout       | The timeout (milliseconds) used in all operations and waits through the page. The default is 30,000 ms (30 seconds).                                                                                             |    ✔️    |
|        wait         | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object.                                                                             |    ✔️    |
|     page_action     | Added for automation. Pass a function that takes the `page` object, runs after navigation, and does the necessary automation.                                                                                      |    ✔️    |
|     page_setup      | A function that takes the `page` object, runs before navigation. Use it to register event listeners or routes that must be set up before the page loads.                                                     |    ✔️    |
|    wait_selector    | Wait for a specific css selector to be in a specific state.                                                                                                                                          |    ✔️    |
|     init_script     | An absolute path to a JavaScript file to be executed on page creation for all pages in this session.                                                                                             |    ✔️    |
| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._                                                              |    ✔️    |
|    google_search    | Enabled by default, Scrapling will set a Google referer header.                                                                                                                                     |    ✔️    |
|    extra_headers    | A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._                                                                |    ✔️    |
|        proxy        | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'.                                                                              |    ✔️    |
|     real_chrome     | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser.                                                                                |    ✔️    |
|       locale        | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. |    ✔️    |
|     timezone_id     | Changes the timezone of the browser. Defaults to the system timezone.                                                                                                                             |    ✔️    |
|       cdp_url       | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.                                                                                             |    ✔️    |
|    user_data_dir    | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions**                                   |    ✔️    |
|     extra_flags     | A list of additional browser flags to pass to the browser on launch.                                                                                                                             |    ✔️    |
|   additional_args   | Additional arguments to be passed to Playwright's context as additional settings, and they take higher priority than Scrapling's settings.                                                                              |    ✔️    |
|   selector_config   | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class.                                                                                                            |    ✔️    |
|   blocked_domains   | A set of domain names to block requests to. Subdomains are also matched (e.g., `

Use the FetcherSession class to make multiple HTTP requests while maintaining a shared configuration, connection pool, and cookie persistence. FetcherSession automatically detects whether it is being used in synchronous or asynchronous code. It is significantly faster than creating a new session for every request and is more resource-efficient.

When to use FetcherSession:

• Making multiple requests to the same or different sites.
• Maintaining cookies or authentication state between requests.
• Utilizing connection pooling for performance.
• Requiring consistent configuration (like impersonate or http3) across a series of requests.

from scrapling.fetchers import FetcherSession

# Create a session with default configuration
with FetcherSession(
    impersonate='chrome',
    http3=True,
    stealthy_headers=True,
    timeout=30,
    retries=3
) as session:
    # All requests share the same session, cookies, and connection pool
    page1 = session.get('https://scrapling.requestcatcher.com/get')
    page2 = session.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'})
    page3 = session.get('https://api.github.com/events')

The Spider class allows you to build robust, concurrent crawlers.

Key Features:

• Concurrency: Set concurrent_requests to control speed.
• Parsing: Implement an async def parse(self, response: Response) method to extract data and yield results or follow links.
• Multi-Session Support: Use configure_sessions(self, manager) to register different session types (e.g., 'fast' for normal pages and 'stealth' for protected ones). Use the sid parameter in Request to select a session.
• Checkpointing: Pass a crawldir to the Spider constructor to enable pause/resume functionality. Pressing Ctrl+C saves progress automatically.

from scrapling.spiders import Spider, Request, Response
from scrapling.fetchers import FetcherSession, AsyncStealthySession

class MultiSessionSpider(Spider):
    name = "multi"
    start_urls = ["https://example.com/"]
    
    def configure_sessions(self, manager):
        manager.add("fast", FetcherSession(impersonate="chrome"))
        manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)
    
    async def parse(self, response: Response):
        for link in response.css('a::attr(href)').getall():
            if "protected" in link:
                yield Request(link, sid="stealth")
            else:
                yield Request(link, sid="fast", callback=self.parse)

# To enable pause/resume:
# QuotesSpider(crawldir="./crawl_data").start()