scrapy-splash

While you can interact with Splash by sending POST requests to render.html manually, scrapy-splash provides several critical improvements:

• Reduces Boilerplate: Automates the construction of JSON bodies and headers.
• Correct URL Handling: Fixes response.url to be the target page URL instead of the Splash server URL. The original URL is available via response.real_url.
• Improved Scrapy Integration:
  • Handles response.status and response.headers transparently.
  • Ensures CONCURRENT_REQUESTS_PER_DOMAIN and DOWNLOAD_DELAY work correctly by mapping requests to their actual target domains.
  • Fixes duplication filtering (dupefilter) to correctly canonicalize URLs sent in JSON bodies.
• Error Debugging: SplashMiddleware automatically logs the content of HTTP 400 (Bad Request) responses from Splash to help debugging. This can be disabled via the SPLASH_LOG_400 = False setting.
• Cookie & State Management: Simplifies cookie handling which is otherwise difficult to implement manually with Splash.
• Efficiency: Provides optimized storage for large, static Splash arguments (like lua_source) in Scrapy disk request queues and supports Splash 2.1+ caching via save_args and load_args.

To integrate scrapy-splash into your Scrapy project, update your settings.py with the following configurations:

1. Set the Splash URL: Define the address of your running Splash server.
2. Enable Middlewares: Add SplashCookiesMiddleware and SplashMiddleware to DOWNLOADER_MIDDLEWARES. You must also adjust the priority of HttpCompressionMiddleware to allow advanced response processing.
3. Enable Spider Middleware: Add SplashDeduplicateArgsMiddleware to SPIDER_MIDDLEWARES to support the cache_args feature (saves disk space and network traffic).
4. Set Request Fingerprinter: Use scrapy_splash.SplashRequestFingerprinter to ensure requests are fingerprinted correctly for Splash.

SPLASH_URL = 'http://192.168.59.103:8050'

DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}

SPIDER_MIDDLEWARES = {
    'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}

REQUEST_FINGERPRINTER_CLASS = 'scrapy_splash.SplashRequestFingerprinter'

Since Splash is stateless, you must manually manage cookies to maintain sessions. scrapy-splash provides helpers to send current cookies to Splash and merge updated cookies back into Scrapy.

To enable session handling:

1. Use the /execute endpoint.
2. Use a Lua script that accepts a cookies argument and returns a cookies field in the result.
3. Set request.meta['splash']['session_id'] to a unique identifier (e.g., '1' or 'foo').

If you use SplashRequest with the /execute endpoint and a compatible Lua script, session_id is set automatically.

To 'fork' a session (start from existing cookies but save updates to a new session), set request.meta['splash']['new_session_id'] in addition to session_id.

function main(splash)
    splash:init_cookies(splash.args.cookies)

    -- ... your script

    return {
        cookies = splash:get_cookies(),
        -- ... other results, e.g. html
    }
end

There are two ways to provide credentials for HTTP Basic Authentication when accessing the Splash server:

1. Global Settings: Set SPLASH_USER and SPLASH_PASS in your Scrapy settings.
2. Per-Request Headers: Use meta['splash']['splash_headers'] to pass custom headers, such as an Authorization header, for specific requests.

WARNING: Do not use Scrapy's HttpAuthMiddleware (http_user / http_pass) for Splash authentication. This may expose your Splash credentials to remote websites if you send non-Splash requests.

# Option 1: Global settings
SPLASH_USER = 'user'
SPLASH_PASS = 'userpass'

# Option 2: Per-request via splash_headers
from w3lib.http import basic_auth_header

class MySpider(scrapy.Spider):
    def start_requests(self):
        auth = basic_auth_header('user', 'userpass')
        yield SplashRequest(url, self.parse, splash_headers={'Authorization': auth})

Using Splash 2.1+, you can pass custom arguments (like a CSS selector and padding) to a Lua script to capture a screenshot of a specific element's bounding box.

import scrapy
from scrapy_splash import SplashRequest

script = """
-- Arguments: url, css, pad
function pad(r, pad)
  return {r[1]-pad, r[2]-pad, r[3]+pad, r[4]+pad}
end

function main(splash)
  local get_bbox = splash:jsfunc([[
    function(css) {
      var el = document.querySelector(css);
      var r = el.getBoundingClientRect();
      return [r.left, r.top, r.right, r.bottom];
    }
  ]])

  assert(splash:go(splash.args.url))
  assert(splash:wait(0.5))
  splash:set_viewport_full()

  local region = pad(get_bbox(splash.args.css), splash.args.pad)
  return splash:png{region=region}
end
"""

class MySpider(scrapy.Spider):
    # ...
    def start_requests(self):
        yield SplashRequest(url, self.parse_element_screenshot,
            endpoint='execute',
            args={
                'lua_source': script,
                'pad': 32,
                'css': 'a.title'
            }
         )

    def parse_element_screenshot(self, response):
        image_data = response.body  # binary PNG data

To avoid sending the entire Lua script with every request, use the cache_args parameter in SplashRequest. This requires Splash 2.1+. You can also use a Lua script to return headers, status, and cookies from the last response.

import scrapy
from scrapy_splash import SplashRequest

script = """
function main(splash)
  splash:init_cookies(splash.args.cookies)
  assert(splash:go{
    splash.args.url,
    headers=splash.args.headers,
    http_method=splash.args.http_method,
    body=splash.args.body,
    })
  assert(splash:wait(0.5))

  local entries = splash:history()
  local last_response = entries[#entries].response
  return {
    url = splash:url(),
    headers = last_response.headers,
    http_status = last_response.status,
    cookies = splash:get_cookies(),
    html = splash:html(),
  }
end
"""

class MySpider(scrapy.Spider):
    # ...
    def start_requests(self):
        yield SplashRequest(url, self.parse_result,
            endpoint='execute',
            cache_args=['lua_source'],
            args={'lua_source': script},
            headers={'X-My-Header': 'value'},
        )

    def parse_result(self, response):
        # response.body contains result HTML
        # response.headers contains headers from the last web page loaded
        # cookies are collected into Set-Cookie response header for Scrapy
        pass