DOCSEARCHbyAlgolia
Submit Docs

Splash Documentation

repository·master·Indexed 26 days ago

https://github.com/scrapinghub/splash

A lightweight, stateless JavaScript rendering service with an HTTP API used for web scraping dynamic content. Features include endpoints for rendering HTML, PNG, and JPEG screenshots, HAR data export, and JSON-encoded webpage data. It supports custom Lua scripts via the execute and run endpoints, JavaScript execution within page contexts, JavaScript profiles, request filtering using Adblock Plus rules, and proxy profiles.

Tokens
27.4K
Snippets
77
Records
174
Agent score
87%

What's inside Splash

  1. Overview of Splash JavaScript rendering service

    master

    Splash is a lightweight web browser with an HTTP API, implemented in Python 3 using Twisted and QT5. It is designed to handle JavaScript rendering for web scraping tasks.

    Key features include:

    • Processing multiple webpages in parallel.
    • Retrieving HTML results and/or taking screenshots.
    • Optimizing rendering speed by turning OFF images or using Adblock Plus rules.
    • Executing custom JavaScript in the page context.
    • Writing Lua browsing scripts.
    • Developing Lua scripts in Splash-Jupyter Notebooks.
    • Obtaining detailed rendering information in HAR format.

  2. Available Lua Libraries in Splash

    master

    Splash provides several Lua libraries for scripting. When the Sandbox is enabled (default), only a subset of standard Lua 5.2 libraries is available.

    Standard Libraries (Pre-imported):

    • string
    • table
    • math
    • os

    Non-standard Modules (Must be imported via require):

    • json: For encoding/decoding JSON data.
    • base64: For encoding/decoding Base64 data.
    • treat: For fine-tuning how Splash handles Lua variables and return results.

    To use non-standard modules, you must use require before they can be used in your main function.

    base64 = require("base64")
function main(splash)
    return base64.encode('hello')
end

  3. Understand the Element Object in Splash scripting

    master

    In Splash scripting, Element objects act as wrappers around JavaScript DOM nodes (such as Node, Element, or HTMLElement). These objects are automatically created whenever a method returns a DOM node.

    Key behaviors:

    • Methods like select and select_all return Element objects.
    • evaljs can return Element objects, but currently, they must be top-level nodes or NodeList objects; they cannot be nested inside other objects or arrays.

  4. Install Splash using Docker

    master

    To run Splash, you need Docker version >= 17 installed. You can pull the latest stable image and run it with the following commands.

    On Linux:

    $ sudo docker pull scrapinghub/splash
$ sudo docker run -it -p 8050:8050 --rm scrapinghub/splash

    On OS X:

    $ docker pull scrapinghub/splash
$ docker run -it -p 8050:8050 --rm scrapinghub/splash

    Once started, Splash is available at 0.0.0.0 on port 8050 (http).

    # Linux
sudo docker pull scrapinghub/splash
sudo docker run -it -p 8050:8050 --rm scrapinghub/splash

# OS X
docker pull scrapinghub/splash
docker run -it -p 8050:8050 --rm scrapinghub/splash

  5. Call Splash methods in Lua scripts

    master

    In Splash Lua scripts, methods are called on the splash object using the colon (:) syntax. You can invoke methods using either positional arguments (with parentheses ()) or named/keyword arguments (with curly braces {}). All Splash methods support both styles.

    • Positional arguments: splash:method(val1, val2)
    • Named arguments: splash:method{key1=val1, key2=val2}
    • Mixed arguments: splash:method{val1, key2=val2}
    -- Examples of positional arguments:
splash:go("http://example.com")
splash:wait(0.5, false)
local title = splash:evaljs("document.title")

-- The same using keyword arguments:
splash:go{url="http://example.com"}
splash:wait{time=0.5, cancel_on_redirect=false}
local title = splash:evaljs{source="document.title"}

-- Mixed arguments example:
splash:wait{0.5, cancel_on_redirect=false}

  6. Configure custom Lua modules in Splash

    master

    To use custom Lua modules in your Splash scripts, you must perform three steps:

    1. Set the module path: Use the --lua-package-path option when starting the Splash server. This option accepts a semicolon-separated list of paths. Each path must include a ? character, which Lua replaces with the module name.
    2. Allowlist modules: Because Splash uses a Lua sandbox by default, you must explicitly allow your modules using the --lua-sandbox-allowed-modules option. This option accepts a semicolon-separated list of module names.
    3. Load modules: Use the standard Lua require function within your script to load the allowed modules.

    Note: Sandbox restrictions are not applied within custom Lua modules, meaning they have access to full Lua capabilities (like the os module), which should be used with caution to avoid blocking the event loop.

  7. Handle errors in Splash Lua scripts

    master

    Splash uses two error reporting conventions:

    1. Developer Errors: For incorrect function arguments, Splash raises an exception. If an exception in the main function is unhandled, Splash returns an HTTP 400 response.
    2. Runtime Errors: For errors outside developer control (e.g., a non-responding website), functions return an ok, reason pair.

    Strategies:

    • Manual Exceptions: Use the Lua error("message") function to raise an exception and trigger an HTTP 400 response.
    • Handling Status Flags: Check the ok boolean returned by methods. If ok is false, the second return value contains the error reason.
    • Using pcall: Use Lua's pcall to catch exceptions and prevent Splash from returning an HTTP 400.
    • Using assert: To convert a status flag error into an exception (triggering an HTTP 400), use assert(splash:method(...)).
    -- Manual error handling
local ok, msg = splash:go("http://example.com")
if not ok then
    -- handle error somehow, e.g.
    error(msg)
end

-- Shortcut using assert to trigger HTTP 400 on failure
assert(splash:go("http://example.com"))

-- Manual exception
error("A message to be returned in a HTTP 400 response")

  8. Mount custom configuration folders in Docker

    master

    You can use the Docker -v option to share local directories containing configuration files with the Splash container.

    Common mount points include:

    • Request filters: /etc/splash/filters
    • Proxy profiles: /etc/splash/proxy-profiles
    • JavaScript profiles: /etc/splash/js-profiles
    • Lua modules: /etc/splash/lua_modules

    Example: Mounting filters, proxy profiles, and JS profiles:

    docker run -p 8050:8050 \
    -v <my-proxy-profiles-dir>:/etc/splash/proxy-profiles \
    -v <my-js-profiles-dir>:/etc/splash/js-profiles \
    scrapinghub/splash

    Example: Mounting Lua modules with sandbox permissions: If using the default Lua sandbox, you must list allowed modules using the --lua-sandbox-allowed-modules option:

    docker run -p 8050:8050 \
    -v <my-lua-modules-dir>:/etc/splash/lua_modules \
    scrapinghub/splash \
    --lua-sandbox-allowed-modules 'module1;module2'

    Warning: Folder sharing (-v) may have issues on OS X and Windows.

    docker run -p 8050:8050 -v <my-filters-dir>:/etc/splash/filters scrapinghub/splash

  9. Use the Splash object in custom Lua modules

    master

    When writing modules that need to interact with the Splash instance (e.g., for waiting or executing JS), you have two primary patterns:

    1. Passing splash as an argument

    This is the most explicit and composable method. Define functions that accept the splash object as their first parameter.

    2. Monkey-patching the Splash class

    You can add methods directly to the splash object by requiring the internal splash module and adding methods to its class. This allows for a more compact syntax in your scripts (e.g., splash:my_method()).

    -- Pattern 1: Passing splash as an argument
local utils = {}

function utils.wait_for(splash, condition)
    while not condition() do
        splash:wait(0.05)
    end
end

return utils

-- Usage:
local utils = require("utils")
function main(splash)
    splash:go(splash.args.url)
    utils.wait_for(splash, function()
       return splash:evaljs("document.querySelector('h1') != null")
    end)
    return splash:html()
end

-- Pattern 2: Monkey-patching the Splash class
local Splash = require("splash")

function Splash:wait_for(condition)
    while not condition() do
        self:wait(0.05)
    end
end

-- Usage:
require("wait_for")
function main(splash)
    splash:go(splash.args.url)
    splash:wait_for(function()
       return splash:evaljs("document.querySelector('h1') != null")
    end)
    return splash:html()
end

  10. Configure and use Request Filters (Adblock Plus)

    master

    Splash can filter network requests using Adblock Plus rules (e.g., to block ads, tracking, or specific file types like fonts).

    1. Enable support: Start the Splash server with the --filters-path=<path> flag.
    2. Create filters: Place .txt files containing Adblock Plus rules in the specified folder.
    3. Apply filters: Use the filters=<filter_name> query parameter. You can provide multiple filters separated by commas.

    Note: If a default.txt file exists in the folder, it is applied automatically. Use filters=none to disable default filters.

    Server Startup Example:

    python3 -m splash.server --filters-path=/etc/splash/filters

    Request Example:

    # Apply a specific filter
curl 'http://localhost:8050/render.png?url=http://domain.com/page.html&filters=nofonts'

# Apply multiple filters
curl 'http://localhost:8050/render.png?url=http://domain.com/page.html&filters=nofonts,easylist'

    Performance Tip: For large filter lists (like EasyList), ensure the pyre2 library is installed to avoid significant performance degradation.

  11. Convert Splash Notebook scripts to HTTP API

    master

    After developing a script in a Splash Notebook, you can convert it for use with the Splash HTTP API by downloading the file as .lua and wrapping/modifying the code based on the endpoint you are using.

    For the /run endpoint

    Add a return statement at the end of your script to return the final result.

    For the /execute endpoint

    Wrap your code inside a function main(splash) and include a return statement inside that function.

    Note: Scripts in Jupyter notebooks are not sandboxed, regardless of your Splash sandbox settings. Some functions available in the notebook might be unavailable in the HTTP API if sandboxing is enabled.

  12. Enable Cross-Domain JavaScript Access

    master

    By default, Splash prevents JavaScript from accessing iframes with a different security origin. To allow this (e.g., to extract HTML from an iframe), start Splash with the --js-cross-domain-access option.

    Warning: This is a security risk as it may expose sensitive information like cookies. It is OFF by default.

    docker run -it -p 8050:8050 scrapinghub/splash --js-cross-domain-access