DOCSEARCHbyAlgolia
Submit Docs

shot-scraper

repository·main·Indexed 25 days ago

https://github.com/simonw/shot-scraper

A CLI utility built on Playwright for taking screenshots of websites, recording video demos, and scraping sites using JavaScript. It includes features for dumping Chromium accessibility trees, capturing HTTP Archive (HAR) files, extracting resources, and retrieving rendered HTML for specific CSS selectors. Version 1.11 supports authentication context saving for sites requiring login and integration with GitHub Actions for automated screenshot workflows.

Tokens
19.6K
Snippets
47
Records
77
Agent score
81%

What's inside shot-scraper

  1. Configure storyboard top-level settings

    main

    A storyboard is a YAML file containing global configuration for the video recording session. Key top-level keys include:

    • output: The WebM filename (can be overridden with -o/--output).
    • url: The starting URL, domain, or local HTML path.
    • viewport: A mapping with width and height (defaults to 1280x720).
    • cursor: Configuration for the cursor. Can be true, false, or a mapping with visible, clicks, color, size, and click_size.
    • wait_for: A Playwright selector or text selector to wait for after the initial page loads.
    • wait_for_url: A URL pattern to wait for.
    • javascript: JavaScript to run once after the initial page loads but before any scenes start.
    • sh, python, server: Commands to run at the session level.
    • scenes: A required list of scene objects.
    output: demo.webm
url: https://shot-scraper.datasette.io/en/stable/
viewport:
  width: 1280
  height: 720
cursor:
  visible: true
  clicks: true
  color: "#ff4f00"
  size: 18
  click_size: 44
wait_for: "text=Quick start"

scenes:
- name: Documentation home
  do:
  - pause: 1

  2. Run custom code between storyboard steps

    main

    You can execute code at different levels of the storyboard to prepare environments or manipulate the page:

    1. Top-level sh: or python:: Runs once before the server starts and the browser opens.
    2. Scene-level sh: or python:: Runs before the scene opens a page or runs its do actions.
    3. Action-level sh: or python:: Inside a do: list, these run between other actions.
    4. Action-level javascript: or js:: Inside a do: list, this runs code inside the browser page context (can modify DOM, localStorage, etc.).

    Note: If any sh: or python: command exits with a non-zero status, the recording stops with an error.

    scenes:
- name: Update then reload
  do:
  - sh: echo "<h1>Updated</h1>" > /tmp/demo-root/index.html
  - open: http://localhost:8000/
  - wait_for: 'h1:has-text("Updated")'

  3. Configure scene settings and actions

    main

    Each scene in a storyboard defines a specific segment of the video. A scene can have its own setup and a list of actions to perform.

    Scene Keys

    • name: A label shown in the progress output.
    • open: A URL or path to navigate to at the start of this specific scene.
    • wait_for / wait_for_url: Specific conditions to meet before starting actions.
    • sh / python: Commands to run before the scene's actions.
    • do: A list of browser/page actions to execute.

  4. Install the required Playwright browsers

    main

    After installing the package, you must install the Playwright browser engine (Chromium by default) and ffmpeg to enable screenshot capabilities. Run the following command once:

    shot-scraper install

    If you need to use browsers other than the default Chromium, specify the browser using the -b or --browser flag.

  5. Save a web page to PDF

    main

    Use the shot-scraper pdf command to save a PDF version of a web page, mimicking the Print -> Save to PDF functionality in Chromium. You can provide either a URL or a path to a local file on disk.

    By default, shot-scraper will save the file using the name of the target resource. Use the -o or --output flag to specify a custom filename.

  6. Record videos using shot-scraper video

    main

    Use the shot-scraper video command to record a WebM video from a YAML storyboard. A storyboard defines the starting URL, viewport settings, cursor visibility, and a sequence of scenes that perform actions on a webpage.

    Basic Usage

    shot-scraper video storyboard.yml

    To also generate an MP4 version (requires ffmpeg), use the --mp4 flag:

    shot-scraper video storyboard.yml -o demo.webm --mp4

  8. Execute custom JavaScript on a page

    main

    Modify the page after it loads but before the screenshot is taken using --javascript (for inline strings) or --js-file (for external files).

    --js-file supports:

    • - to read from standard input.
    • gh:username/repo/script.js to load a script directly from GitHub.

  9. Extract resources from HAR files

    main

    You can automatically extract all resources (images, scripts, CSS, etc.) from a HAR file into a directory using the --extract or -x flag.

    When using --extract, shot-scraper creates a directory named after the domain (or the base name of your output file) containing the assets. Files are given extensions based on their content-type (e.g., /api/data becomes data.json).

    You can combine --extract with --zip to create a .har.zip file while simultaneously extracting the assets into a directory.

  10. Run JavaScript from files or GitHub

    main

    Instead of passing long strings in the CLI, you can load JavaScript from files or remote repositories.

    From a local file

    shot-scraper javascript datasette.io -i script.js

    From standard input

    echo "document.title" | shot-scraper javascript datasette.io

    From GitHub

    Use the gh: prefix to load scripts from a public GitHub repository. If the script is in a repository named shot-scraper-scripts, you can use a shorthand convention:

    # Full path
shot-scraper javascript datasette.io -i gh:simonw/shot-scraper-scripts/readability.js

# Shorthand (omitting repo name and .js extension)
shot-scraper javascript datasette.io -i gh:simonw/readability
    shot-scraper javascript datasette.io -i gh:simonw/readability

  11. Dump the final HTML of a page

    main

    The shot-scraper html command outputs the final HTML of a page after all JavaScript has executed. This is useful for inspecting the rendered state of a web application.

    To output to a file instead of stdout, use the -o or --output flag.

    shot-scraper html https://datasette.io/
# To save to a file:
shot-scraper html https://datasette.io/ -o index.html

  12. Run complex JavaScript logic and async/await

    main

    Multiple Statements

    To run multiple statements, use the arrow function syntax () => { ... }. You can include logic like error handling or array transformations:

    shot-scraper javascript https://www.example.com/ "
() => {
  var paragraphs = document.querySelectorAll('p');
  if (paragraphs.length == 0) {
    throw 'No paragraphs found';
  }
  return Array.from(paragraphs, el => el.innerText);
}"

    Async/Await and External Modules

    You can pass an async function to use await. This is particularly useful for importing modules from external URLs (e.g., via jsdelivr):

    shot-scraper javascript \
  https://simonwillison.net/2022/Mar/14/scraping-web-pages-shot-scraper/ "
async () => {
  const readability = await import('https://cdn.jsdelivr.net/npm/@mozilla/readability@0.6.0/+esm');
  return (new readability.Readability(document)).parse();
}"

    Using Promises and Delays

    To delay execution (e.g., waiting for an animation), return a Promise that resolves with the desired data:

    shot-scraper javascript datasette.io "
new Promise(done => setInterval(
  () => {
    done({
      title: document.title,
      tagline: document.querySelector('.tagline').innerText
    });
  }, 1000
));"