pywebcopy Documentation

The MultiParser object wraps multiple scraping libraries (BeautifulSoup, lxml, pyquery, and requests-html) into a single interface. This allows you to switch between different parsing strategies (like CSS selectors or XPath) using a single object without re-initializing different libraries.

To use it, pass the raw HTML content and the encoding (which can be auto-detected from a requests response) to the MultiParser constructor.

from pywebcopy.parsers import MultiParser
import requests

req = requests.get('http://google.com')
html = req.content
encoding = req.encoding

# Initialize the unified parser
wp = MultiParser(html, encoding)

Before using PyWebCopy, be aware of its technical constraints:

• No JavaScript Parsing: PyWebCopy does not include a virtual DOM or JavaScript engine. It cannot discover links or content that are dynamically generated via JavaScript.
• HTTP Response Only: It only downloads what the HTTP server returns; it cannot download the raw server-side source code.
• Dynamic Sites: Advanced data-driven websites may not function correctly once copied offline.

Install the pywebcopy package using pip to start cloning or archiving web pages and sites.

$ pip install pywebcopy

PyWebCopy uses a requests.Session object for HTTP activity. You can interact with forms by retrieving them from a page object and submitting them. This is useful for accessing password-protected content.

from pywebcopy.configs import get_config

config = get_config('http://httpbin.org/')
wp = config.create_page()
wp.get(config['project_url'])

# Access and fill a form
form = wp.get_forms()[0]
form.inputs['email'].value = 'bar'
form.inputs['password'].value = 'baz'

# Submit and continue crawling
wp.submit_form(form)
wp.get_links()

Use the save_webpage function to download a specific URL and its associated resources (images, stylesheets, etc.) to a local folder. The library automatically remaps links to match the local path.

from pywebcopy import save_webpage

save_webpage(
      url="https://httpbin.org/",
      project_folder="E://savedpages//",
      project_name="my_site",
      bypass_robots=True,
      debug=True,
      open_in_browser=True,
      delay=None,
      threaded=False,
)

MultiParser provides direct access to requests-html features for simplified searching:

• Native XPath: Use .xpath(selector) to return requests_html.Element objects.
• Text-based Search: Use .find(selector, containing='text') to select only elements that contain specific text.

# Native XPath support
elements = wp.xpath('a')

# Find elements containing specific text
results = wp.find('a', containing='kenneth')

Use the save_website function to crawl and download an entire website.

Warning: Crawling an entire site can overload the target server; use with caution.

from pywebcopy import save_website

save_website(
      url="https://httpbin.org/",
      project_folder="E://savedpages//",
      project_name="my_site",
      bypass_robots=True,
      debug=True,
      open_in_browser=True,
      delay=None,
      threaded=False,
)

Once a MultiParser instance is created, you can access the underlying libraries through specific attributes:

• BeautifulSoup: Access via .bs4 to use standard BeautifulSoup methods like find_all().
• lxml: Access via .lxml to use lxml methods (e.g., xpath()).
• PyQuery: Access via .pq to use PyQuery CSS selector methods like select().

# BeautifulSoup
links = wp.bs4.find_all('a')

# lxml
elements = wp.lxml.xpath('//a')

# PyQuery
results = wp.pq.select('selector')

The following options and actions are available when using the pywebcopy CLI.

CLI Actions:
  -p, --page          Quickly saves a single page.
  -s, --site          Saves the complete site.
  -t, --tests         Runs tests for this library.

Options:
  --version           show program's version number and exit
  -h, --help          show this help message and exit
  --url=URL           url of the entry point to be retrieved.
  --location=LOCATION Location where files are to be stored.
  -n NAME, --name=NAME Project name of this run.
  -d DELAY, --delay=DELAY Delay between consecutive requests to the server.
  --bypass_robots     Bypass the robots.txt restrictions.
  --threaded          Use threads for faster downloading.
  -q, --quite         Suppress the logging from this library.
  --pop               open the html page in default browser window after finishing the task.

You can use pywebcopy directly from the terminal without writing Python code. Use the -m pywebcopy module flag to execute commands.

# Get help
$ python -m pywebcopy --help

# Save a single page
$ python -m pywebcopy -p --url=URL --location=LOCATION --name=NAME

# Save a complete site
$ python -m pywebcopy -s --url=URL --location=LOCATION --name=NAME

# Run tests
$ python -m pywebcopy -t