MechanicalSoup Documentation

MechanicalSoup is a Python library for automating website interaction. It uses Requests for HTTP sessions and BeautifulSoup for document navigation. It automatically handles cookies, follows redirects, and can submit forms, but it does not support JavaScript.

A common pattern is to use mechanicalsoup.StatefulBrowser to maintain a session, open a URL, select a form, fill in fields, and submit it.

MechanicalSoup is built upon and integrates with the following libraries. You may need to consult their documentation for advanced HTTP handling or HTML manipulation tasks:

• Requests: Handles the HTTP layer.
• BeautifulSoup: Handles HTML parsing and manipulation.

To install from a local clone of the repository (which installs the version in your current working directory), follow these steps:

1. Clone the repository.
2. Navigate into the directory.
3. Run the setup script.

git clone https://github.com/MechanicalSoup/MechanicalSoup.git
cd MechanicalSoup
python setup.py install

To see how MechanicalSoup is used in production web scraping scenarios, you can examine the source code of these projects:

• Chamilo Tools
• gmusicapi: An unofficial API for Google Play Music.
• PatZilla: Patent information research tool.

Documentation is generated from docstrings in *.py files and .rst files in the docs/ directory. You can build the Sphinx documentation locally using the following steps:

pip install sphinx
python setup.py build_sphinx
firefox docs/_build/html/index.html

To avoid warnings and ensure consistent behavior across different systems, specify an HTML parser in the soup_config argument when initializing the browser.

mechanicalsoup.StatefulBrowser(soup_config={'features': 'lxml', '...': '...'})

# If lxml is not installed:
mechanicalsoup.StatefulBrowser(soup_config={'features': 'parser.html', ...})

If you need help or want to join discussions, you can find the MechanicalSoup community at:

• StackOverflow: Search for the mechanicalsoup tag.
• Gitter: Join the MechanicalSoup Lobby.

To start a session that maintains state (like cookies and the current URL) across multiple requests, use the mechanicalsoup.StatefulBrowser class. You can customize the browser behavior (such as user-agent or HTML parser) via the __init__ method.

import mechanicalsoup
browser = mechanicalsoup.StatefulBrowser()

To set up a development environment, install the necessary requirements for both the package and the test suite, then execute the tests using pytest.

pip install -r requirements.txt -r tests/requirements.txt
pytest

To install the latest stable version of MechanicalSoup, use pip to download it from the Python Package Index (PyPI).

pip install MechanicalSoup