mechanize Documentation

Avoid manual cookie extraction

When using mechanize.urlopen() or OpenerDirector.open(), the module automatically handles cookie extraction and addition. Do not manually call .extract_cookies() or .add_cookie_header() on a cookie object if you are using these high-level functions, as it can lead to errors.

Persisting cookies with ignore_discard

When using .save() to write cookies to a file, or .load()/.revert() to read them, single-session cookies will expire unless you explicitly set the ignore_discard argument to True. If you find cookies disappearing after a save/load cycle, ensure this argument is used.

JavaScript limitations

Mechanize does not support cookies set via JavaScript.

import mechanize

cj = mechanize.LWPCookieJar()
opener = mechanize.build_opener(mechanize.HTTPCookieProcessor(cj))
mechanize.install_opener(opener)

r = mechanize.urlopen("http://foobar.com/")
# Use ignore_discard and ignore_expires to prevent session cookies from being lost
cj.save("/some/file", ignore_discard=True, ignore_expires=True)

To perform stateful web browsing, instantiate a mechanize.Browser() object. You can navigate to URLs, follow links using regular expressions or link objects, select HTML forms, and fill them using dictionary-like syntax before submitting. The browser maintains a history, allowing you to use .back() and .reload() to navigate through previous responses.

import re
import mechanize

br = mechanize.Browser()
br.open("http://www.example.com/")
# follow second link with element text matching regular expression
response1 = br.follow_link(text_regex=r"cheese\s*shop", nr=1)
print(br.title())
print(response1.geturl())
print(response1.info())  # headers
print(response1.read())  # body

br.select_form(name="order")
# Browser passes through unknown attributes (including methods)
# to the selected HTMLForm.
br["cheeses"] = ["mozzarella", "caerphilly"]
# Submit current form. Browser calls .close() on the current response on navigation
response2 = br.submit()

# print currently selected form (don't call .submit() on this, use br.submit())
print(br.form)

response3 = br.back()  # back to cheese shop (same data as response1)
# the history mechanism returns cached response objects
# we can still use the response, even though it was .close()d
response3.get_data()  # like .seek(0) followed by .read()
response4 = br.reload()  # fetches from server

for form in br.forms():
    print(form)
# .links() optionally accepts the keyword args of .follow_/.find_link()
for link in br.links(url_regex="python.org"):
    print(link)
    br.follow_link(link)  # takes EITHER Link instance OR keyword args
    br.back()

You can customize the behavior of mechanize.Browser by configuring proxies, authentication, and request headers.

• Proxies: Use set_proxies() to define HTTP/FTP proxy mappings. Use add_proxy_password() for proxy authentication.
• Authentication: Use add_password() to provide credentials for specific website URLs.
• Headers: Use finalize_request_headers (a lambda or function) to inject or modify headers on all outgoing requests.
• Security: Use set_ca_data() with an unverified SSL context to bypass SSL certificate verification.

import mechanize
import ssl

br = mechanize.Browser()
# Explicitly configure proxies
br.set_proxies({"http": "joe:password@myproxy.example.com:3128",
                "ftp": "proxy.example.com",
                })
# Add HTTP Basic/Digest auth username and password for HTTP proxy access.
br.add_proxy_password("joe", "password")
# Add HTTP Basic/Digest auth username and password for website access.
br.add_password("http://example.com/protected/", "joe", "password")
# Add an extra header to all outgoing requests
br.finalize_request_headers = lambda request, headers: headers.__setitem__(
  'My-Custom-Header', 'Something')
# Do not verify SSL certificates
br.set_ca_data(context=ssl._create_unverified_context(cert_reqs=ssl.CERT_NONE))