Beautiful Soup 4

To begin parsing HTML or XML, import BeautifulSoup from bs4 and pass your document string to the constructor. This creates a BeautifulSoup object representing the document as a nested data structure. You can use .prettify() to print the document with a clean, indented structure.

from bs4 import BeautifulSoup

html_doc = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
</body>
</html>
"""

soup = BeautifulSoup(html_doc)
print(soup.prettify())

You can install Beautiful Soup 4 using several methods depending on your environment:

Using pip (Recommended)

Install via PyPi using pip:

pip install beautifulsoup4

Using apt-get (Debian/Ubuntu)

On recent Debian or Ubuntu Linux systems, use the system package manager:

apt-get install python-bs4

Using easy_install

easy_install beautifulsoup4

Manual Installation

If you don't have a package manager, download the source tarball and run:

python setup.py install

Note: Always install beautifulsoup4 rather than BeautifulSoup (which refers to the older version 3) when writing new code.

Once you have a BeautifulSoup object, you can navigate the tree using several methods:

Attribute-style navigation

Access tags directly by their name:

• soup.title: Returns the <title> tag.
• soup.p: Returns the first <p> tag.
• soup.title.name: Returns the name of the tag (e.g., 'title').
• soup.title.string: Returns the text content inside the tag.
• soup.title.parent.name: Navigates to the parent tag and returns its name.

Searching for tags

• soup.find_all('a'): Returns a list of all <a> tags.
• soup.find(id="link3"): Finds the first tag with a specific attribute, such as id.

Accessing tag attributes

Access attributes like a dictionary:

• soup.p['class']: Returns the value of the class attribute.

# Example navigation
print(soup.title.string)
print(soup.find_all('a'))
print(soup.find(id="link3"))

Common data extraction tasks can be performed using loops and built-in methods:

Extracting all URLs from <a> tags

Iterate through all <a> tags and use .get('href') to retrieve the link destination.

Extracting all text

Use .get_text() to strip all HTML tags and return only the visible text content of the document.

# Extract all URLs
for link in soup.find_all('a'):
    print(link.get('href'))

# Extract all text
print(soup.get_text())