DOCSEARCHbyAlgolia
Submit Docs

w3lib Documentation

repository·master·Indexed 19 days ago

https://github.com/scrapy/w3lib

A Python library providing a collection of web-related utility functions for HTML manipulation, URL sanitization, and HTTP header processing. It includes tools for converting HTML bytes to Unicode, removing HTML tags and comments, managing URL query parameters, canonicalizing URLs, and handling HTTP Basic Authentication headers. Requires Python 3.10+.

Tokens
5.4K
Snippets
29
Records
32
Agent score
64%

What's inside w3lib

  1. Overview of w3lib web utility functions

    master

    w3lib is a Python library providing various web-related utility functions, including:

    • HTML Manipulation: Removing comments or tags from HTML snippets, extracting base URLs, and translating entities in HTML strings.
    • HTTP Utilities: Converting raw HTTP headers to dictionaries (and vice-versa) and constructing HTTP authentication headers.
    • URL Handling: Sanitizing URLs (browser-like behavior) and extracting arguments from URLs.
    • Encoding: Converting HTML pages to unicode.

  5. Create a safe download URL with safe_download_url()

    master

    Use safe_download_url() to generate a URL suitable for downloading. This function calls safe_url_string(), strips any fragments, and normalizes the path. If the path attempts to navigate outside the document root (e.g., via ../), it is modified to stay within the root.

    import w3lib.url
# Normalizes path and removes fragments
download_url = w3lib.url.safe_download_url('http://example.com/dir/../file.html#section')
# Result: 'http://example.com/file.html'

  7. Extract encoding from HTML meta tags

    master

    Use html_body_declared_encoding to find the encoding specified within the HTML body (e.g., in <meta> tags or XML declarations). It searches the first 4096 bytes of the provided string or bytes and stops if a <body> tag is encountered. Returns the encoding name or None.

    import w3lib.encoding
html_body_declared_encoding("<html><head><meta charset='utf-8'></head><body></body></html>")
# returns 'utf-8'

  8. Convert raw HTML bytes to Unicode

    master

    The html_to_unicode function is a high-level utility that converts raw HTML bytes into a Unicode string by attempting to guess the encoding using the following priority:

    1. BOM (Byte Order Mark): If present, it is used and stripped.
    2. HTTP Content-Type header: If provided.
    3. Meta or XML tag declarations: Found in the HTML body.
    4. Auto-detection: If an auto_detect_fun is provided.
    5. Default encoding: Specified by default_encoding (defaults to 'utf8').

    This method is robust and will not fail on decoding errors; instead, it inserts the Unicode replacement character � for invalid sequences.

    To use an external library like chardet for auto-detection, pass it via auto_detect_fun.

    import w3lib.encoding

# Example with meta tag detection
html_to_unicode(
    content_type_header=None, 
    html_body_str=b"<html><head><meta charset='UTF-8' /></head><body></body></html>"
)
# returns ('utf-8', '<html><head><meta charset="UTF-8" /></head><body></body></html>')

# Example using chardet for auto-detection
import chardet
html_to_unicode(
    content_type_header=None,
    html_body_str=b'\xec\x9a\x9c...',
    auto_detect_fun=lambda x: chardet.detect(x).get('encoding')
)

  9. Convert bytes to unicode with to_unicode()

    master

    Use to_unicode(text, encoding, errors) to ensure you have a str object. If the input is already a str, it is returned unchanged. If the input is bytes, it is decoded using the specified encoding (defaults to 'utf-8') and errors handling (defaults to 'strict').

    from w3lib.util import to_unicode

# From bytes to str
result = to_unicode(b'hello world')
# result is 'hello world' (str)

# Handling specific encoding
result = to_unicode(b'\xe4\xbd\xa0\xe5\xa5\xbd', encoding='utf-8')
# result is '你好' (str)

  10. Sanitize a URL with safe_url_string()

    master

    Use safe_url_string() to return a URL that is compatible with a wide range of web browsers and servers. It ensures compliance with the URL living standard, RFC 3986, RFC 2396, and RFC 2732 (as interpreted by Java 8's java.net.URI).

    If a bytes URL is provided, it is converted to str using the specified encoding (defaulting to 'utf-8'). The path_encoding parameter (default 'utf-8') controls how the path component is encoded and quoted. If quote_path is True (default), the path component is quoted; otherwise, it is left as is.

    import w3lib.url
# Returns a sanitized version of the URL
safe_url = w3lib.url.safe_url_string('http://example.com/path with spaces/index.html')

  11. Remove tags and their content with `remove_tags_with_content`

    master

    Removes specific HTML tags along with everything contained between their opening and closing tags.

    • which_ones: An iterable of tag names to remove (including content). If empty, the string is returned unmodified.
    import w3lib.html
doc = '<div><p><b class="bold">Bold text</b> <a href="#">Link</a></p></div>'
w3lib.html.remove_tags_with_content(doc, which_ones=('b',))
# Returns: '<div><p>  <a href="#">Link</a></p></div>'

  12. Read Byte Order Mark (BOM)

    master

    Use read_bom to detect a Byte Order Mark in a byte sequence. It returns a tuple of (encoding, bom_bytes). If no BOM is detected, it returns (None, None). Supported encodings include UTF-32 (BE/LE), UTF-16 (BE/LE), and UTF-8.

    import w3lib.encoding
w3lib.encoding.read_bom(b'\xfe\xff\x6c\x34')
# returns ('utf-16-be', b'\xfe\xff')