repository·master·Indexed 23 days ago
https://github.com/rushter/selectolaxA high-performance HTML5 parser written in Cython providing fast CSS selector-based extraction. It features the modern Lexbor backend (preferred) and a legacy Modest backend. Key capabilities include parsing full documents or fragments, DOM tree navigation, element manipulation via .decompose() and .unwrap_tags(), and advanced text matching using the :lexbor-contains pseudo-class.
The selectolax.lexbor module provides high-performance HTML parsing capabilities using the Lexbor engine. It is composed of three primary classes:
LexborHTMLParser: The main entry point used to parse HTML content into a document tree.LexborNode: Represents an individual node within the parsed document tree.LexborSelector: Provides the interface for finding nodes using CSS selectors.selectolax.parser module provides the core classes for parsing HTML documents and interacting with the resulting DOM tree. The primary workflow involves using HTMLParser to parse a raw HTML string, which returns a tree of Node objects that can be queried using Selector objects or built-in methods.Selectolax provides two backends for parsing HTML:
selectolax.lexbor.LexborHTMLParser to access it.selectolax.modest.ModestHTMLParser (implied) for compatibility requirements.As of 2024, developers should default to the Lexbor backend for all new projects.
Selectolax supports two underlying engines for parsing:
Selectolax supports specialized Lexbor pseudo-classes for advanced text matching within CSS selectors:
:lexbor-contains("text" i): Performs a case-insensitive search for the specified text.:lexbor-contains("text"): Performs a case-sensitive search for the specified text.html = '<div><p>hello </p><p id="main">lexbor is AwesOme</p></div>'
parser = LexborHTMLParser(html)
# Case-insensitive search
results_ci = parser.css('p:lexbor-contains("awesome" i)')
print(f"Case-insensitive results: {len(results_ci)}")
# Case-sensitive search
results_cs = parser.css('p:lexbor-contains("AwesOme")')
print(f"Case-sensitive results: {len(results_cs)}")
print(f"Matching text: {results_cs[0].text()}")Selectolax provides three primary ways to initialize a parser using the LexborHTMLParser backend. It is recommended to use the Lexbor backend for better performance and features.
LexborHTMLParser(html) to parse a complete HTML document. It automatically adds missing <html>, <head>, and <body> tags.LexborHTMLParser(html, is_fragment=True) for snippets or partial HTML. This behaves like a browser's DocumentFragment and drops <html>, <head>, and <body> tags if they are present.parser.create_node("tag_name").from selectolax.lexbor import LexborHTMLParser
html = "<body>...</body>"
fragment = "<div>...</div>"
# Parse HTML as a full document
parser = LexborHTMLParser(html)
# Parse HTML as a fragment
frag_parser = LexborHTMLParser(html, is_fragment=True)
# Create a new node for `parser`.
node = parser.create_node("div")To remove unwanted or dangerous elements (like <script> or <style> tags) from the DOM, iterate over the selected nodes and call the .decompose() method on each.
dirty_html = '''
<div>
<p>Good content</p>
<script>alert('xss')</script>
<style>body { color: red; }</style>
<p>More content</p>
</div>
'''
parser = LexborHTMLParser(dirty_html)
# Remove unwanted tags
for tag in parser.css('script, style'):
tag.decompose()
print(parser.body.html)When parsing forms, you can extract metadata from the <form> tag and individual field data from <input>, <select>, and <textarea> elements using the .attrs dictionary and .text() method.
attrs.get('type'), attrs.get('name'), and attrs.get('value'). For checkboxes, check if 'checked' is in attrs.<option> children to get their values and text..text() to get the content.form_html = """
<form id="contact-form" method="post" action="/submit">
<div class="form-group">
<label for="name">Name:</label>
<input type="text" id="name" name="name" value="John Doe" required>
</div>
<div class="form-group">
<label for="email">Email:</label>
<input type="email" id="email" name="email" placeholder="john@example.com">
</div>
<div class="form-group">
<label for="country">Country:</label>
<select id="country" name="country">
<option value="us">United States</option>
<option value="ca" selected>Canada</option>
<option value="uk">United Kingdom</option>
</select>
</div>
<div class="form-group">
<label>
<input type="checkbox" name="newsletter" checked> Subscribe to newsletter
</label>
</div>
<div class="form-group">
<label for="message">Message:</label>
<textarea id="message" name="message" rows="4">Hello there!</textarea>
</div>
<button type="submit">Submit</button>
</form>
"""
parser = LexborHTMLParser(form_html)
# Extract form metadata
form = parser.css_first('form')
print(f"Form ID: {form.attrs.get('id')}")
print(f"Form method: {form.attrs.get('method')}")
print(f"Form action: {form.attrs.get('action')}")
# Extract input fields
print("\nInput fields:")
for input_field in parser.css('input'):
field_type = input_field.attrs.get('type', 'text')
name = input_field.attrs.get('name')
value = input_field.attrs.get('value', '')
checked = 'checked' in input_field.attrs
print(f" {name} ({field_type}): {value} {'[checked]' if checked else ''}")
# Extract select options
print("\nSelect fields:")
for select in parser.css('select'):
name = select.attrs.get('name')
print(f" {name}:")
for option in select.css('option'):
value = option.attrs.get('value')
text = option.text()
selected = 'selected' in option.attrs
print(f" {value}: {text} {'[selected]' if selected else ''}")
# Extract textarea
print("\nTextarea fields:")
for textarea in parser.css('textarea'):
name = textarea.attrs.get('name')
content = textarea.text()
print(f" {name}: {content}")You can install the standard version of selectolax from PyPI using pip. If you encounter compilation errors (common when installing older versions on newer Python versions), install the cython extra to ensure a successful build.
To install from source for development, clone the repository and follow the setup instructions.
You can navigate the DOM tree using .prev and .next properties on a node.
Note: When navigating between elements in HTML that contains whitespace or newlines between tags, these whitespace characters are often represented as text nodes. You may need to call .prev or .next multiple times to skip these text nodes and reach the next actual element.
html = """
<nav>
<a href="/" >Home</a>
<a href="/about">About</a>
<a href="/contact" class="active">Contact</a>
<a href="/blog">Blog</a>
</nav>
"""
parser = LexborHTMLParser(html)
active_link = parser.css_first("a.active")
if active_link:
print(f"Active link: {active_link.text()}")
# We need to call it twice, because there are text nodes (spaces and new lines) between <a> elements
if active_link.prev:
print(f"Previous link: {active_link.prev.prev.text()}")
if active_link.next:
print(f"Next link: {active_link.next.next.text()}")You can extract attributes like href for links and src for images by using CSS selectors that target those specific attributes (e.g., a[href] or img[src]). Access the values via the .attrs dictionary.
html = '''
<div>
<a href="https://example.com">Link 1</a>
<a href="/page2">Link 2</a>
<img src="image1.jpg" alt="Image 1">
<img src="image2.png" alt="Image 2">
</div>
'''
parser = LexborHTMLParser(html)
# Extract all links
for link in parser.css('a[href]'):
print(f"Link: {link.text()} -> {link.attrs['href']}")
# Extract all images
for img in parser.css('img[src]'):
print(f"Image: {img.attrs.get('alt', 'No alt')} -> {img.attrs['src']}")