repository·master·Indexed 43 days ago
https://github.com/scrapy/cssselectA Python library that parses CSS3 selectors and translates them into XPath 1.0 expressions. It provides GenericTranslator for XML and HTMLTranslator for HTML documents, enabling CSS-style querying in environments that rely on XPath engines like lxml.
cssselect is a Python library designed to bridge the gap between CSS selectors and XPath.
Core Functionality:
It takes CSS3 selectors as input and translates them into XPath 1.0 expressions.
Workflow:
div.content).cssselect to convert it to an XPath expression.lxml) to extract matching elements from an XML or HTML document.The library provides two main translator classes:
HTMLTranslator: Use this for HTML documents. It includes specialized logic for HTML-specific element types and attributes, making translations more accurate for web scraping.GenericTranslator: Use this for generic XML documents. It follows a more standard approach that does not assume HTML semantics.Note that certain pseudo-classes like :link, :enabled, :disabled, and :checked depend on document knowledge and will only match elements when using HTMLTranslator. In GenericTranslator, they will never match.
You can customize how CSS selectors are translated into XPath by subclassing HTMLTranslator or GenericTranslator and overriding their internal methods. This allows you to implement custom logic for specific pseudo-classes without modifying the library source.
Warning: The customization API (the set of methods in translation classes) is not yet stable and may change in future versions.
namespace-prefix|element syntax. This maps one-to-one to the XPath namespace-prefix:element syntax. The mapping of prefixes to specific namespace URIs is handled by the underlying XPath implementation (e.g., lxml).To convert CSS selectors into XPath expressions, use either HTMLTranslator for HTML documents or GenericTranslator for generic XML documents. HTMLTranslator provides more specialized translations for HTML-specific elements and attributes.
The resulting XPath expression can then be used with an XPath engine like lxml to query documents.
from cssselect import GenericTranslator, SelectorError
try:
expression = GenericTranslator().css_to_xpath('div.content')
except SelectorError:
print('Invalid selector.')
print(expression)
# Output: descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' content ')]You can install the cssselect library using pip. This library allows you to parse CSS3 selectors and translate them into XPath 1.0 expressions, which can then be used with engines like lxml to query XML or HTML documents.
pip install cssselectThe XPathExpr class is an internal representation of a partially constructed XPath expression. While primarily used by the translators, it defines how paths, elements, and conditions are composed.
Core attributes:
path: The structural path (e.g., /div/p).element: The current node test (e.g., * or div).condition: The predicate logic (e.g., [@class='foo']).Key methods for building expressions:
add_condition(condition, conjunction='and'): Appends a new predicate to the current expression.add_name_test(): Converts the current element into a node test (e.g., self::element).join(combinator, other_xpath_expr): Joins the current expression with another using a combinator (like / or //).The Selector class handles pseudo-elements via the pseudo_element attribute. The mapping depends on the input syntax:
| CSS3 syntax | Older syntax | Resulting pseudo_element |
|---|---|---|
a::before | a:before | 'before' (string) |
li::marker | (N/A) | 'marker' (string) |
a::foo(2) | (N/A) | FunctionalPseudoElement(...) |
li:marker | (N/A) | None (Invalid pseudo-class) |
The library implements CSS3 selectors with several specific behaviors and limitations regarding pseudo-classes and Level 4 features.
Because there is no interactivity or link history, the following pseudo-classes are accepted but will never match anything:
:hover, :active, :focus, :target, :visited*:first-of-type, *:last-of-type, *:nth-of-type, *:nth-last-of-type, and *:only-of-type work when a specific element type is provided, but do not work with the universal selector *.:is() and :where(): Arguments must be a comma-separated list of compound selectors. Combinators (e.g., a b) are not allowed inside them. :not() and :scope are also rejected inside these.:has(): Takes a single argument consisting of an optional leading combinator (>, +, or ~) followed by one compound selector (type, class, or universal selectors only). It does not support IDs or selector lists.:not(): Takes a single argument (a complex selector). Selector lists are not supported.:scope: Supported, but only at the start of a selector.i (case-insensitive) is supported (e.g., [href^="HTTP" i]). s is accepted but has no effect.#37signals).:contains(text): Supported (an early draft extension).!= operator: Supported (e.g., [foo!=bar]).The parser raises SelectorSyntaxError when it encounters invalid CSS selector syntax. Common triggers include:
[attr='value).next_ident() or next_ident_or_star() and the stream provides a token that does not match the expected type (e.g., expecting an IDENT but receiving a NUMBER).When parsing CSS strings, you may encounter syntax errors. You can catch specific syntax errors or use the common parent exception to handle all selector-related issues.
SelectorSyntaxError: Raised when a selector does not match the grammar.SelectorError: The base class for SelectorSyntaxError and ExpressionError. Catching this will handle both types.from cssselect.parser import parse, SelectorError
try:
selectors = parse("invalid selector")
except SelectorError:
# This catches both SelectorSyntaxError and ExpressionError
print("Failed to parse CSS")When using css_to_xpath or selector_to_xpath, the library may raise an ExpressionError (which inherits from SelectorError and RuntimeError). This occurs when:
:is(), :where(), or :matches()).