repository·master·Indexed 27 days ago
https://github.com/mgdm/htmlqA command-line tool for querying and extracting data from HTML using CSS selectors, similar to jq for JSON. Version 0.5.0 supports extracting raw HTML, plain text, and attribute values, with features for rewriting relative URLs, detecting base URLs from documents, and pretty-printing HTML nodes.
Use rewrite_relative_url to transform relative href attributes into absolute URLs within specific HTML elements. This function targets <a>, <link>, and <area> elements. It uses the provided base URL to resolve relative paths.
Special handling: If an href starts with ////, the leading slashes are trimmed, and the remaining path is used.
pretty_print function to convert a kuchiki::NodeRef into a formatted, human-readable HTML string. This function handles indentation and line breaks, distinguishing between block-level and inline elements (such as a, span, strong, etc.) to ensure the output is properly structured.Use detect_base to find the <base> element within an HTML document and extract its href attribute as a Url. This is useful for determining the starting point for resolving relative links in a document. Returns None if no <base> element is found or if the href attribute is missing or invalid.
use kuchikiki::NodeRef;
use url::Url;
// document: The root node of the HTML document
pub fn detect_base(document: &NodeRef) -> Option<Url> {
// ... implementation
}The following flags and options are available for the htmlq CLI tool:
| Flag | Long Name | Description |
|---|---|---|
| (positional) | selector: The CSS selector used to filter nodes (defaults to html) | |
-f, --filename | --filename | Path to the input HTML file (defaults to - for stdin) |
-o, --output | --output | Path to the output file (defaults to - for stdout) |
-b, --base | --base | URL to prepend to relative links |
-B, --detect-base | --detect-base | Look for the <base> tag in the input HTML to determine the base URL |
-t, --text | --text | Output only the contained text of the filtered nodes |
-i, --ignore-whitespace | --ignore-whitespace | Skip text nodes that consist solely of whitespace |
-p, --pretty | --pretty | Reformat the HTML to be more human-readable |
-r, --remove-nodes | --remove-nodes | A list of selectors for nodes that should be excluded from the output |
-a, --attributes | --attributes | A list of attribute names to extract from the matched nodes |
htmlq command-line tool allows you to parse HTML and extract specific elements using CSS selectors. It supports reading from files or stdin and writing to files or stdout. You can extract raw HTML, plain text, specific attribute values, or pretty-printed HTML.