repository·master·Indexed 20 days ago
https://github.com/clj-commons/hickoryA library for parsing HTML into Clojure data structures, supporting both Clojure and ClojureScript. It allows developers to analyze, transform, and regenerate HTML by converting it into either Hiccup vectors or a map-based DOM format. Hickory includes tools for tree navigation and editing via zippers in `hickory.zip`, as well as CSS-style selectors in `hickory.select` for querying Hickory-format data.
Hickory is a library for parsing HTML into Clojure data structures. It allows you to analyze, transform, and output HTML by converting it into one of two formats:
clojure.xml that represents a DOM-like structure.Hickory is compatible with both Clojure and Clojurescript.
HiccupRepresentable protocol is implemented by objects that can be converted into Hiccup-format data structures. Nodes produced by parse or parse-fragment implement this protocol by default.The HickoryRepresentable protocol allows objects to be represented as HTML DOM node maps, similar to clojure.xml.
Each DOM node is represented as a map or a string (for Text or CDATASections). Maps contain the following keys:
:type: The node type, which will be one of [:comment, :document, :document-type, :element].:tag: The node's tag (applicable if :type is :element).:attrs: A map of the node's attributes (applicable if :type is :element).:content: A vector of the node's child nodes (applicable if :type is :element).The Hickory format is a structured data representation of HTML designed for easy searching, modification, and roundtripping. Unlike Hiccup (which is optimized for writing HTML), Hickory is optimized for processing.
A Hickory node is either a string (representing text or CDATA) or a map.
If a node is a map, it contains a subset of the following keys based on its :type:
:type: One of :comment, :document, :document-type, or :element.:tag: The node's tag (e.g., :img). Only present for :element nodes.:attrs: A map of keyword attributes (e.g., {:href "/a"}). Only present for :element nodes.:content: A vector of child nodes. Present for :comment, :document, and :element nodes.The hickory.zip namespace provides zippers for both Hiccup and Hickory formats, allowing you to navigate, edit, and reconstruct trees. This is useful for complex transformations.
hiccup-zip: Creates a zipper for Hiccup vector trees.hickory-zip: Creates a zipper for Hickory DOM maps.Once a zipper is created, you can use standard clojure.zip functions (like zip/next, zip/root, zip/replace) to navigate the tree and perform modifications. After modifications, use zip/root to retrieve the entire updated tree.
(use 'hickory.zip)
(require '[clojure.zip :as zip])
(require '[hickory.render :refer [hickory-to-html]])
;; Example: Navigating and replacing a node in a Hiccup tree
(-> (hiccup-zip (as-hiccup (parse "<a href=foo>bar<br></a>")))
zip/next zip/next
(zip/replace [:head {:id "a"}])
zip/root)
;; Example: Navigating and replacing a node in a Hickory tree, then rendering to HTML
(-> (hickory-zip (as-hickory (parse "<a href=foo>bar<br></a>")))
zip/next zip/next
(zip/replace {:type :element :tag :head :attrs {:id "a"} :content nil})
zip/root
hickory-to-html)
;; => "<html><head id=\"a\"></head>..." When parsing markup in a Node.js environment, Hickory requires a Node DOM implementation. You can install alternatives like jsdom or xmldom via npm. Use the following setup patterns to initialize the global DOM objects required by Hickory.
;; Using jsdom
(set! js/document (.jsdom (cljs.nodejs/require "jsdom")))
;; Using xmldom
(set! js/DOMParser (.-DOMParser (cljs.nodejs/require "xmldom")))To use Hickory in your project, add the following dependency to your project.clj or your Maven-compatible build tool's configuration file.
[org.clj-commons/hickory "0.7.3"]To process HTML, use parse or parse-fragment to create a parsed representation. This representation can then be converted into either a Hiccup vector tree or a Hickory DOM map.
parse: Expects a full HTML document. It uses an HTML5 parser (Jsoup in Clojure, browser DOM in ClojureScript) to ensure a well-formed document.parse-fragment: Expects a smaller HTML fragment. It returns a list of parsed nodes. Because fragments may not have a single common parent, you must process each item in the returned list individually using as-hiccup or as-hickory.Once parsed, convert the objects using:
as-hiccup: Converts to Hiccup vector format.as-hickory: Converts to Hickory DOM map format.(use 'hickory.core)
;; Parsing a full document
(def parsed-doc (parse "<a href=\"foo\">foo</a>"))
(as-hiccup parsed-doc)
;; => [[:html {} [:head {}] [:body {} [:a {:href "foo"} "foo"]]]
(as-hickory parsed-doc)
;; => {:type :document, :content [{:type :element, :attrs nil, :tag :html, :content [...]}]}
;; Parsing a fragment (returns a list)
(def parsed-frag (parse-fragment "<a href=\"foo\">foo</a> <a href=\"bar\">bar</a>"))
;; Note: You must map the conversion functions over the list returned by parse-fragment
(map as-hiccup parsed-frag)
(map as-hickory parsed-frag)The hickory.select namespace provides powerful, CSS-like selectors that operate on Hickory-format data. A selector is a function that takes a zipper loc and returns the loc if it matches, or nil otherwise.
node-type: Selects nodes by their :type (e.g., (node-type :comment)).tag: Selects nodes by their :tag (e.g., (tag :div)).attr: Selects nodes based on an attribute key in :attrs. Can take an optional predicate function: (attr :id #(.startsWith % "foo")).id: Selects nodes with a specific :id attribute (case-insensitive).class: Selects nodes containing a specific class in their class string.any / element: Selects any element node.root: Returns the root node.nth-child / nth-last-child: Selects the $n$-th child/last-child (supports :odd and :even).nth-of-type / nth-last-of-type: Selects the $n$-th child of a specific tag type.first-child / last-child: Shortcuts for the first/last child.n-moves-until: Selects nodes at a specific distance from a boundary.Combinators take selectors and return a new combined selector:
and: All argument selectors must be true.or: At least one argument selector must be true.not: The argument selector must be false.el-not: The argument must be an element and the selector must be false.child: Matches a specific chain of direct child relationships.descendant: Matches a chain of descendant relationships (can skip intermediate levels).select: Returns a sequence of the nodes (zipper locs) that match the selector.select-locs: Returns a sequence of the zipper locs themselves (useful for analyzing surroundings).select-next-loc: Used for manual traversal when you need to modify the tree while walking it. Do not use select-locs if you intend to modify the tree, as the returned locs will become stale after the first modification.(require '[hickory.select :as s])
;; Example: Using descendant and combinators to find a specific value
(-> (s/select (s/descendant (s/class "subModule")
(s/class "standings")
(s/and (s/tag :tr)
s/first-child)
(s/and (s/tag :td)
(s/nth-child 2))
(s/tag :a))
site-htree)
first :content first string/trim)Once you have a node (typically created via parse or parse-fragment), you can convert it into different data representations:
as-hiccup: Converts the node into a Hiccup-format data structure. The node must implement the HiccupRepresentable protocol.as-hickory: Converts the node into a Hickory-format data structure. The node must implement the HickoryRepresentable protocol.(as-hiccup node)
(as-hickory node)Hickory provides several functions to parse HTML strings into a DOM structure. The resulting nodes can then be converted into Hiccup or Hickory formats.
parse: Parses an entire HTML document.parse-fragment: Parses an HTML fragment (a group of tags that would typically reside under a <body> tag) into a list of DOM elements.parse-dom-with-domparser: Parses using a DOMParser.parse-dom-with-write: Parses an HTML document or fragment using document.implementation.createHTMLDocument and document.write.;; Parsing an entire document and converting to Hiccup
(-> (parse "<a style=\"visibility:hidden\">foo</a><div style=\"color:green\\"><p>Hello</p></div>")
as-hiccup)
;; Parsing an entire document and converting to Hickory
(-> (parse "<a style=\"visibility:hidden\">foo</a><div style=\"color:green\\"><p>Hello</p></div>")
as-hickory)The following table defines the keys used in the Hickory map-based node format.
Key | Description
-----------|----------------------------------------------------------------------
:type | One of :comment, :document, :document-type, :element
:tag | A node's tag (e.g., :img). Only for :element nodes.
:attrs | A map of keyword attributes (e.g., {:href "/a"}). Only for :element nodes.
:content | A vector of child nodes. For :comment, :document, and :element nodes.