xmlquery

repository·master·Indexed 19 days ago

https://github.com/antchfx/xmlquery

A Go package for querying XML documents using XPath expressions. It supports parsing XML from URLs, strings, and io.Readers, and provides a StreamParser for memory-efficient processing of large files. The library allows for node manipulation, attribute management, and serialization of XML trees back to strings or writers.

Tokens
5.3K
Snippets
21
Records
26
Agent score
66%

What's inside xmlquery

  1. Understand Node types in xmlquery

    master

    The xmlquery library represents XML documents as a tree of Node objects. Each node has a NodeType that defines its role in the document structure. Common node types include:

    • DocumentNode: The root of the document tree.
    • ElementNode: A standard XML element (e.g., <item>).
    • TextNode: The text content within an element.
    • CharDataNode: Content wrapped in <![CDATA[...]]>.
    • CommentNode: XML comments (e.g., <!-- comment -->).
    • AttributeNode: An attribute of an element.
    • DeclarationNode: Document type declarations (e.g., <!DOCTYPE...>).
    • ProcessingInstruction: XML processing instructions (e.g., <?target instruction?>).
    • NotationNode: A directive in the document (e.g., <!text...>).
  2. How StreamParser works for large XML files

    master

    The StreamParser allows you to process large XML documents in a streaming fashion, which prevents high memory consumption by only keeping the target nodes in memory.

    To use it, you must provide a streamElementXPath that points to the elements you want to extract. You can optionally provide a streamElementFilter for more complex logic (e.g., filtering elements based on their content or attributes).

    Key behaviors:

    • Read() returns the next *Node that matches your criteria.
    • When Read() is called, the previous target node is automatically removed from the document tree to free up memory.
    • The process continues until io.EOF is returned, indicating no more matching nodes were found.
    // Scenario 1: Simple streaming
    // xml := `<AAA><BBB>b1</BBB><BBB>b2</BBB></AAA>`
    sp, err := xmlquery.CreateStreamParser(strings.NewReader(xml), "/AAA/BBB")
    for {
        n, err := sp.Read()
        if err == io.EOF {
            break
        }
        if err != nil {
            panic(err)
        }
        fmt.Println(n.OutputXML(true))
    }
    
    // Scenario 2: Advanced filtering
    // xml := `<AAA><BBB>b1</BBB><BBB>b2</BBB></AAA>`
    // We only want BBB elements where the text is NOT 'b1'
    sp, err := xmlquery.CreateStreamParser(strings.NewReader(xml), "/AAA/BBB", "/AAA/BBB[. != 'b1']")
    for {
        n, err := sp.Read()
        if err == io.EOF {
            break
        }
        // ... handle n
    }
  3. Extract data from XML nodes

    master

    Once you have a node, you can extract its content or attributes:

    • n.InnerText(): Returns the text content of the node.
    • n.SelectElement(name): Finds a child element by name.
    • Accessing attributes: Use XPath to select the attribute directly (e.g., //book/@id), then access the InnerText property of the resulting node.
    // Get text from a child element
    if n := channel.SelectElement("title"); n != nil {
    	fmt.Printf("title: %s\n", n.InnerText())
    }
    
    // Get attribute value via XPath
    list := xmlquery.Find(doc, "//book/@id")
    fmt.Println(list[0].InnerText) // outputs the @id value
  4. Parse UTF-16 XML files

    master

    To parse UTF-16 XML files, use xmlquery.ParseWithOptions() and provide a ParserOptions object containing a CharsetReader in the DecoderOptions.

    // Example: Converting UTF-16 to UTF-8 for parsing
    // (Requires unicode and transform packages)
    options := xmlquery.ParserOptions{
    	Decoder: &xmlquery.DecoderOptions{
    		CharsetReader: func(charset string, input io.Reader) (io.Reader, error) {
    			return input, nil
    		},
    	},
    }
    doc, err := xmlquery.ParseWithOptions(utf8Reader, options)
  5. Find nodes using XPath queries

    master

    Use the following functions to locate nodes in an XML document:

    • xmlquery.Find(doc, xpath): Returns a slice of matching nodes. Note: This function panics if the XPath query is invalid.
    • xmlquery.FindOne(doc, xpath): Returns the first matching node.
    • xmlquery.QueryAll(doc, xpath): Returns a slice of matching nodes. Note: This returns an error instead of panicking if the XPath query is invalid.
    • xmlquery.QuerySelector(doc, expr): Uses a pre-compiled xpath.Expr to find a node (useful for performance).
    // Find all matching nodes
    list, err := xmlquery.QueryAll(doc, "a")
    
    // Find the first matching node
    book := xmlquery.FindOne(doc, "//book[2]")
    
    // Find all matching nodes (panics on invalid XPath)
    list := xmlquery.Find(doc, "//author")
  6. Parse XML documents from various sources

    master

    Use the following methods to parse XML depending on your source:

    • From a URL: Use xmlquery.LoadURL(url).
    • From a string: Use xmlquery.Parse(strings.NewReader(s)).
    • From an io.Reader: Use xmlquery.Parse(reader).
    • From a file: Open the file with os.Open and pass it to xmlquery.Parse(f).
    // Parse from URL
    doc, err := xmlquery.LoadURL("http://www.example.com/sitemap.xml")
    
    // Parse from string
    s := `<?xml version="1.0" encoding="utf-8"?><rss version="2.0"></rss>`
    doc, err := xmlquery.Parse(strings.NewReader(s))
    
    // Parse from io.Reader (e.g., a file)
    f, err := os.Open("../books.xml")
    doc, err := xmlquery.Parse(f)
  7. Evaluate XPath expressions (sum, count, etc.)

    master

    To evaluate XPath expressions that return values (like sum() or count()) rather than nodes, you must use the xpath package to compile the expression and then evaluate it using an XPathNavigator created from your xmlquery document.

    import (
    	"github.com/antchfx/xmlquery"
    	"github.com/antchfx/xpath"
    )
    
    // Evaluate total price of all books
    expr, err := xpath.Compile("sum(//book/price)")
    price := expr.Evaluate(xmlquery.CreateXPathNavigator(doc)).(float64)
    fmt.Printf("total price: %f\n", price)
    
    // Count the number of books
    expr, err := xpath.Compile("count(//book)")
    count := expr.Evaluate(xmlquery.CreateXPathNavigator(doc)).(float64)
  8. Parse large XML files using a stream parser

    master

    To save memory when processing large XML files, use xmlquery.CreateStreamParser. This allows you to iterate through specific elements one by one rather than loading the entire document into memory.

    CreateStreamParser accepts the source reader and one or more XPath expressions to filter the elements you want to stream.

    // Simple stream parsing
    f, _ := os.Open("../books.xml")
    p, err := xmlquery.CreateStreamParser(f, "/bookstore/book")
    for {
    	n, err := p.Read()
    	if err == io.EOF {
    		break
    	}
    	if err != nil {
    		panic(err)
    	}
    	fmt.Println(n)
    }
    
    // Advanced stream parsing with element filtering
    p, err := xmlquery.CreateStreamParser(f, "/bookstore/book", "/bookstore/book[price>=10]")
  9. Query with custom namespace prefixes

    master

    When working with XML that uses namespaces, you can compile XPath expressions with a namespace map to allow querying with custom prefixes.

    nsMap := map[string]string{
    	"q": "http://xmlns.xyz.com/process/2003",
    	"r": "http://www.w3.org/1999/XSL/Transform",
    	"s": "http://www.w3.org/2001/XMLSchema",
    }
    expr, _ := xpath.CompileWithNS("//q:activity", nsMap)
    node := xmlquery.QuerySelector(doc, expr)
  10. Configure XML output options

    master

    When using OutputXMLWithOptions or WriteWithOptions, you can pass OutputOption functions to customize the serialization:

    • WithOutputSelf(): Configures the output to include the root node itself.
    • WithEmptyTagSupport(): Writes empty elements as <empty/> instead of <empty></empty>.
    • WithoutComments(): Skips CommentNode elements in the output.
    • WithPreserveSpace(): Preserves whitespace in the output.
    • WithoutPreserveSpace(): Trims whitespace in the output.
    • WithIndentation(indentation string): Sets the string used for formatting the output (e.g., " " or "\t").
    // Example: Write indented XML with empty tag support and no comments
    err := node.WriteWithOptions(os.Stdout, 
        WithIndentation("  "),
        WithEmptyTagSupport(),
        WithoutComments(),
    )
  11. Parse XML with custom options

    master

    Use ParseWithOptions(r io.Reader, options ParserOptions) to customize the parsing behavior. One common use case is enabling line number tracking by setting WithLineNumbers: true in the ParserOptions struct. This is useful for error reporting or debugging specific locations in the source XML.

    // Example of parsing with line numbers enabled
    n, err := xmlquery.ParseWithOptions(r, xmlquery.ParserOptions{
        WithLineNumbers: true,
    })