DOCSEARCHbyAlgolia
Submit Docs

Meeseeks

repository·main·Indexed 43 days ago

https://github.com/mischov/meeseeks

An Elixir library for high-performance parsing and data extraction from HTML and XML using CSS or XPath selectors. It supports HTML5 compliant parsing, custom selectors via the Meeseeks.Selector behaviour, and provides a suite of extractors for retrieving attributes, text, and tags from Meeseeks.Result objects. Version ~> 0.18.0.

Tokens
9K
Snippets
34
Records
39
Agent score
55%

What's inside Meeseeks

  1. XPath Selector Limitations in Meeseeks

    main

    While Meeseeks.XPath supports most of the XPath 1.0 syntax, it has several specific limitations and unsupported features. Understanding these is necessary to avoid runtime errors when constructing queries.

    Unsupported features include:

    • Top-level filter expressions: Expressions that attempt to filter the entire result set at the top level (e.g., xpath("(//ol|//ul)[2]")) are prohibited.
    • Attribute steps outside of predicates: You cannot select attributes directly as nodes (e.g., xpath("//p/@class")). You can only use attributes within predicates to filter elements (e.g., xpath("//p[@class]")).
    • Variable references: Expressions using variables (e.g., xpath("*[position()=$p]")) are not supported.
    • Specific XPath functions: The functions id(), lang(), and translate() are not supported.
    • Namespace resolution: Namespace prefixes are not automatically resolved to URIs. You must search for the prefix itself rather than the expanded URI.
    • HTML5 Processing Instructions: When using the meeseeks_html5ever parser, processing instructions are treated as comments.

  2. Select nodes using CSS or XPath

    main

    Meeseeks provides several functions to search for nodes within a Meeseeks.Document or Meeseeks.Result.

    Selection Functions

    • all/3: Returns a list of all matching nodes.
    • one/3: Returns the first matching node (depth-first) or nil if no match is found.
    • fetch_all/3: Similar to all/3, but returns {:ok, results} or {:error, %Meeseeks.Error{type: :select, reason: :no_match}}.
    • fetch_one/3: Similar to one/3, but returns {:ok, result} or {:error, %Meeseeks.Error{type: :select, reason: :no_match}}.

    Generating Selectors

    Use the css/1 macro from Meeseeks.CSS or the xpath/1 macro from Meeseeks.XPath to create selectors.

    All selection functions accept a queryable (source, document, or Meeseeks.Result), one or more selectors, and an optional initial context.

    import Meeseeks.CSS
result = Meeseeks.one(document, css("#main p"))
#=> #Meeseeks.Result<{ <p>1</p> }

import Meeseeks.XPath
result = Meeseeks.one(document, xpath("//*[@id='main']//p"))
#=> #Meeseeks.Result<{ <p>1</p> }>

  3. Parse HTML and XML sources

    main

    To query a source, you must first parse it into a Meeseeks.Document.

    • Meeseeks.parse/1: Parses a string as HTML (default).
    • Meeseeks.parse/2: Parses a string with a specified format. The second argument can be :html, :xml, or :tuple_tree.

    Note: While selection functions can accept unparsed strings, parsing them into a Meeseeks.Document first is significantly more efficient when performing multiple selections on the same content.

    document = Meeseeks.parse("<div id=main><p>1</p><p>2</p><p>3</p></div>")
#=> #Meeseeks.Document<{...}>

  4. Deploying Meeseeks with Docker

    main

    When containerizing Meeseeks, ensure the Rust compiler is installed before running mix deps.compile.

    Multi-stage Builds

    If you use a multi-stage Dockerfile (separating the build stage from the release stage):

    • Build Stage: Must have Rust installed.
    • Release Stage: Must have libgcc installed. For Alpine-based images, this is not included by default.

    Required Configuration

    You must set the following environment variable during the build stage to prevent mix compile from failing: RUSTFLAGS='--codegen target-feature=-crt-static'

    # In the release stage (e.g., Alpine-based)
RUN apk add --no-cache libgcc

# In the build stage
ENV RUSTFLAGS='--codegen target-feature=-crt-static'

  5. How to use dynamic values in XPath expressions

    main

    Since Meeseeks.XPath does not support variable references (e.g., $p), you must construct your XPath string using Elixir string interpolation or concatenation to inject dynamic values.

    p = 2
xpath("*[position()=" <> Integer.to_string(p) <> "]")

  6. When to use Meeseeks instead of Floki

    main

    Choose Meeseeks over Floki in the following scenarios:

    • Parsing malformed HTML: Meeseeks provides a correct-by-default solution for parsing HTML that may not be well-formed (e.g., scraped content), whereas Floki's default :mochiweb_html-based parser is not HTML5 compliant and can produce unexpected results.
    • Parsing XML: Meeseeks includes an XML parser, which Floki lacks.
    • XPath Selectors: Meeseeks supports XPath selectors, whereas Floki does not.
    • Custom Selectors: Meeseeks allows for custom selectors, which is not supported in Floki.

  7. Install Meeseeks via Hex

    main

    Add meeseeks to your mix.exs dependencies to use the library for HTML/XML parsing and data extraction. Because it uses pre-compiled NIFs via meeseeks_html5ever, you do not need to have Rust installed on your system.

    defp deps do
  [
    {:meeseeks, "~> 0.18.0"}
  ]
end

  8. Create a custom selector by implementing the Meeseeks.Selector behaviour

    main

    To create a custom selector in Meeseeks, define a module that implements the Meeseeks.Selector behaviour. The core logic resides in the match/4 function, which determines if a specific node in a document satisfies the selector's criteria.

    Implementation Requirements

    • Use use Meeseeks.Selector to implement the behaviour.
    • Implement match(selector, node, document, context):
      • selector: The instance of your custom selector struct.
      • node: The current node being inspected (e.g., a %Meeseeks.Document.Comment{} or %Meeseeks.Document.CodeBlock{}).
      • document: The full document context.
      • context: Additional context provided during the selection process.
    • Return true if the node matches the selector's criteria, otherwise return false.
    defmodule CommentContainsSelector do
  use Meeseeks.Selector

  alias Meeseeks.Document

  defstruct value: ""

  # Match logic for a specific node type (e.g., Comment)
  def match(selector, %Document.Comment{} = node, _document, _context) do
    String.contains?(node.content, selector.value)
  end

  # Fallback for all other node types
  def match(_selector, _node, _document, _context) do
    false
  end
end

# Usage
selector = %CommentContainsSelector{value: "TODO"}
Meeseeks.one("<!-- TODO: Close vuln! -->", selector)
#=> #Meeseeks.Result<{ <!-- TODO: Close vuln! --> }>

  9. How to select nodes by namespace

    main

    Namespace prefixes are not resolved to namespace URIs in Meeseeks.XPath. To find a node within a specific namespace, you must search for the namespace prefix of the node rather than the expanded namespace-uri().

    # Instead of xpath("*[namespace-uri()='https://example.com/ns']")
xpath("*[namespace-uri()='example']")

  10. Deploying Meeseeks to Heroku

    main

    Because most Heroku Elixir buildpacks do not include the Rust compiler, you must manually configure your app to support Meeseeks' compilation requirements.

    1. Add a Rust buildpack to your application and ensure it is positioned to run before the Elixir buildpack.
    2. Create a RustConfig file in your project's root directory containing the environment variable RUST_SKIP_BUILD=1 to manage the build process correctly.
    # Add the Rust buildpack (ensure it runs before Elixir)
heroku buildpacks:add -i 1 https://github.com/emk/heroku-buildpack-rust.git

# Create the RustConfig file in the project root
echo "RUST_SKIP_BUILD=1" > RustConfig

  11. How to extract attributes from elements

    main

    In Meeseeks.XPath, you cannot use attribute steps (like /@class) to select attributes directly. Instead, use an XPath predicate to filter for elements that possess the attribute, and then use the Meeseeks.attr/2 extractor to retrieve the value.

    # Correct way to get the 'class' attribute of elements that have one
Meeseeks.all(doc, xpath("//p[@class]"))
|> Enum.map(&Meeseeks.attr(&1, "class"))

  12. When to use Floki instead of Meeseeks

    main

    Choose Floki over Meeseeks in the following scenarios:

    • Avoiding Rust dependencies: Floki's default :mochiweb_html-based parser does not require Rust in your build process. Meeseeks requires Rust to be present during the build.
    • Parsing well-formed HTML: If your HTML is guaranteed to be well-formed, Floki's default parser is sufficient.
    • Modifying HTML documents: Floki provides the ability to make updates to an HTML document, whereas Meeseeks does not support document modification.