DOCSEARCHbyAlgolia
Submit Docs

Floki HTML Parsing Library for Elixir

repository·main·Indexed 24 days ago

https://github.com/philss/floki

An HTML parsing library for Elixir that enables querying and manipulating HTML documents using CSS selectors. It provides tools for parsing documents and fragments via Floki.parse_document/2, extracting text and attributes, and converting nodes back to HTML. Floki supports standard CSS selectors, custom pseudo-selectors like :fl-contains, and allows configuration of alternative parsers such as html5ever or fast_html for improved performance or HTML5 compliance.

Tokens
6.1K
Snippets
9
Records
26
Agent score
79%

What's inside Floki

  1. Install Floki

    main

    To use Floki in an Elixir project, add it to your mix.exs dependencies and run mix deps.get.

    If you are using Livebook or running a standalone script, use Mix.install/2.

    # In mix.exs
defp deps do
  [
    {:floki, "~> 0.38.0"}
  ]
end

# In Livebook or a script
Mix.install([
  {:floki, "~> 0.38.0"}
])

  2. Configure alternative HTML parsers in Floki

    main

    By default, Floki uses a patched version of mochiweb_html. For better performance or HTML5 compliance, you can switch to html5ever or fast_html via your application configuration.

    Using html5ever (Rust-based)

    1. Add {:html5ever, "~> 0.16.0"} to your dependencies.
    2. Configure in config/config.exs:
      config :floki, :html_parser, Floki.HTMLParser.Html5ever

    Using fast_html (C-based)

    1. Add {:fast_html, "~> 2.0"} to your dependencies. (Requires a C compiler, GNU Make, and CMake).
    2. Configure in config/config.exs:
      config :floki, :html_parser, Floki.HTMLParser.FastHtml

    Alternatively, you can pass the parser module directly to Floki.parse_document/2 or Floki.parse_fragment/2 as an option.

    # in config/config.exs
config :floki, :html_parser, Floki.HTMLParser.Html5ever

  3. Implement a custom HTML parser for Floki

    main

    To extend Floki with a custom parser, implement the Floki.HTMLParser behaviour. Your module must implement the following callbacks to handle both documents and fragments, with support for attribute mapping.

    Required Callbacks

    • parse_document(html, parser_args)
    • parse_fragment(html, parser_args)
    • parse_document_with_attributes_as_maps(html, parser_args)
    • parse_fragment_with_attributes_as_maps(html, parser_args)

    Each callback should return {:ok, html_tree} or {:error, reason}.

  4. Iterate over an HTMLTree using Enumerable

    main

    The Floki.HTMLTree implements the Enumerable protocol. This allows you to use standard Enum functions (like Enum.map/2, Enum.reduce/3, Enum.count/1) directly on the tree.

    When iterating, the tree is traversed based on its node_ids.

    • Enum.count(tree) returns the number of nodes in the tree.
    • Enum.member?(tree, node) checks if a specific node exists in the tree by its node_id.

  5. Troubleshoot :leex module availability

    main
    Floki requires the :leex module to compile. This is usually included in complete Erlang installations. If you encounter the error module :leex is not available, you must install the erlang-dev and erlang-parsetools packages for your operating system.

  6. Suppress Floki log messages

    main
    Floki may emit debug messages regarding selector/HTML parsing issues or info messages regarding deprecated APIs. To suppress these logs, configure the :logger module in your application's compile-time configuration by setting the :compile_time_purge_matching option.

  7. Search for nodes using CSS selectors with Floki.find/2

    main
    Once a document is parsed, use Floki.find/2 to locate specific elements using CSS selectors (e.g., .class-name, #id, tag.class). It returns a list of matching node tuples.

  8. Extract attributes from elements with Floki.attribute/3

    main

    Use Floki.attribute/3 to retrieve the value of a specific attribute from elements matching a selector. It returns a list of attribute values.

    # Fetch attribute from the whole document using a selector
Floki.attribute(document, ".example", "class")
# => ["example"]

# Fetch attribute from nodes already found
document
|> Floki.find(".example")
|> Floki.attribute("class")
# => ["example"]

  9. Convert nodes back to HTML with Floki.raw_html/1

    main

    Use Floki.raw_html/1 to convert a node or a list of nodes back into a raw HTML string.

    document
|> Floki.find(".example")
|> Floki.raw_html()
# => <div class="example"></div>

  10. Parse HTML documents with Floki.parse_document/2

    main

    Use Floki.parse_document/2 to convert an HTML string into a searchable document tree. The function returns {:ok, document} on success.

    Each HTML node in the resulting tree is represented as a tuple: {tag_name, attributes, children_nodes}. Note that even if a node only contains text, the text is represented as a string inside the children_nodes list.

    html = """
  <html>
  <body>
    <div class="example"></div>
  </body>
  </html>
"""

{:ok, document} = Floki.parse_document(html)
# => {:ok, ["html", [], ["body", [], ["div", ["class", "example"], []]]]}

  11. Extract text from elements with Floki.text/2

    main

    Use Floki.text/2 to extract the text content from a node or a set of nodes found via a selector.

    document
|> Floki.find(".headline")
|> Floki.text()
# => "Floki"

  12. Configure the HTML parser for Floki

    main

    Floki uses a dynamic dispatch system to select an HTML parser. By default, it uses Floki.HTMLParser.Mochiweb. You can change the parser globally in your application configuration or per-call via function options.

    Global Configuration

    Set the :html_parser option in your config/config.exs to use a different parser like Html5ever or FastHtml across your entire application.

    Per-call Configuration

    Pass the :html_parser option directly to parsing functions to override the global setting for a specific operation.

    Parser Options

    You can pass specific configuration arguments to the underlying parser using the :parser_args option.

    # Global configuration in config/config.exs
import Config
config :floki, :html_parser, Floki.HTMLParser.Mochiweb

# Per-call configuration
Floki.parse_document(document, html_parser: Floki.HTMLParser.FastHtml)