DOCSEARCHbyAlgolia
Submit Docs

html-to-markdown

repository·main·Indexed 21 days ago

https://github.com/xberg-io/html-to-markdown

A high-performance, multi-language library with a Rust core designed to convert complex or malformed HTML into clean CommonMark or Djot. It provides bindings for 16 languages, including Node.js (via NAPI-RS and WASM), PHP, and C/C++ (FFI), as well as a CLI. The engine utilizes a three-tier architecture (byte scanner, DOM walker, and standards-conformant parser) to ensure speed and accuracy. Key features include metadata extraction for SEO and TOC generation, and customizable ConversionOptions for heading styles, wrapping, and output formats.

Tokens
269K
Snippets
721
Records
1.1K
Agent score
73%

What's inside html-to-markdown

  1. Overview of html-to-markdown

    main

    html-to-markdown

    html-to-markdown is a high-performance tool designed to convert messy, real-world HTML into clean Markdown (CommonMark) or Djot. It is built with a Rust core and provides bindings for 16 different languages, ensuring consistent results across different environments.

    Key Capabilities

    • Robust Parsing: Automatically handles unclosed tags, CDATA, custom elements, broken entities, nested tables, and mixed encodings without requiring manual parsing strategy tuning.
    • Metadata Extraction: Extracts structured metadata from the <head> section, including Open Graph, Twitter, JSON-LD, microdata, RDFa, and header hierarchy.
    • Structured Data: Supports GFM (GitHub Flavored Markdown) tables with padded cells, alignment, and pipe escaping.
    • Flexible Output: Supports both CommonMark and Djot (via output_format = "djot").
    • High Performance: Capable of processing 19–116 MB/s on standard corpora.
    • Extensibility: Includes a Visitor API for transforming the converted Markdown AST and configurable preprocessing presets (Standard, Strict, and Lenient).

  2. What is html-to-markdown?

    main

    The html-to-markdown package provides a high-performance, consistent way to convert HTML into Markdown across multiple programming languages. It uses a Rust-based core to ensure that the output is identical regardless of the language binding used (e.g., Python, Node.js, Go, Java, .NET, etc.).

    Key features include:

    • Consistent Rendering: Output matches the Rust core exactly, preventing 'Markdown drift' between different language bindings.
    • Structured Results: Provides not just the Markdown string, but also structured metadata such as links, headings, images, tables, and warnings.
    • Production-Ready: HTML is parsed via the Rust core and is sanitized by default.
    • Language-Specific Optimizations: Depending on the binding, it offers native integrations like NAPI for Node.js, Magnus for Ruby, Panama FFM for Java, or Rustler NIFs for Elixir.

  3. Language bindings for other environments

    main

    While html-to-markdown-rs is the core Rust engine, it is available in several other languages via bindings:

    • JavaScript/TypeScript: html-to-markdown-node (NAPI-RS) or html-to-markdown-wasm (WebAssembly)
    • Python: html-to-markdown (PyO3)
    • Dart/Flutter: h2m (official) or html_to_markdown_rust (unofficial FFI)
    • PHP: html-to-markdown (PIE + Composer)
    • Ruby: html-to-markdown (Magnus + rb-sys)
    • CLI: html-to-markdown-cli

  4. Core capabilities of html-to-markdown

    main

    The html-to-markdown project provides a high-performance conversion engine with the following features:

    • Multi-language support: A single Rust core accessible via 16 languages including Rust, Python, Node.js, WASM, Java, Go, C#, PHP, Ruby, Elixir, R, Dart, Kotlin (Android), Swift, Zig, and a C ABI.
    • Markdown Formats: Produces CommonMark-compatible Markdown with GFM-style tables.
    • Djot Support: Can output in Djot format by setting output_format = "djot".
    • Robustness: Handles 'Real-HTML' edge cases such as unclosed tags, CDATA, custom elements, malformed entities, nested tables, and mixed encodings without content loss.
    • Advanced Features: Supports metadata extraction, a visitor API, inline images, and configurable preprocessing presets.

  5. Key features of html-to-markdown

    main

    Conversion Capabilities

    • Real-HTML Robustness: Automatically handles unclosed tags, CDATA, custom elements, malformed entities, nested tables, and mixed encodings.
    • Output Formats: Supports CommonMark (default) and Djot (by setting output_format = "djot").
    • GFM Tables: Supports GitHub Flavored Markdown tables including padded cells, alignment, and pipe escaping.
    • Metadata Extraction: Automatically parses <head> for Open Graph, Twitter, JSON-LD, and more.
    • Inline Images: Supports opt-in mirroring of data URIs and remote image references.

    Architecture & Performance

    • Tiered Dispatch: Uses a multi-stage process (Byte scanner → DOM walker → html5ever repair) to ensure high-fidelity output.
    • High Speed: Capable of processing 19–116 MB/s on standard corpora.
    • Visitor API: Provides a feature-gated way to traverse and transform the converted Markdown AST.

  6. Understand the End-to-End Testing Infrastructure

    main
    The test_apps/ directory contains end-to-end (e2e) tests designed to validate that published packages from public registries (like PyPI, npm, RubyGems, etc.) are installable and functional in real-world scenarios. This infrastructure ensures that breaking changes are caught and that version synchronization works across all language-specific packages.

  7. Identify the @xberg-io/html-to-markdown-node-linux-x64-musl package

    main
    The @xberg-io/html-to-markdown-node-linux-x64-musl package provides the prebuilt x86_64-unknown-linux-musl binary specifically for use with @xberg-io/html-to-markdown-node. This package is intended for Linux environments using the musl C library (common in Alpine Linux and other lightweight distributions) on x86_64 architecture.

  8. How to use HtmlVisitor to customize HTML→Markdown conversion

    main

    The HtmlVisitor allows you to intercept the conversion process and customize how specific HTML elements are rendered into Markdown. You can override specific methods to change behavior.

    Return Values

    Each callback must return a VisitResult:

    • Continue: Use the standard rendering.
    • Skip: Drop the element entirely.
    • PreserveHtml: Pass the original HTML through verbatim.
    • Custom(text): Replace the rendering with the provided text.
    • Error(message): Abort conversion with the provided message.

    Execution Order

    For an element like <div><p>text</p></div>:

    1. visit_element_start for <div>
    2. visit_element_start for <p>
    3. visit_text for "text"
    4. visit_element_end for <p>
    5. visit_element_end for </div>
    // In Python, Ruby, JS/TS, define a plain class and return:
// "continue", "skip", "preserve_html", {"custom": "..."}, or {"error": "..."}

  9. Implement custom conversion logic with the Visitor Pattern

    main

    The Visitor pattern allows you to intercept the HTML traversal process and apply custom logic to specific elements. You can implement a visitor module by using HtmlToMarkdown.Visitor and providing callbacks for the elements you want to modify or filter.

    Common Use Cases:

    • Custom Markdown dialects: Target specific elements to output formats like Obsidian or Notion.
    • Content filtering: Use :skip to remove unwanted elements like ads or tracking pixels.
    • URL rewriting: Modify href or src attributes (e.g., updating CDN domains).
    • Validation: Check for accessibility requirements like alt text or heading hierarchy.

    Return Values for Callbacks:

    • {:custom, "markdown_string"}: Provides a manually constructed markdown string for that element.
    • :skip: Tells the engine to ignore this element entirely.
    • :continue: Tells the engine to proceed with its default conversion logic.
    defmodule MyVisitor do
  use HtmlToMarkdown.Visitor

  @impl true
  def handle_link(_ctx, href, text, _title) do
    # Rewrite CDN URLs
    href = if String.starts_with?(href, "https://old-cdn.com") do
      String.replace(href, "https://old-cdn.com", "https://new-cdn.com")
    else
      href
    end
    {:custom, "[#{text}](#{href})"}
  end

  @impl true
  def handle_image(_ctx, src, _alt, _title) do
    # Skip tracking pixels
    if String.contains?(src, "tracking"), do: :skip, else: :continue
  end
end

html = "<a href=\"https://old-cdn.com/file.pdf\">Download</a>"
opts = %HtmlToMarkdown.Options{visitor: MyVisitor}
{:ok, result} = HtmlToMarkdown.convert(html, opts)
result.content

  10. How to implement a custom HtmlVisitor

    main

    An HtmlVisitor allows you to customize how specific HTML elements are traversed and converted to Markdown. You can override methods to change the rendering behavior of elements.

    Implementation Details

    Method Naming Convention:

    • visit_*_start: Called before entering an element (pre-order traversal).
    • visit_*_end: Called after exiting an element (post-order traversal).
    • visit_*: Called for specific element types (e.g., visit_link, visit_image).

    Return Values (VisitResult): Each callback must return one of the following:

    • Continue (default): Keep standard rendering.
    • Skip: Drop the element from the output entirely.
    • PreserveHtml: Pass the original HTML through verbatim.
    • Custom(text): Replace the rendering with the provided text.
    • Error(message): Abort conversion with the provided message.

    Language-Specific Idioms:

    • Rust: Return one of the VisitResult variants directly.
    • Python, Ruby, JavaScript/TypeScript: Define a plain class. Return a string ("continue", "skip", "preserve_html") or a tagged map ({"custom": "..."}, {"error": "..."}).

    Performance Tips

    • visit_text is called very frequently (~100+ times per document); keep it fast.
    • Return Continue quickly for elements you do not intend to customize.
    • Avoid heavy computation inside visitor methods; use caching if necessary.

    To use your visitor, wrap it in a VisitorHandle and pass it to ConversionOptions. The handle is thread-safe and can be cloned.

    // Example logic for a visitor (JavaScript/TypeScript idiom)
class MyVisitor {
  visit_link(node) {
    return { custom: "[Custom Link Text]" };
  }
  visit_image(node) {
    return "skip";
  }
}

  11. How VisitResult controls conversion flow

    main

    A VisitResult is returned by a visitor callback to control how the conversion proceeds for a specific node. You can use it to either follow the default conversion logic, provide your own markdown, skip elements, or preserve raw HTML.

    • Continue: Proceed with default conversion behavior.
    • Custom: Replace default output with custom markdown. The visitor is responsible for the markdown of this node and its children. (Field 0 is a string).
    • Skip: Ignore the element and all its children entirely.
    • PreserveHtml: Include the element's raw HTML verbatim in the output without converting it to markdown.
    • Error: Halt the conversion process and return a specific error message. (Field 0 is a string).
    | Value | Description |
|-------|-------------|
| `Continue` | Continue with default conversion behavior |
| `Custom` | Replace default output with custom markdown The visitor takes full responsibility for the markdown output of this node and its children. — Fields: `0`: `string` |
| `Skip` | Skip this element entirely (don't output anything) The element and all its children are ignored in the output. |
| `PreserveHtml` | Preserve original HTML (don't convert to markdown) The element's raw HTML is included verbatim in the output. |
| `Error` | Stop conversion with an error The conversion process halts and returns this error message. — Fields: `0`: `string` |

  12. Compare CLI vs SDK vs MCP

    main

    Choosing the right interface depends on your use case:

    • CLI: Best for one-shot conversions, shell pipelines, fetching single URLs, or ad-hoc extraction using --json | jq. It is a standalone tool for terminal usage.
    • SDK: Best for embedding conversion logic directly into application code, batch processing, or high-performance loops where the overhead of spawning a process is undesirable.
    • MCP Server: Use html-to-markdown mcp to expose convert_html and extract_metadata as tools for AI agents. This allows an MCP client to convert HTML strings directly without shell-outs.

    All interfaces share the same ConversionResult data shape.