DOCSEARCHbyAlgolia
Submit Docs

html-to-markdown

repository·main·Indexed 26 days ago

https://github.com/johanneskaufmann/html-to-markdown

A Go-based library and CLI tool (html2markdown) for transforming complex HTML into clean Markdown. It features custom tag rendering, plugin extensibility, and support for CommonMark and GitHub Flavored Markdown tables. The tool allows for CSS selector-based filtering to include or exclude specific HTML nodes and can convert relative links to absolute links using a specified domain.

Tokens
3.9K
Snippets
8
Records
32
Agent score
86%

What's inside html-to-markdown

  1. Test changes using Golden File tests

    main

    To ensure your changes do not break existing conversion logic, use the built-in 'Golden File' testing pattern:

    1. Add your problematic HTML snippet to one of the .in.html files located in the testdata folders.
    2. Run go test -update to generate/update the expected output.
    3. Inspect the resulting .out.md files in Git to verify the changes.
    4. After modifying internal logic, run go test -update again to check the impact.
    go test -update

  2. Extend functionality with plugins and custom logic

    main

    You can extend the converter by writing custom logic and registering it.

    • Custom Logic: Write your own code and register it with the converter.
    • Execution Order: If you want your logic to run before the library's default rules, use PriorityEarly.
    • Plugins: You can package your custom logic into a reusable plugin.

    For detailed implementation details, refer to the WRITING_PLUGINS.md guide in the repository.

  3. Install the CLI

    main

    You can install the html2markdown CLI tool using several methods:

    Homebrew (macOS):

    brew install JohannesKaufmann/tap/html2markdown

    Go Install:

    go install github.com/JohannesKaufmann/html-to-markdown/v2/cli/html2markdown@latest

    Build from Source:

    go build ./cli/html2markdown

  5. Handle character encoding for HTML input

    main

    The library requires all input to be UTF-8 encoded. It does not perform charset detection or conversion. If you provide HTML in other encodings (e.g., ISO-8859-1, Windows-1252), you may see replacement characters (�) in your markdown output.

    To ensure correct conversion:

    1. Decode your HTML to UTF-8 before passing it to ConvertString(), ConvertNode(), or other conversion methods.
    2. For HTTP content, use the Content-Type header to identify the charset.
    3. For files, detect the charset via <meta> tags or byte inspection. The golang.org/x/net/html/charset package is recommended for this process.

  10. Convert HTML strings with `ConvertString()`

    main

    The simplest way to convert HTML to Markdown is using the htmltomarkdown.ConvertString() function. This is a wrapper around converter.NewConverter() that includes the base and commonmark plugins by default.

    To convert relative links to absolute links, use the converter.WithDomain(domain) option.

    package main

import (
	"fmt"
	"log"

	htmltomarkdown "github.com/JohannesKaufmann/html-to-markdown/v2"
	"github.com/JohannesKaufmann/html-to-markdown/v2/converter"
)

func main() {
	input := `<img src="/assets/image.png" />`

	markdown, err := htmltomarkdown.ConvertString(
		input,
		converter.WithDomain("https://example.com"),
	)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(markdown)
	// Output: ![](https://example.com/assets/image.png)
}

  11. Advanced conversion with `NewConverter()`

    main

    For full control over the conversion process, instantiate a converter directly using converter.NewConverter().

    Important: When using NewConverter directly, you must manually register the base and commonmark plugins to get standard functionality.

    package main

import (
	"fmt"
	"log"

	"github.com/JohannesKaufmann/html-to-markdown/v2/converter"
	"github.com/JohannesKaufmann/html-to-markdown/v2/plugin/base"
	"github.com/JohannesKaufmann/html-to-markdown/v2/plugin/commonmark"
)

func main() {
	input := `<strong>Bold Text</strong>`

	conv := converter.NewConverter(
		converter.WithPlugins(
			base.NewBasePlugin(),
			commonmark.NewCommonmarkPlugin(
				commonmark.WithStrongDelimiter("__"),
			),
		),
	)

	markdown, err := conv.ConvertString(input)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(markdown)
	// Output: __Bold Text__
}

  12. Configure tag types and renderers

    main

    You can customize how specific HTML tags are handled using the Register interface on the converter. This allows you to define if a tag is block or inline, whether it should be removed, or if it should use a custom renderer.

    Tag Types:

    • converter.TagTypeRemove: Removes the tag from the output.
    • converter.TagTypeInline: Treats the node as an inline element.
    • converter.TagTypeBlock: Treats the node as a block element.

    Pre-built Renderers:

    • base.RenderAsHTML: Renders the node (including children) as raw HTML.
    • base.RenderAsHTMLWrapper: Renders the node as HTML and its children as Markdown.

    Priorities:

    • converter.PriorityEarly: Used to override default behaviors (e.g., keeping <style> tags).
    • converter.PriorityStandard: The default priority.
    conv.Register.TagType("nav", converter.TagTypeRemove, converter.PriorityStandard)

conv.Register.RendererFor("b", converter.TagTypeInline, base.RenderAsHTML, converter.PriorityEarly)

conv.Register.RendererFor("article", converter.TagTypeBlock, base.RenderAsHTMLWrapper, converter.PriorityStandard)