mdream Documentation

mdream provides two distinct Docker images depending on whether you need to convert existing HTML or crawl live websites:

1. harlanzw/mdream:core: Use this for high-performance HTML to Markdown conversion. It contains a single native Rust binary (~600 KB). It reads from stdin and writes to stdout. It cannot fetch URLs or render JavaScript.
2. harlanzw/mdream:crawl: Use this for fetching URLs, crawling entire websites, and generating llms.txt files. It includes Node.js and Playwright Chrome (~1.5 GB) to handle JavaScript-heavy sites.

Note: harlanzw/mdream:latest is an alias for :crawl. It is recommended to use the explicit :core or :crawl tags.

| Image | Use it for | Engine | Size |
|-------|-----------|--------|------|
| `harlanzw/mdream:core` | Converting HTML you already have to Markdown | Native Rust binary | ~600 KB |
| `harlanzw/mdream:crawl` | Fetching/crawling URLs, `llms.txt` generation | Node + Playwright Chrome | ~1.5 GB |

The plugin's behavior changes depending on the Vite command being run:

Development (vite dev)

Intercepts incoming requests via middleware. If a request matches the Request Matching criteria, it resolves the HTML path (checking .html, the base path, or /index.html), converts it to Markdown using mdream, and responds with Content-Type: text/markdown; charset=utf-8 and Cache-Control: no-cache.

Build (vite build)

Processes all HTML assets in the output bundle using the generateBundle hook. For files matching include and not matching exclude, it converts the HTML to Markdown and emits a corresponding .md file into the bundle (preserving directory structure). You can use outputDir to place these files in a specific subdirectory.

Preview (vite preview)

Uses middleware to read built HTML files from the output directory (default dist). It resolves paths in the order: <outDir>/<basePath>.html, <outDir>/<basePath>/index.html, or <outDir>/index.html. Responses use Cache-Control: public, max-age=3600.

The @mdream/crawl package provides six hooks that allow you to intercept and transform data at different stages of the crawl pipeline. Hooks receive mutable objects; you must mutate them in-place to apply changes. You can use these via defineConfig or by passing them directly to the crawlAndGenerate function.

Available hooks:

• crawl:url: Called before fetching. Set ctx.skip = true to skip the network request.
• crawl:html: Called after fetching, before HTML-to-Markdown conversion. Mutate ctx.html to transform raw HTML.
• crawl:page: Called after HTML-to-Markdown conversion. Mutate page.title, page.metadata, etc. (replaces the legacy onPage callback).
• crawl:content: Called before markdown is written to disk. Mutate ctx.content or ctx.filePath.
• crawl:done: Called after all pages are crawled, before llms.txt generation. Use ctx.results to filter or reorder results.

import { crawlAndGenerate, defineConfig } from '@mdream/crawl'

// Example using defineConfig
export default defineConfig({
  hooks: {
    'crawl:url': (ctx) => {
      if (ctx.url.includes('/assets/')) ctx.skip = true
    },
    'crawl:page': (page) => {
      page.title = page.title.replace(/ - Docs$/, '')
    },
    'crawl:done': (ctx) => {
      const filtered = ctx.results.filter(r => r.content.length > 100)
      ctx.results.length = 0
      ctx.results.push(...filtered)
    }
  },
})

// Example using programmatic API
await crawlAndGenerate({
  urls: ['https://example.com'],
  outputDir: './output',
  hooks: {
    'crawl:page': (page) => {
      page.title = page.title.replace(/ \| Brand$/, '')
    }
  },
})

The JS engine (@mdream/js) allows you to intercept and modify the conversion pipeline using imperative hook-based plugins. You can create a plugin using createPlugin from @mdream/js/plugins and pass it to the hooks option in htmlToMarkdown.

Available hooks include:

• beforeNodeProcess: Intercept before any node processing. Return { skip: true } to skip the node.
• onNodeEnter: Prepend a string when entering an element node.
• onNodeExit: Append a string when exiting an element node.
• processAttributes: Modify element attributes (e.g., for Tailwind class extraction).
• processTextNode: Transform text nodes. Return { content: string, skip: boolean }.

import { htmlToMarkdown } from '@mdream/js'
import { createPlugin } from '@mdream/js/plugins'

const myPlugin = createPlugin({
  onNodeEnter(node) {
    if (node.name === 'h1')
      return '** '
  },
  processTextNode(textNode) {
    if (textNode.parent?.attributes?.id === 'highlight') {
      return { content: `**${textNode.value}**`, skip: false }
    }
  },
})

const markdown = htmlToMarkdown(html, { hooks: [myPlugin] })