dom-to-semantic-markdown

The library provides several features to improve the quality of Markdown generated from HTML for LLM consumption:

• Semantic Structure Preservation: Maintains the meaning of structural elements like <header>, <footer>, <nav>, etc.
• Metadata Extraction: Automatically captures title, description, keywords, Open Graph tags, Twitter Card tags, and JSON-LD data.
• Token Efficiency: Reduces token usage via URL refication and concise content representation.
• Main Content Detection: Automatically identifies and extracts the primary content area of a webpage.
• Table Column Tracking: Injects unique identifiers into table columns to help LLMs correctly correlate data across rows.

You can install the library via npm to use it in your projects.

npm install dom-to-semantic-markdown

The output is optimized for Large Language Models (LLMs). For best results:

1. Extract the Markdown content using the library.
2. Provide a brief instruction or context.
3. Wrap the extracted Markdown in triple backticks (```).
4. Follow the Markdown with your question or prompt.

Example Prompt Structure:

The following is a semantic Markdown representation of a webpage. Please analyze its content:

```markdown
{paste your extracted markdown here}
```

{your question, e.g., "What are the main points discussed in this article?"}

You can use the d2m CLI tool either without a permanent installation using npx, or by installing it globally on your system.

Using npx (No installation required)

Run the tool directly using the latest version:

npx d2m@latest -i <input_file> -o <output_file>

Global Installation

To install the tool globally so you can use the d2m command anywhere, follow these steps:

1. Clone the repository and navigate to the CLI example directory:

   git clone https://github.com/romansky/dom-to-semantic-markdown.git
   cd examples/cli

2. Install dependencies: npm install
3. Build the project: npm run build
4. Link the package globally: npm link

npx d2m@latest -i tryme.html -o output.md

The output of the conversion is a SemanticMarkdownAST, which is a union type of various node objects. Each node represents a semantic part of the document. Common nodes include:

• TextNode: Plain text (type: 'text').
• HeadingNode: Headings with levels 1-6 (type: 'heading').
• LinkNode: Hyperlinks (type: 'link') containing href and content.
• ImageNode: Images (type: 'image') with src and optional alt text.
• ListNode & ListItemNode: Ordered or unordered lists.
• TableNode, TableRowNode, & TableCellNode: Structured table data.
• CodeNode: Code blocks or inline code (type: 'code') with an optional language.
• SemanticHtmlNode: Semantic HTML wrappers like article, section, nav, header, footer, etc.
• MetaDataNode: Extracted metadata (type: 'meta') containing standard, openGraph, twitter, or jsonLd data.

You can intercept the conversion of specific elements by providing an overrideElementProcessing function in the ConversionOptions. This is useful for handling custom web components or specific HTML tags that require unique Markdown representations.

If the callback returns an array of SemanticMarkdownAST nodes, those nodes are inserted into the result instead of the default conversion.

const options = {
  overrideElementProcessing: (element, options, indentLevel) => {
    if (element.tagName.toLowerCase() === 'custom-widget') {
      return [{ type: 'text', content: 'Custom Widget Content' }];
    }
    return undefined; // Fallback to default processing
  }
};

const ast = htmlToMarkdownAST(element, options);

When converting a webpage that contains tables, you can use the -t and -e flags to add unique identifiers to table columns (e.g., <!-- col-0 -->). This helps Large Language Models (LLMs) better understand and parse the table structure.

npx d2m@latest -u https://softwareyoga.com/latency-numbers-everyone-should-know/ -t -e

In a browser environment, you can pass a DOM node (like document.body) directly to convertHtmlToMarkdown.

import {convertHtmlToMarkdown} from 'dom-to-semantic-markdown';

const markdown = convertHtmlToMarkdown(document.body);
console.log(markdown);

Use the -m basic flag to extract basic metadata such as title, description, and keywords from the source URL and include them in the Markdown output's YAML frontmatter.

npx d2m@latest -u https://xkcd.com -m basic

Common CLI usage patterns:

# Convert input.html to output.md
d2m -i input.html -o output.md

# Fetch and convert a webpage to Markdown
d2m -u https://example.com -o output.md

# Extract main content from input.html
d2m -i input.html -e

# Enable table column tracking
d2m -i input.html -t

# Include basic metadata
d2m -i input.html -m basic

# Include extended metadata
d2m -i input.html -m extended