repository·main·Indexed 26 days ago
https://github.com/kepano/defuddleA library and CLI tool for extracting article content and metadata from web pages. It provides a Node.js API via the Defuddle class and parseSource function, a command-line interface for parsing URLs or HTML files into formats like Markdown and JSON, and a Cloudflare Worker implementation (defuddle.md) that acts as an HTTP API wrapper.
defuddle.md is a Cloudflare Worker that provides an HTTP API for cleaning web content. You can retrieve cleaned content from any URL by passing that URL as the path in your request.
Note: When testing locally, use curl rather than a web browser.
curl https://defuddle.md/https://example.com/articleTo run the defuddle.md Cloudflare Worker on your local machine, follow these steps:
npm install -g wranglerwebsite directory and start the development server:cd website
npx wrangler devcurl:curl http://localhost:8787/https://example.com/articleIf changes to the source code are not reflected in your local environment, clear the Wrangler cache by deleting the .wrangler directory:
rm -rf .wranglernpm install -g wrangler
cd website
npx wrangler devWhen calling the Defuddle extraction function, you can pass a DefuddleOptions object to customize the extraction behavior. Key options include:
url: The URL of the page to be parsed.markdown: Convert output to Markdown (defaults to false).separateMarkdown: Include Markdown in the response (defaults to false).contentSelector: A CSS selector to use as the main content element, bypassing auto-detection.language: Preferred language for extraction (BCP 47 tag, e.g., 'en'). Used in Accept-Language headers and for selecting transcript tracks.includeReplies: Controls reply inclusion. Can be 'extractors' (default, uses site-specific extractors), true (all replies/comments), or false (exclude all).debug: Enable debug logging (defaults to false).profile: Enable per-step profiling; timings are returned in result.profile (defaults to false).fetch: Custom fetch function override for all HTTP requests (useful for proxying in restricted environments).Removal & Cleaning Options (Defaults are true unless noted):
removeExactSelectors: Remove elements matching exact selectors (e.g., ads).removePartialSelectors: Remove elements matching partial selectors.removeHiddenElements: Toggle removal of hidden elements.removeLowScoring: Toggle content scoring/removal.removeSmallImages: Toggle small image removal.removeContentPatterns: Toggle content-based pattern removal (read time, boilerplate, etc.).removeImages: Remove images (defaults to false).standardize: Toggle HTML standardization (footnotes, headings, code blocks, etc.).Extraction Logic:
useAsync: Allow async extractors to fetch content from third-party APIs if local HTML extraction fails (defaults to true).The DefuddleResponse object contains the extracted content and associated metadata. It extends DefuddleMetadata and includes:
content: The extracted HTML content.contentMarkdown: (Optional) The extracted Markdown content.extractorType: (Optional) The type of extractor used.metaTags: (Optional) An array of MetaTagItem objects.debug: (Optional) DebugInfo containing the contentSelector used and a list of removals performed.profile: (Optional) A Record<string, number> containing step timings if profile: true was set in options.variables: (Optional) A dictionary of ExtractorVariables (key-value strings).Defuddle class. You can import it as a default export from the defuddle package.If you are integrating defuddle into a Node.js application, you can use the parseSource function to process HTML from various inputs.
parseSource(source?: string, options: ParseOptions, input?: NodeJS.ReadStream): Promise<ParseResult>
source (optional): A string representing a URL, a file path, or - for stdin. If undefined, it defaults to reading from stdin.options: A ParseOptions object (see below).input (optional): A NodeJS.ReadStream. Defaults to process.stdin.| Key | Type | Description |
|---|---|---|
output | string | Path to write the output file |
markdown | boolean | Convert content to markdown |
md | boolean | Alias for markdown |
json | boolean | Output as JSON with metadata |
debug | boolean | Enable debug mode |
property | string | Extract a specific property (e.g., title) |
lang | string | Preferred language (BCP 47) |
userAgent | string | Custom User-Agent header |
frontmatter | boolean | Prepend YAML frontmatter |
Returns a Promise resolving to a ParseResult object:
interface ParseResult {
output: string;
}When debug: true is enabled, the response includes DebugInfo to help troubleshoot content extraction:
export interface DebugRemoval {
step: string;
selector?: string;
reason?: string;
text: string;
}
export interface DebugInfo {
contentSelector: string;
removals: DebugRemoval[];
}The MetaTagItem represents an individual HTML meta tag extracted from the page:
export interface MetaTagItem {
name?: string | null;
property?: string | null;
content: string | null;
}The DefuddleMetadata interface defines the standard metadata fields returned in a successful extraction:
| Field | Type | Description |
|---|---|---|
title | string | The title of the content |
description | string | The description of the content |
domain | string | The domain of the source |
favicon | string | URL to the site favicon |
image | string | URL to the main image |
language | string | The language of the content |
parseTime | number | Time taken to parse |
published | string | Publication date/time |
author | string | Author of the content |
site | string | The site name |
schemaOrgData | any | Schema.org structured data |
wordCount | number | Total word count |
The defuddle CLI allows you to extract article content from a web URL, a local HTML file, or via stdin.
parse [source]Parses HTML content from the provided source.
Arguments:
source: An HTML file path, a URL (starting with http:// or https://), or - to read from stdin.Options:
-o, --output <file>: Specify the output file path. If omitted, results are printed to stdout.-m, --markdown or --md: Convert the extracted content to markdown format.-j, --json: Output the result as a JSON object containing metadata (title, description, domain, etc.) and content.-f, --frontmatter: Prepend YAML frontmatter (containing title, author, source, etc.) to the output.-p, --property <name>: Extract only a specific property from the result (e.g., title, description, domain, wordCount).--debug: Enable debug mode.-l, --lang <code ext>: Set the preferred language using BCP 47 codes (e.g., en, fr, ja).-u, --user-agent <string>: Provide a custom User-Agent header for HTTP requests to avoid 403/FORBIDDEN errors.Parse a URL and output to a file as Markdown:
defuddle parse https://example.com -m -o article.mdPipe HTML from stdin:
cat page.html | defuddle parse -Extract only the title as JSON:
defuddle parse https://example.com --property titleThe library exports several key types for configuring the engine and handling responses. You can import these directly from the package entry point:
DefuddleOptions: Configuration object for initializing Defuddle.DefuddleResponse: The shape of the data returned after processing.DefuddleMetadata: Metadata associated with the defuddling process.DebugInfo: Information used for debugging the engine's state.DebugRemoval: Information regarding what content was removed during processing.import type {
DefuddleOptions,
DefuddleResponse,
DefuddleMetadata,
DebugInfo,
DebugRemoval
} from 'defuddle';