agent-fetch

repository·main·Indexed 43 days ago

A full-content web fetcher for AI agents and content workflows featuring Chrome TLS fingerprinting and multi-strategy extraction. It supports converting web pages and local PDFs into clean Markdown, JSON, or plain text using strategies like Readability, RSC, and WordPress API. Includes a CLI for single-page extraction and recursive site crawling with depth, concurrency, and URL filtering controls.

Tokens: 7.5K
Snippets: 32
Records: 43
Agent score: 54%

What's inside agent-fetch

How agent-fetch extraction strategies work

main

agent-fetch runs multiple extraction strategies in parallel and selects the most complete result based on content length and quality.

Supported Strategies

Strategy	Description
Readability	Uses Mozilla's Reader View algorithm (strict + relaxed passes). Best for semantic HTML.
Text density	Uses statistical text-to-tag ratio analysis (CETD). Best for complex layouts.
JSON-LD	Parses `schema.org` structured data. Best for sites with rich metadata.
Next.js	Extracts from `__NEXT_DATA__` props. Best for Next.js Pages Router.
React Server Components	Parses streaming RSC payloads. Best for Next.js App Router.
WordPress REST API	Fetches via `/wp-json/wp/v2/` endpoints. Best for WordPress sites.
CSS selectors	Probes semantic containers like `<article>` or `.post-content`. Used as a fallback.

Winner Selection Logic

Strategies extracting 500+ characters are considered candidates.
If text-density or RSC finds 2x more content than Readability, they win.
Otherwise, the longest result is chosen.
Metadata (author, date, site name) is composed from the best source for each specific field across all strategies.

Fetch and extract article content
main
The default command fetches a URL using browser impersonation and extracts the complete article content as markdown. To get structured data for programmatic use, use the --json flag.
JSON Output Structure: When using --json, the output includes metadata and the content. You can parse it to present a formatted view like this:
```
---
title: {title}
author: {byline || "Unknown"}
source: {siteName}
url: {url}
date: {publishedTime || "Unknown"}
fetched_in: {latencyMs}ms
---

## {markdown || textContent}

{markdown || textContent}
```
Error Handling: If the fetch fails, check the suggestedAction field in the JSON response:
suggestedAction Meaning Next Action
retry_with_extract Needs full browser Inform user; agent-fetch is HTTP-only
wait_and_retry Rate limited Wait 60s and retry
skip Cannot access site Inform user
```
npx agent-fetch "<url>" --json
```
https://github.com/teng-lin/agent-fetch/blob/main/skills/agent-fetch/SKILL.md

suggestedAction	Meaning	Next Action
`retry_with_extract`	Needs full browser	Inform user; agent-fetch is HTTP-only
`wait_and_retry`	Rate limited	Wait 60s and retry
`skip`	Cannot access site	Inform user

Crawl multiple pages

main

The crawl command follows links to extract content from multiple pages. By default, it crawls up to 3 levels deep with a limit of 100 pages. You can control concurrency, depth, limits, and URL patterns.

# Basic crawl
npx agent-fetch crawl "<url>"

# Advanced crawl with concurrency and depth control
npx agent-fetch crawl "<url>" --depth 5 --limit 50 --concurrency 3

# Include/exclude specific URL patterns
npx agent-fetch crawl "<url>" --include "*/blog/*" --exclude "**/archive/**"

# Crawl with rate limiting and cross-origin support
npx agent-fetch crawl "<url>" --delay 1000 --no-same-origin

# Output as JSONL for processing
npx agent-fetch crawl "<url>" --json

Add agent-fetch to AI Agents (Claude Code, Cursor, etc.)
main
To allow AI agents like Claude Code, Codex, Cursor, or Copilot to automatically use agent-fetch when they need to read URLs, install the Agent Skill via agentskills.io:
```
npx skills add teng-lin/agent-fetch
```
No further configuration is required; the skill teaches the agent when and how to invoke the tool.
```
npx skills add teng-lin/agent-fetch
```
https://github.com/teng-lin/agent-fetch/blob/main/README.md
Install @teng-lin/agent-fetch
main
You can install the package via npm to use it in your projects, or run it immediately using npx.
To install as a dependency:
```
npm install @teng-lin/agent-fetch
```
To run without installing:
```
npx agent-fetch https://example.com/page
```
```
npm install @teng-lin/agent-fetch
```
https://github.com/teng-lin/agent-fetch/blob/main/README.md
Install and run agent-fetch via npx
main
agent-fetch is a local CLI tool that runs via npx, so no permanent installation is required. You can verify it is working by running the help command.
```
npx agent-fetch --help
```
https://github.com/teng-lin/agent-fetch/blob/main/skills/agent-fetch/SKILL.md
Extract content from local PDF files
main
If the provided argument is a local file path ending in .pdf (and is not a URL), agent-fetch will read the file from disk and extract its content into Markdown or text format.
```
# Extract local PDF to Markdown
agent-fetch ./document.pdf

# Extract local PDF to JSON
agent-fetch --json ./document.pdf
```
https://github.com/teng-lin/agent-fetch/blob/main/src/cli.ts
Crawl websites with the agent-fetch crawl command
main
The agent-fetch crawl <url> subcommand allows you to perform a recursive crawl of a website. It follows links up to a specified depth and limit, providing a stream of fetched content.
Key features:
- Control crawl depth and page limits.
- Filter URLs using glob patterns (--include and --exclude).
- Manage concurrency and request delays to be polite to servers.
- Stay within the same origin by default.
```
# Basic crawl
agent-fetch crawl https://example.com

# Advanced crawl: depth 2, limit 50 pages, 10 parallel requests, only include /blog/ paths
agent-fetch crawl https://example.com --depth 2 --limit 50 --concurrency 10 --include "https://example.com/blog/*"
```
https://github.com/teng-lin/agent-fetch/blob/main/src/cli.ts

Configure site-specific extraction settings via sites.json

main

You can customize how agent-fetch extracts content from specific domains by providing a sites.json configuration file. This allows you to set custom headers, enable specific extraction strategies (like Next.js or WordPress), or configure mobile API access for certain sites.

Configuration Loading Order

The environment variable AGENT_FETCH_SITES_JSON (supports ~/ prefix for home directory).
The bundled config/sites.json file.
An empty default if neither is found.

SiteConfig Schema

Each entry in the JSON object is keyed by the domain name (e.g., example.com).

Key	Type	Description
`userAgent`	`string`	Custom User-Agent to use for this site
`referer`	`string` (URL)	Custom Referer header
`preferJsonLd`	`boolean`	Prefer JSON-LD extraction (structured data)
`useNextData`	`boolean`	Use Next.js `__NEXT_DATA__` extraction
`nextDataPath`	`string`	JSON path (dot notation) to extract content from `__NEXT_DATA__`
`useWpRestApi`	`boolean`	Use WordPress REST API for extraction
`wpJsonApiPath`	`string`	Custom WP JSON API path for non-standard endpoints
`mobileApiConfig`	`object`	Configuration for Mobile API extraction (see MobileApiConfig)
`notes`	`string`	Optional notes about the configuration

MobileApiConfig Schema

Used when mobileApiConfig is provided to enable extraction via a mobile backend.

Key	Type	Description
`apiUrl`	`string` (URL)	Base URL where `articleId` will be appended
`appIdentifier`	`string`	App identifier header
`token`	`string`	Authorization token
`tokenType`	`string`	Header name for the token (e.g., `authorization`, `x-access-token`)

{
  "example.com": {
    "userAgent": "MyCustomAgent/1.0",
    "useNextData": true,
    "nextDataPath": "props.pageProps.article",
    "mobileApiConfig": {
      "apiUrl": "https://api.example.com/articles/",
      "appIdentifier": "example-app",
      "token": "secret-token",
      "tokenType": "x-access-token"
    }
  }
}

Use the agent-fetch CLI to fetch and extract content
main
The primary command agent-fetch <url> fetches a web page and extracts its content into Markdown or plain text. By default, it outputs the article's metadata (title, author, etc.) followed by the Markdown content.
Common use cases include:
- Extracting a single article as Markdown.
- Fetching raw HTML using --raw.
- Extracting specific elements using CSS selectors.
- Handling authentication via cookies or cookie files.
```
# Default: Metadata + Markdown
agent-fetch https://example.com

# Markdown content only (no metadata)
agent-fetch -q https://example.com

# Plain text only
agent-fetch --text https://example.com

# Full JSON output
agent-fetch --json https://example.com

# Raw HTML output
agent-fetch --raw https://example.com
```
https://github.com/teng-lin/agent-fetch/blob/main/src/cli.ts

Use httpFetch programmatically

main

Import httpFetch from @teng-lin/agent-fetch to integrate web extraction into your TypeScript or JavaScript applications.

Basic Usage

import { httpFetch } from '@teng-lin/agent-fetch';

const result = await httpFetch('https://example.com/article');

if (result.success) {
  console.log(result.markdown);     // Full article as markdown
  console.log(result.title);         // Page title
  console.log(result.byline);        // Author name
  console.log(result.textContent);   // Plain text
  console.log(result.latencyMs);    // Fetch latency in ms
}

Usage with Options

const result = await httpFetch('https://slow-site.com/article', {
  timeout: 30000,        // 30 second timeout (default: 20000)
  preset: 'chrome-143',  // TLS preset
});

import { httpFetch } from '@teng-lin/agent-fetch';

const result = await httpFetch('https://example.com/article');

if (result.success) {
  console.log(result.markdown);
}

Extract content using custom CSS selectors
main
Use --select to target specific elements for extraction and --remove to strip unwanted elements like navigation, sidebars, or advertisements.
```
# Extract only the article, remove navigation and ads
npx agent-fetch "<url>" --select "article" --remove "nav, .sidebar, [class*='ad']"

# Extract all divs with class "post-content"
npx agent-fetch "<url>" --select ".post-content"
```
https://github.com/teng-lin/agent-fetch/blob/main/skills/agent-fetch/SKILL.md