DOCSEARCHbyAlgolia
Submit Docs

feedgrab Documentation

repository·main·Indexed 20 days ago

https://github.com/ibigqiang/feedgrab

A universal content reader and multi-platform scraping tool (v0.26.0) that fetches and normalizes content from 18+ platforms, including X (Twitter), Xiaohongshu, YouTube, WeChat, and GitHub. It converts data into structured, Obsidian-compatible Markdown files. The tool is available as a Python CLI/library, Claude Code Skills, and an MCP Server for AI agents, featuring automatic platform detection and multi-modal support for text, video subtitles, and audio transcription.

Tokens
59.9K
Snippets
157
Records
255
Agent score
68%

What's inside feedgrab

  1. Overview of feedgrab

    main

    feedgrab is an open-source content collector designed to fetch content from over 18 mainstream media platforms using URLs or keywords. It automatically detects the platform, fetches the content (text, video, or audio), and exports it as metadata-rich, Obsidian-compatible Markdown files.

    Core Workflow:

    1. Input: Provide a URL or keyword.
    2. Detection: The tool auto-detects the platform.
    3. Fetching: Uses various methods depending on the platform:
      • Text: Jina Reader
      • Video: yt-dlp for subtitles or Whisper transcription for audio.
      • APIs: Bilibili, RSS, Telegram, YouTube Data API v3, GitHub REST API, Feishu Open API, or Discourse Topic JSON.
      • X/Twitter: A multi-layered approach including GraphQL, FxTwitter, Syndication, oEmbed, Jina, and Playwright.
    4. Output: Saves files to a structured directory: output/[Platform]/[Author]_[Date]：[Title].md.

  2. Overview of feedgrab capabilities

    main

    feedgrab is a multi-platform content fetching tool that provides a Python CLI and an MCP server. It is designed to extract text, media, and structured data from a wide variety of URLs, including social media, video platforms, and documentation sites.

    Core Capabilities:

    • Universal URL Fetching: Fetch content from almost any URL using a dispatcher (UniversalReader).
    • Deep X/Twitter Extraction: Uses GraphQL for deep fetching of tweets, threads, and media. Supports batch fetching of bookmarks, user tweets, and Twitter Lists.
    • Video & Podcast Processing: Extracts metadata and can generate full transcripts via subtitles or Whisper.
    • Batch Operations: Supports batch fetching for Twitter users (followers/following/members), WeChat accounts, and XHS (Xiaohongshu) notes.
    • Structured Analysis: Integrates with 'skills' to provide content analysis and structured reports.

  3. What is feedgrab

    main

    feedgrab is an open-source content scraping and batch collection tool designed for 18+ mainstream social media and content platforms. It automatically detects the platform from a provided URL or keyword, scrapes the content, and outputs unified, Obsidian-compatible Markdown files with complete metadata.

    Key features include:

    • Automatic Platform Detection: Identifies the source platform from the input.
    • Unified Output: Converts various content types into structured Markdown files compatible with Obsidian.
    • Low Cost: Prioritizes free interfaces and APIs, minimizing the need for paid subscriptions.
    • Multi-modal Support: Handles text (via Jina Reader), video (via InnerTube API and yt-dlp subtitles), and audio (via Whisper transcription).
    • Extensive Platform Support: Includes Bilibili, RSS, Telegram, YouTube, GitHub, Feishu, Discourse, and X (Twitter).

  4. Supported Platforms and Extraction Methods

    main

    feedgrab supports a wide range of platforms with varying extraction capabilities for text and media.

    Key Platform Capabilities:

    • YouTube: Uses InnerTube API (zero dependency/quota) and YouTube Data API v3. Supports video/audio transcription via yt-dlp and Groq Whisper.
    • Bilibili: API metadata extraction and 3-level subtitle fallback (including WBI signature support).
    • Xiaoyuzhou (小宇宙): SSR __NEXT_DATA__ extraction and automatic Groq Whisper transcription.
    • X / Twitter: Advanced 6-tier fallback strategy (GraphQL, FxTwitter, Syndication, oEmbed, Jina, Playwright).
    • WeChat Official Accounts (微信公众号): Jina/Playwright extraction, Sogou search, or MP backend API (by account or album).
    • GitHub: REST API for repository metadata, including Chinese README priority and relative image link completion.
    • Xiaohongshu (小红书): API (xhshow) and browser-based extraction (via Pinia Store injection). Supports single posts, author batch, and keyword search.
    • Feishu/Lark: Open API and CDP direct connection. Supports wiki batch extraction and table embedding.
    • Reddit: .json extraction via CDP/Playwright with support for different reply modes (REDDIT_REPLY_MODE=top/tree/all).
    • Paid News Sites: 7-tier bypass strategy (JSON-LD, Googlebot/Bingbot UA, AMP, archive.today, etc.).

  5. Video & Podcast Digest Skill Overview

    main

    The Video & Podcast Digest Skill automates the process of converting media links into full transcripts and structured summaries. When a supported media URL is detected, the skill follows a pipeline to extract subtitles or download audio, transcribe it using Groq Whisper, and generate a summary.

    Supported Platforms

    PlatformTypeSubtitlesWhisper Transcription
    YouTubeVideo✅✅
    BilibiliVideo✅✅
    X/TwitterVideo❌✅
    Xiaoyuzhou (小宇宙)Podcast❌✅
    Apple PodcastsPodcast❌✅
    Direct links (mp3/mp4/m3u8)Any❌✅

    Triggering

    The skill is auto-triggered by detecting specific URL patterns:

    • YouTube: youtube.com, youtu.be
    • Bilibili: bilibili.com, b23.tv
    • X/Twitter: x.com, twitter.com (video tweets)
    • Xiaoyuzhou: xiaoyuzhoufm.com
    • Apple Podcasts: podcasts.apple.com
    • Direct Media: .mp3, .mp4, .m3u8, .m4a, .webm

  6. Overview of feedgrab's three usage layers

    main

    feedgrab is designed to be used at three different levels of integration depending on your needs:

    1. Python CLI/Library: The core layer providing basic content scraping and a unified data structure. This is the mandatory foundation.
    2. Claude Code Skills: An optional layer that adds AI-driven capabilities like video transcription, content analysis, and automated scraping via npx skills add iBigQiang/feedgrab.
    3. MCP Server: An optional layer that exposes reading capabilities as Model Context Protocol (MCP) tools, allowing AI agents to use feedgrab directly.

  7. Understand Twitter metadata and Obsidian-compatible output

    main

    When using Twitter with valid cookies, feedgrab extracts rich metadata and formats it into an Obsidian-compatible YAML front matter block. This ensures compatibility with Obsidian Properties.

    Extracted Metadata Fields

    The following fields are included in the output if they are available:

    • title: The title of the content.
    • source: The original URL.
    • author: The Twitter handle (e.g., @username).
    • author_name: The display name of the author.
    • published: The publication date.
    • created: The creation date.
    • cover_image: The URL of the first image/video found.
    • tweet_count: Number of tweets in a thread.
    • has_thread: Boolean indicating if it's a thread.
    • likes, retweets, replies, bookmarks, views: Engagement metrics.
    • tags: Associated tags (e.g., clippings, twitter).
    ---
title: "OpenClaw新手完整学习路径"
source: "https://x.com/AI_Jasonyu/status/123"
author:
  - "@AI_Jasonyu"
author_name: "鱼总聊AI"
published: 2026-02-26
created: 2026-02-26
cover_image: "https://pbs.twimg.com/media/xxx.jpg"
tweet_count: 3
has_thread: true
likes: 1234
retweets: 567
replies: 89
bookmarks: 234
views: 45678
tags:
  - "clippings"
  - "twitter"
---

  8. Use the Syndication API as a fallback for Twitter data

    main

    When GraphQL or oEmbed requests fail due to expired cookies or rate limiting, feedgrab uses the Twitter Syndication API (cdn.syndication.twimg.com) as a 'Tier 0.5' fallback. This allows the system to still retrieve approximately 80% of tweet data, including text, media URLs, interaction metrics, and user info.

    Data Capabilities:

    • Text, images, and videos
    • Hashtags, likes, and replies
    • Article detection (if an article is detected, feedgrab uses Jina to fetch the full body)
    • Missing: Retweets, bookmarks, and view counts.

    Fallback Hierarchy:

    1. Tier 0: GraphQL
    2. Tier 0.5: Syndication
    3. Tier 1: oEmbed
    4. Tier 2: Jina
    5. Tier 3: Playwright
    Endpoint: https://cdn.syndication.twimg.com/tweet-result?id={tweetId}&token={token}

  9. Understand the feedgrab output directory structure

    main

    The project uses a flattened directory structure to organize scraped content and metadata. The structure is organized under the platform (e.g., X/ for Twitter).

    Directory Layout:

    • {OUTPUT_DIR}/X/index/:
      • item_id_url.json: The global deduplication index containing item_id mappings to URLs.
      • bookmarks_*.json: Batch records for bookmark scraping.
      • status_{screen_name}_all_*.json: Batch records for user tweet scraping.
    • {OUTPUT_DIR}/X/status/: Single-tweet scrapes.
    • {OUTPUT_DIR}/X/status_{display_name}/: Batch scrapes of a specific user's tweets.
    • {OUTPUT_DIR}/X/bookmarks/: All bookmarks (unorganized by folder).
    • {OUTPUT_DIR}/X/bookmarks_{folder_name}/: Bookmarks belonging to a specific Twitter folder.

  10. Use breakpoint resumption for Twitter scraping

    main

    The scraping process is split into two phases to prevent data loss during crashes or network interruptions:

    1. Phase 1 (Discovery): Tweets are written in real-time to a .api_discovery_{username}.jsonl cache file. If interrupted, the process resumes from the smallest ID found in the cache.
    2. Phase 2 (Processing): The deduplication index is automatically persisted every 50 items. When re-running, the system automatically skips tweets that have already been saved.

  11. Use curl_cffi for browser-like TLS fingerprinting

    main

    To avoid detection by servers that check TLS fingerprints (JA3/JA4), feedgrab uses curl_cffi to impersonate a Chrome browser. This is handled internally via the http_client.py utility.

    Technical Details:

    • The client uses Session(impersonate="chrome") to match real browser TLS handshakes.
    • It provides a compatibility layer that wraps curl_cffi exceptions into standard requests exceptions (requests.Timeout, requests.ConnectionError, requests.HTTPError), so existing error handling logic remains functional.
    • This is used across multiple fetchers including jina, bilibili, twitter, wechat_search, and youtube to ensure consistent fingerprinting.

  12. Understand the X/Twitter six-tier fallback strategy

    main

    Feedgrab employs a multi-tier strategy to ensure content is fetched even when certain methods fail or authentication is missing:

    1. Tier 0 (GraphQL API): Requires auth_token and ct0. Provides complete threads, media, and engagement metrics.
    2. Tier 0.3 (FxTwitter API): No auth required. Provides text, images, videos, and engagement (including views/bookmarks) but lacks blue_verified, listed_count, and thread expansion.
    3. Tier 0.5 (Syndication API): No auth required. Provides text, images, videos, and basic engagement. Useful when cookies expire.
    4. Tier 1 (oEmbed API): No auth required. Single public tweet text only.
    5. Tier 2 (Jina Reader): No auth required. Profiles and non-tweet pages.
    6. Tier 3 (Playwright): Optional session. Last resort for login-required content.