x-reader

repository·main·Indexed 20 days ago

https://github.com/runesleo/x-reader

A universal content reader (v0.2.0) designed to fetch, normalize, and digest content from 7+ platforms, including YouTube, X, WeChat, Telegram, Bilibili, and Xiaohongshu. It provides a Python library and CLI for structured content extraction, an MCP server for LLM integration, and specialized Claude Code skills for Whisper-based video/podcast transcription and multi-dimensional AI content analysis.

Tokens
8.2K
Snippets
27
Records
36
Agent score
75%

What's inside x-reader

  1. Overview of the Video & Podcast Digest Skill

    main

    The Video & Podcast Digest Skill is a pipeline that converts media links (video or podcast) into a full transcript and a structured summary. It automatically detects the media type, extracts subtitles or audio, transcribes the audio using Whisper (via Groq), and generates a formatted digest.

    Supported Platforms

    PlatformTypeSubtitlesWhisper Transcription
    YouTubeVideo
    BilibiliVideo
    X/TwitterVideo
    Xiaoyuzhou (小宇宙)Podcast
    Apple PodcastsPodcast
    Direct links (mp3/mp4/m3u8)Any
  2. Customize the Content Analyzer dimensions

    main

    The analysis report is highly customizable. You can modify the dimensions used in the Multi-Dimensional Analysis (Step 2) and the Personalized Relevance (Step 3) to match your specific domain.

    Examples of customization:

    • Traders: Add dimensions like Market Impact and Risk Assessment.
    • Developers: Add dimensions like Architecture Patterns and Tech Debt.

    Standard Dimensions available for customization:

    • Summary: Core thesis, source, and type.
    • Key Insights: Includes Core Arguments, Tools & Methods, Workflow Ideas, Data & Numbers, Risks & Warnings, Resources, and Mental Model Shifts.
    • Action Items: Categorized by Quick Wins, Deeper Work, and Exploration.
    • Personalized Relevance: Mapping insights to My Projects, My Knowledge Base, and My Decision Log.
  3. How the Video & Podcast Digest Pipeline works

    main

    The skill follows a multi-step pipeline based on the detected URL pattern:

    1. Step 0: Detect Media Type: Identifies if the URL is a Podcast (Xiaoyuzhou, Apple), Video (Bilibili, YouTube), or a direct audio link.
    2. Step 1: Extraction:
      • Videos: Attempts to extract subtitles using yt-dlp. If subtitles exist, it skips to transcription. If not, it downloads the audio.
      • Xiaoyuzhou: Extracts the CDN audio URL from the __NEXT_DATA__ object in the HTML and downloads it.
      • Apple Podcasts: Uses yt-dlp to extract audio.
      • Bilibili: Uses the Bilibili API directly to fetch the audio stream (bypassing yt-dlp 412 errors).
    3. Step 2: Transcription: Uses Whisper via the Groq API. If the audio file is >25MB, it is split into 10-minute segments using ffmpeg and transcribed sequentially.
    4. Step 3: Summary: Generates a structured text summary (Overview, Key Points, Quotes, etc.) based on whether the media is a short video or a long podcast.
  4. How x-reader's three-layer architecture works

    main

    x-reader is designed with three distinct layers that can be used independently or together:

    1. Python CLI/Library (Core): The foundation. It handles platform identification, content fetching (using Jina Reader for text, yt-dlp for subtitles, etc.), and returns data in a unified schema. This layer is required.
    2. Claude Code Skills (Optional): An extension for AI agents. It provides full Whisper transcription for videos/podcasts and structured AI analysis reports. These are located in the skills/ directory.
    3. MCP Server (Optional): An interface layer that exposes the core reading capabilities as tools for MCP-compatible clients (like Claude Desktop).

    Platform Support Summary:

    • YouTube: Text and Video/Audio (via subtitles or Groq Whisper).
    • Bilibili: Text (via API) and Video/Audio (via Skills).
    • X / Twitter: Text (via oEmbed, FxTwitter, Jina, or Playwright with saved sessions).
    • Telegram: Text (via Telethon).
    • RSS/WeChat/Xiaohongshu: Text support available.
  5. Set up x-reader as an MCP Server

    main

    You can expose x-reader as Model Context Protocol (MCP) tools. This requires cloning the repository and installing the mcp extra.

    Setup Steps:

    1. Clone the repo and install with MCP support:
      git clone https://github.com/runesleo/x-reader.git
      cd x-reader
      pip install -e ".[mcp]"
    2. Run the server:
      python mcp_server.py

    Claude Desktop Configuration: Add the following to your ~/.claude/claude_desktop_config.json to use it in Claude Desktop:

    {
        "mcpServers": {
            "x-reader": {
                "command": "python",
                "args": ["/path/to/x-reader/mcp_server.py"]
            }
        }
    }

    Exposed MCP Tools:

    • read_url(url): Fetch any URL.
    • read_batch(urls): Fetch multiple URLs concurrently.
    • list_inbox(): View previously fetched content.
    • detect_platform(url): Identify platform from URL.
  6. Use the Content Analyzer Skill

    main

    The Content Analyzer Skill transforms any content (URLs, text, or transcripts) into a structured analysis report with actionable insights. It follows a three-step pipeline: fetching the content, performing multi-dimensional analysis, and mapping insights to your personal context.

    Triggers

    You can trigger the analysis using the following methods:

    • Use the command: /analyze [URL]
    • Use natural language: "Analyze this article" or "What are the key takeaways?"
    • Automatic trigger: The skill is automatically triggered after a video or podcast transcription is completed by the video skill.
    /analyze https://example.com/article
  7. Extract audio from Bilibili via API

    main

    Because yt-dlp often returns a 412 error for Bilibili even with cookies, the skill uses the Bilibili API to fetch the audio stream directly. This requires extracting the BV number and the CID first.

    # 1. Extract BV number from URL
    BV="BV1xxxxx"  # Replace with actual BV number
    
    # 2. Get video info (title, duration, CID)
    curl -s "https://api.bilibili.com/x/web-interface/view?bvid=$BV" \
      -H "User-Agent: Mozilla/5.0" -H "Referer: https://www.bilibili.com/" \
      | python3 -c "import json,sys; d=json.load(sys.stdin)['data']; print(f\"Title: {d['title']}\nDuration: {d['duration']}s\nCID: {d['cid']}\")"
    
    # 3. Get audio stream URL
    CID=<CID from previous step>
    AUDIO_URL=$(curl -s "https://api.bilibili.com/x/player/playurl?bvid=$BV&cid=$CID&fnval=16&qn=64" \
      -H "User-Agent: Mozilla/5.0" -H "Referer: https://www.bilibili.com/" \
      | python3 -c "import json,sys; print(json.load(sys.stdin)['data']['dash']['audio'][0]['baseUrl'])")
    
    # 4. Download audio (Referer header required, otherwise 403)
    curl -L -o /tmp/media_audio.m4s \
      -H "User-Agent: Mozilla/5.0" -H "Referer: https://www.bilibili.com/" "$AUDIO_URL"
    
    # 5. Convert to mp3
    ffmpeg -y -i /tmp/media_audio.m4s -acodec libmp3lame -q:a 5 /tmp/media_audio.mp3
  8. Extract subtitles from YouTube or Bilibili

    main

    For video platforms, the skill first attempts to extract existing subtitles to avoid the cost/time of transcription.

    YouTube: Prefers English, falls back to Chinese. Bilibili: Uses Chinese subtitles.

    # YouTube (prefer English, fallback Chinese)
    yt-dlp --skip-download --write-auto-sub --sub-lang "en,zh-Hans" -o "/tmp/media_sub" "VIDEO_URL"
    
    # Bilibili
    yt-dlp --skip-download --write-auto-sub --sub-lang "zh-Hans,zh" -o "/tmp/media_sub" "VIDEO_URL"
  9. Handle large audio files (>25MB)

    main

    The Groq API has a maximum file size limit of 25MB. If the audio file exceeds this, it must be split into segments (e.g., 10-minute chunks) using ffmpeg before transcription. Segments must be processed sequentially to avoid Groq 524 timeouts.

    # Get total duration
    DURATION=$(ffprobe -v error -show_entries format=duration -of csv=p=0 /tmp/media_audio.* | head -1)
    
    # Split into 10-minute segments (keeps each under 25MB)
    SEGMENT_SEC=600
    SEGMENTS=$(python3 -c "import math; print(math.ceil(float('$DURATION')/$SEGMENT_SEC))")
    
    # Cut segments
    for i in $(seq 0 $((SEGMENTS-1))); do
      START=$((i * SEGMENT_SEC))
      ffmpeg -y -i /tmp/media_audio.* -ss $START -t $SEGMENT_SEC -acodec libmp3lame -q:a 5 \
        "/tmp/media_segment_${i}.mp3" 2>/dev/null
    done
  10. Install Claude Code Skills

    main

    To enable video/podcast transcription (via Whisper) and AI-powered content analysis in Claude Code, you must manually copy the skills/ directory from the cloned repository to your Claude Code skills directory. This is not included in the pip installation.

    Installation Steps:

    1. Clone the repository.
    2. Set your skills directory environment variable.
    3. Copy the video and analyzer skill folders.
    export CLAUDE_SKILLS_DIR="/path/to/claude-code-skills"
    mkdir -p "$CLAUDE_SKILLS_DIR"
    cp -r skills/video "$CLAUDE_SKILLS_DIR/video"
    cp -r skills/analyzer "$CLAUDE_SKILLS_DIR/analyzer"
  11. Extract audio from Xiaoyuzhou (小宇宙)

    main

    Xiaoyuzhou is a Next.js SPA. The audio URL can be extracted from the __NEXT_DATA__ object within the initial HTML using curl and grep.

    # Extract CDN direct link from __NEXT_DATA__
    AUDIO_URL=$(curl -sL -H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" \
      "EPISODE_URL" \
      | grep -oE 'https://media\.xyzcdn\.net/[^"]+\.(m4a|mp3)' \
      | head -1)
    
    echo "Audio URL: $AUDIO_URL"
    
    # Download audio
    curl -L -o /tmp/media_audio.mp3 "$AUDIO_URL"
  12. Handle X (Twitter) gated content and sessions

    main

    To read X Articles or login-required X pages, use the Playwright fallback by logging in once via the CLI.

    Login Command:

    x-reader login twitter

    Reading gated content:

    x-reader "https://x.com/user/status/123"

    External Session Cookies: By default, local X cookies stay local. If you want to allow Jina Reader to use your saved X session for gated Articles, set the following environment variable:

    export X_READER_ALLOW_EXTERNAL_SESSION_COOKIES=1
    x-reader login twitter
    x-reader "https://x.com/user/status/123"