DOCSEARCHbyAlgolia
Submit Docs

MediaCrawler

repository·main·Indexed 13 days ago

https://github.com/nanmicoder/mediacrawler

A multi-platform social media data collection tool supporting Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu. It utilizes Playwright-based browser automation and features a Chrome DevTools Protocol (CDP) mode to reduce anti-crawler detection by reusing existing browser sessions, cookies, and extensions. Includes a FastAPI-based WebUI for configuration and monitoring, and supports multiple storage formats including JSONL, Excel, CSV, JSON, SQLite, and MongoDB.

Tokens
27.4K
Snippets
97
Records
142
Agent score
100%

What's inside MediaCrawler

  1. Core features of MediaCrawler

    main

    MediaCrawler is an asynchronous multi-platform crawler framework built with Python asyncio. Key features include:

    • Multi-platform support: Unified crawler interface for 7 major platforms.
    • Multiple login methods: Supports QR code, mobile number, and Cookie-based authentication.
    • Diverse storage options: Supports CSV, JSON, JSONL, SQLite, MySQL, MongoDB, and Excel.
    • Anti-anti-crawler measures: Includes CDP (Chrome DevTools Protocol) mode, proxy IP pools, and request signing.
    • Asynchronous high concurrency: Built on asyncio for efficient crawling.
    • Word cloud generation: Automatically generates word clouds for comments.

  2. Choose a data storage method in MediaCrawler

    main

    MediaCrawler supports several storage formats depending on your needs. Data is typically saved to the data/ directory.

    File-based Storage

    • JSONL: The default format. Each line is a single JSON object. It offers high performance for append-only writes.
    • Excel: Recommended for data analysis. Supports multiple worksheets (Content, Comments, Creators) with professional formatting (headers, auto-column width, borders).
    • CSV: Standard comma-separated values.
    • JSON: Standard JSON array format.

    Database Storage

    For database storage, you must first initialize the database using the --init_db flag before running the crawler with the corresponding --save_data_option.

    TypeInitialization FlagStorage OptionBest For
    SQLite--init_db sqlite--save_data_option sqlitePersonal use, lightweight, no server required (Recommended)
    MySQL--init_db mysql--save_data_option dbRelational data (Note: use db for compatibility)
    PostgreSQL--init_db postgres--save_data_option postgresProduction environments (Recommended)

  3. Understand the MediaCrawler project structure

    main

    MediaCrawler is organized into several functional modules. Understanding this structure is essential for extending the crawler or customizing data storage and platform support:

    • base/: Contains the core abstractions, such as base_crawler.py.
    • media_platform/: Contains the specific implementation logic for different social media platforms (e.g., bilibili, douyin, xhs, zhihu).
    • model/: Defines the data models for each platform (e.g., m_douyin.py, m_zhihu.py).
    • store/: Handles the data storage implementation for each platform.
    • database/: Manages database ORM, sessions, and model definitions.
    • config/: Contains configuration files, including base, database, and platform-specific settings.
    • cache/: Provides caching mechanisms with an abstraction layer and implementations for local_cache and redis_cache.
    • proxy/: Manages proxy logic, including base classes, providers, and IP pools.
    • tools/: Contains utility functions and browser control tools like browser_launcher.py and cdp_browser.py.
    • libs/: Holds JavaScript files used for platform-specific logic, such as douyin.js (Sign function) or stealth.min.js (browser fingerprinting evasion).
    • main.py: The primary entry point for the application.

  4. What is CDP mode and its advantages

    main

    CDP (Chrome DevTools Protocol) mode is an advanced anti-detection crawling technique that controls an existing Chrome or Edge browser. Unlike traditional Playwright automation, CDP mode offers:

    • Real Browser Environment: Uses your actual installed browser with all extensions, plugins, and personal settings.
    • Superior Anti-detection: Browser fingerprints are more authentic, making it harder for websites to identify automation tools.
    • State Retention: Automatically inherits your existing login sessions, Cookies, and browsing history.
    • Extension Support: Leverages installed tools like ad blockers or proxy extensions.
    • Natural Behavior: Browser patterns closely mimic real human users.

  5. Configure Chrome CDP Mode for reduced risk

    main

    MediaCrawler supports CDP (Chrome DevTools Protocol) mode by default. This mode connects to your existing Chrome browser, reusing your login state, cookies, and extensions, which significantly reduces the risk of platform anti-crawler detection.

    Setup Steps:

    1. Install Chrome (version >= 144).
    2. Enable remote debugging: Go to chrome://inspect/#remote-debugging in Chrome and check "Allow remote debugging for this browser instance".
    3. Verify it is running at 127.0.0.1:9222.
    4. When running the crawler, a confirmation dialog will appear in Chrome; click "Accept" within 60 seconds.

    To disable this and use standard Playwright mode, set ENABLE_CDP_MODE = False in config/base_config.py.

  6. MediaCrawler directory structure overview

    main

    The project is organized into several functional directories:

    • main.py: The main program entry point.
    • base/: Contains fundamental abstract classes like base_crawler.py (crawler, login, storage, and client base classes).
    • config/: Configuration management (core, DB, and platform-specific configs).
    • media_platform/: Implementation of platform-specific crawlers (e.g., xhs/, douyin/).
    • store/: Data storage implementations.
    • database/: Database layer including ORM models and session management.
    • proxy/: Proxy IP pool management and providers.
    • cache/: Caching systems (Local, Redis).
    • tools/: Utility modules for browser launching, CDP management, and async file writing.
    • model/: Pydantic data models for platforms.
    • cmd_arg/: Command-line argument definitions.

  7. Understand the browser management modes

    main

    MediaCrawler supports two primary browser execution modes:

    1. Standard Mode (Playwright)

    Uses chromium.launch() to start a fresh browser instance. It automatically injects stealth.js to enhance anti-detection capabilities and creates a new browser context.

    2. CDP Mode (Chrome DevTools Protocol)

    Instead of launching a new process, it detects an existing browser path and uses connect_over_cdp() to connect to a running browser instance. This mode is useful for:

    • User Data Persistence: Using existing browser profiles and cookies.
    • Enhanced Anti-Detection: Leveraging a real, manually-used browser environment.
    • Debugging: Connecting to a browser already running with a debug port.

  8. How MediaCrawler works

    main

    MediaCrawler is a multi-platform social media data collection tool that supports platforms like Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu.

    It uses Playwright as its core automation framework to handle logins and maintain login states. Instead of performing complex JavaScript reverse engineering to bypass encryption, it leverages the browser's context and preserved login state to obtain signature parameters via JS expressions. This significantly lowers the technical barrier for scraping complex platforms.

  9. Understand the MediaCrawler system architecture

    main

    MediaCrawler follows a layered architecture designed for scalability and modularity:

    1. Entry Layer: Handles main.py entry point, command-line arguments (cmd_arg), and configuration management (config).
    2. Core Crawler Layer: Uses a CrawlerFactory to instantiate platform-specific crawlers (e.g., XiaoHongShuCrawler) which inherit from AbstractCrawler.
    3. API Client Layer: Platform-specific clients (e.g., XiaoHongShuClient) inherit from AbstractApiClient to handle HTTP requests and cookie management.
    4. Data Storage Layer: Uses a StoreFactory to route data to various storage implementations like CSV, JSON, SQLite, MySQL, MongoDB, or Excel.
    5. Infrastructure Layer: Manages low-level utilities including browser management (Playwright/CDP), proxy IP pools, caching systems, and login management.

  10. Understand the Excel file structure

    main

    MediaCrawler generates multi-sheet workbooks. Files are saved to data/{platform}/ with the format {platform}_{crawler_type}_{timestamp}.xlsx. The workbook contains three main sheets:

    • Contents: Post/video information (e.g., note_id, title, desc, user_id, liked_count, note_url).
    • Comments: Comment data (e.g., comment_id, note_id, content, user_id, like_count, create_time).
    • Creators: Author/profile information (e.g., user_id, nickname, gender, avatar, fans, follows).

    Empty sheets are automatically removed to keep the file clean.

  11. Key differences between MediaCrawler and MediaCrawlerPro

    main

    MediaCrawlerPro is a complete refactor of the original MediaCrawler project, addressing limitations regarding multi-account support, Linux deployment, and dependency weight.

    Main improvements in the Pro version:

    • Decoupled Dependencies: Removed the heavy Playwright dependency from the crawler core.
    • Simplified Deployment: Added support for Docker and Docker-compose for easier setup.
    • Enhanced Stability: Built-in support for multi-account management and IP proxy pools.
    • Modular Logic: Introduced a new signature service to decouple signature logic, making the crawler more flexible.
    • Platform Support: Includes implementations for Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Baidu Tieba, Zhihu, and more.

  12. MediaCrawler crawler lifecycle

    main

    The typical lifecycle of a MediaCrawler execution follows this sequence:

    1. Initialization: main.py uses CrawlerFactory to create a specific crawler instance (e.g., XiaoHongShuCrawler).
    2. Startup: crawler.start() is called.
    3. Infrastructure Setup: The crawler initializes an IP pool (if enabled) and launches a browser (Standard or CDP mode).
    4. Client Creation: A platform-specific client (e.g., XiaoHongShuClient) is created, and its login status is checked via pong().
    5. Authentication: If not logged in, the AbstractLogin implementation (e.g., login_by_qrcode) is triggered.
    6. Execution Modes:
      • Search Mode: Uses get_note_by_keyword() to find content, then loops through get_note_by_id() to fetch details.
      • Detail Mode: Directly calls get_note_by_id() for a specific ID.
      • Creator Mode: Calls get_creator_info() to fetch all content from a specific user.
    7. Persistence: Data is passed to the StoreFactory to be saved (e.g., store_content()).
    8. Cleanup: crawler.cleanup() is called, and the browser is closed.