wechat_articles_spider

This project is a WeChat Official Account crawler designed to retrieve article information.

Public Account Capabilities:

• Retrieve public account information.
• Get the biz ID (via Qingbo or web pages).
• Get the number of articles published by an account (approximate).

Article Capabilities:

• Get article URLs (via web pages, PC WeChat, mobile WeChat, or WeChat Reading).
• Get detailed article information (including likes, read counts, and comments) for a specific account (requires manual loop adjustment).
• Get information for a specific article.
• Download WeChat articles to local HTML files (with optional image saving).

The project consists of several modules for interacting with WeChat Official Accounts. Note that ArticlesAPI is no longer maintained.

• AccountBiz: Retrieves the biz parameter based on the Official Account name.
• ArticlesInfo: Retrieves specific article details (e.g., read counts, likes) using an article link.
• ArticlesUrls: Retrieves the latest or historical articles for a specific Official Account. Requires personal credentials such as cookie, token, and key.
• Url2Html: Downloads the article HTML content to a local file, which can then be converted to Word, PDF, or other formats.

The wechatarticles package is a spider designed to crawl WeChat Official Account articles and retrieve metrics such as read counts and likes. It is organized into several specialized submodules:

• wechatarticles.AccountBiz: Handles account-related business logic.
• wechatarticles.ArticlesAPI: Provides interfaces for interacting with WeChat article APIs.
• wechatarticles.ArticlesInfo: Manages article information and metadata.
• wechatarticles.ArticlesUrls: Handles the retrieval and management of article URLs.
• wechatarticles.Url2Html: Converts article URLs into HTML content.
• wechatarticles.utils: Contains utility functions used across the package.

To use the crawler, you must manually obtain specific authentication parameters.

• For official_cookie and token, refer to the documentation at docs/get_cookie_token.md.
• For wechat_cookie and appmsg_token, refer to the documentation at docs/get_appmsg_token.md.

The get_params.py script automates the process of running mitmdump to intercept traffic from mp.weixin.qq.com/mp/getappmsgext. It monitors network flows and extracts the appmsg_token and Cookie once a relevant WeChat article request is detected.

Prerequisites

• The mobile device and the computer must be on the same local area network (LAN).
• Manually configure the mobile device's proxy settings to use the computer's static IP and port 8080.

Usage

Run the script from the command line, providing a filename for the output log:

python get_params.py <outfile>

When you open a WeChat article on your mobile device, the script will automatically process the intercepted request and return the parameters.

To scrape WeChat article data (such as read counts and likes), you must first capture the network requests from the WeChat PC client to extract necessary authentication parameters.

1. Start Fiddler to begin monitoring network traffic.
2. Log in to the WeChat PC client.
3. Open any article from the target WeChat Official Account.
4. In Fiddler, look for requests containing the pattern /mp/getappmgsext?.... This is the URL for the article content.
5. Inspect the request details and locate the WebForms section to extract the required parameters.
6. To find specific metrics, inspect the JSON response body; the fields read_num and like_num correspond to the article's read count and like count, respectively.

Note: You can use other proxy/packet capture tools such as Charles, mitmproxy, or anyproxy to achieve the same result.

Install the package using the Python package manager:

pip install wechatarticles

To automatically capture WeChat parameters (like appmsg_token and Cookie), you need to install mitmproxy. This tool acts as a proxy to intercept the necessary network requests.

You can install it via pip:

pip3 install mitmproxy

Alternatively, you can download the appropriate version for your operating system from the official mitmproxy releases.

To use the spider, you must manually retrieve the Cookie and token from your logged-in WeChat Official Account session.

1. Log in to the WeChat Official Account Platform using a personal subscription account.
2. Open your browser's Developer Tools (F12), using Chrome or Firefox.
3. Refresh the page.
4. Navigate to the Network tab in the Developer Tools.
5. Locate the requests in the network log to find and copy the Cookie and token values. These values must be pasted into your code to authenticate the spider.

If the crawler fails, check the following:

1. Network Proxies: Ensure any network proxy or packet capture software (like Fiddler) is either closed or configured with the correct parameters.
2. Parameter Validity: Ensure cookie and token parameters are current. These must be obtained from an arbitrary article belonging to the target public account.
3. Account Status: Ensure you are following the target public account (may be required).
4. Rate Limiting:
   • When fetching URLs via the web, set intervals (e.g., 3 minutes per page).
   • When fetching read/like counts, set an interval of 5-10 seconds per article. If blocked, wait approximately 5-10 minutes before retrying.