f2 Documentation

The Weibo application provides several layers of interfaces for interacting with Weibo data:

1. Handler Interfaces: High-level methods for downloading single or user-specific Weibo content (handle_one_weibo, handle_user_weibo).
2. Data Method Interfaces: Core logic for user management and data retrieval (e.g., fetch_user_info, fetch_user_weibo).
3. Utils Interfaces: Utility classes for configuration, cookie generation, and ID/nickname extraction (e.g., ClientConfManager, WeiboIdFetcher).
4. Crawler Interfaces: Low-level URL-based crawling via WeiboCrawler.
5. Downloader Interfaces: Media and text downloading via WeiboDownloader.

Key Implementation Notes:

• Pagination: Pagination parameters are included in the previous request's data and can be accessed via the built-in filter.
• Async Iteration: All interfaces involving pagination use asynchronous generators. Use async for to iterate through results and automatically handle pagination.
• Max Counts: If max_counts is set to None or omitted, all available Weibo data will be retrieved.
• Authentication: Using logged-in cookie values allows bypassing certain privacy settings.

The Weibo application provides several layers of APIs for interacting with Weibo data:

1. CLI Interface: High-level commands for downloading single posts or entire user feeds (handle_one_weibo, handle_user_weibo).
2. Data Methods API: Developer-facing asynchronous methods for user management, info retrieval, and data fetching (e.g., fetch_user_info, fetch_user_weibo).
3. Utils API: Utility classes for configuration, cookie generation, and ID extraction (e.g., ClientConfManager, WeiboIdFetcher).
4. Crawler API: Low-level crawling logic encapsulated in the WeiboCrawler class.
5. Downloader API: Media and content downloading logic via WeiboDownloader (images, videos, descriptions).

Key Usage Tips

• Pagination: APIs using pagination are implemented as asynchronous generators. You must use async for to iterate through them to handle automatic pagination.
• Pagination Filtering: Pagination parameters are included in previous request data and can be filtered using the built-in filter function.
• Fetch Limits: When max_counts is set to None or omitted, the API will attempt to fetch all available Weibo data.
• Privacy: Using a logged-in cookie can help bypass certain account privacy settings.
• Integration: Designed to integrate easily with FastAPI, Flask, and Django.

The TikTok implementation provides several layers of interfaces for different use cases:

1. CLI Handlers: High-level methods for downloading content (e.g., handle_one_video, handle_user_post, handle_user_like, handle_user_collect, handle_user_mix, handle_search_video, handle_user_live).
2. Data & Function Interfaces: Methods for fetching raw data (e.g., fetch_user_profile, fetch_user_post_videos, fetch_one_video, fetch_search_videos).
3. Utils Interfaces: Utility classes for managing configuration, tokens (TokenManager), parameters (XBogusManager), and ID extraction (SecUserIdFetcher, AwemeIdFetcher).
4. Crawler Interfaces: Low-level URL/endpoint fetching via TiktokCrawler.
5. Downloader Interfaces: Methods for managing download tasks and streams via TiktokDownloader.

Key Implementation Status Symbols:

• 🟢 Implemented
• 🟡 Implementing/Modifying
• 🟤 Not implemented
• 🔵 Future possibility
• 🔴 Deprecated

The Twitter integration provides several layers of interfaces for interacting with Twitter data:

• CLI Interface: High-level methods for downloading specific types of data (single tweet, home tweets, liked tweets, bookmarked tweets).
• Data Methods Interface (Developer API): Core asynchronous methods for fetching user profiles, single tweets, and collections of tweets (home, liked, bookmarked).
• Crawler Interface: Low-level TwitterCrawler class methods for raw data retrieval.
• Downloader Interface: TwitterDownloader class methods for handling download tasks, including text, videos, and image galleries.
• Utils Interface: Utility classes like ClientConfManager, UniqueIdFetcher, and TweetIdFetcher for configuration and ID extraction.

Key Implementation Notes:

• Pagination: Pagination parameters are included in the previous request's data and can be retrieved using the built-in filter.
• Asynchronous Iteration: All APIs supporting pagination use asynchronous generator methods. Use async for to iterate through results.
• Fetch Limits: When max_counts is set to None or omitted, all available tweet data will be fetched.
• Integration: These APIs are designed to be easily integrated with backend frameworks like FastAPI, Flask, and Django.

The utility provides several ways to manage the local storage structure for Twitter data.

Directory Structure

If no path is specified in the configuration, it defaults to Download. The structure follows this pattern:

├── Download
│   ├── twitter
│   │   ├── post
│   │   │   ├── user_nickname
│   │   │   │   ├── [timestamp]_[desc]
│   │   │   │   │   ├── [timestamp]_[desc]-video.mp4
│   │   │   │   │   ├── [timestamp]_[desc]-desc.txt
│   │   │   │   │   └── [timestamp]_[desc]-cover.jpg
│   │   │   ├── like
│   │   │   └── ...

Key Operations

• Create User Directory: Creates a directory for a specific user nickname if it doesn't exist. Returns a Path object.
• Rename User Directory: Renames an existing user directory to a new nickname. If the directory does not exist, it will first create it before renaming.
• Create or Rename User Directory: A combined interface that prevents redundant directory creation when a user changes their nickname. This is integrated into the handler interface and is recommended for developers using the handler pattern.

F2 uses a layered configuration system to separate stable settings from frequently changed ones. This allows you to manage multiple profiles (e.g., different users or download modes) without re-entering sensitive data like cookies.

Configuration Layers

1. App low-frequency/main configuration file (app.yaml): Stores settings that rarely change, such as cookie, file name template, download path, and connection timeouts.
2. F2 configuration file (conf.yaml): Stores core F2 settings like computation parameters and app agents.
3. App default configuration file (defaults.yaml): The base template. Do not modify this file.
4. Custom Configuration File: User-created files for high-frequency changes (e.g., specific user URLs or download modes). These override settings in app.yaml.
5. Test Configuration File (test.yaml): Used specifically for running pytest test cases.

Configuration Priority

When a parameter is defined in multiple places, the priority is: CLI parameters > Custom configuration file > App low-frequency configuration file (app.yaml).