DOCSEARCHbyAlgolia
Submit Docs

BettaFish (微舆)

repository·main·Indexed 12 days ago

https://github.com/666ghj/bettafish

A multi-agent system for automated public opinion analysis and data collection across social media platforms. It includes MindSpider, an AI-powered crawler that uses a two-stage workflow—BroadTopicExtraction for identifying hot topics via LLMs and DeepSentimentCrawling for scraping detailed content using Playwright from platforms such as Weibo, Xiaohongshu, Douyin, and Bilibili.

Tokens
29.9K
Snippets
87
Records
146
Agent score
93%

What's inside BettaFish

  1. Overview of BettaFish

    main

    BettaFish is a multi-agent public opinion analysis system designed to break information cocoons and restore original public sentiment. It automates the process of analyzing 30+ mainstream social platforms and millions of user comments to predict future trends and assist in decision-making.

    Key capabilities include:

    • AI-Driven Monitoring: 24/7 crawler clusters covering platforms like Weibo, Xiaohongshu, TikTok, and Kuaishou.
    • Composite Analysis Engine: Uses a combination of 5 specialized Agent types, fine-tuned models, and statistical models.
    • Multimodal Analysis: Capable of analyzing text, images, and short video content (e.g., TikTok, Kuaishou).
    • Agent 'Forum' Collaboration: Employs a debate moderator model where agents use different toolsets and thinking patterns to engage in chain-of-thought collision.
    • Data Integration: Supports merging internal business databases with public opinion data.
    • Extensible Framework: A lightweight, pure Python modular design that allows for easy integration of custom models and business logic.

  2. Overview of BettaFish (微舆)

    main

    BettaFish (微舆) is an innovative multi-agent public opinion analysis system designed to break information cocoons and provide data-driven decision support. It automates the collection and analysis of data from over 30 mainstream domestic and international social media platforms, processing millions of user comments.

    Users interact with the system via natural language, posing analysis requests as if chatting. The system then triggers a chain of specialized agents to perform deep analysis, resulting in comprehensive research reports.

    Key Capabilities:

    • AI-Driven Global Monitoring: 24/7 AI crawler clusters covering platforms like Weibo, Xiaohongshu, Douyin, and Kuaishou.
    • Composite Analysis Engine: Combines specialized Agents with fine-tuned models and statistical middleware for deep, multi-dimensional insights.
    • Multimodal Processing: Analyzes text, images, and short videos (e.g., Douyin, Kuaishou) and extracts structured information like weather, calendars, and stock data.
    • Agent 'Forum' Collaboration: Uses a unique mechanism where agents with different toolsets and thinking patterns engage in chain-of-thought debates, moderated by a 'host' model to prevent cognitive bias.
    • Hybrid Data Integration: Supports merging public social media trends with private internal business databases via secure interfaces.
    • Extensible Framework: A lightweight, modular Python-based design that allows developers to easily integrate custom models or business logic.

  3. Overview of Third-party JavaScript libraries in ReportEngine

    main
    The ReportEngine/renderers/libs directory contains third-party JavaScript libraries required for HTML report rendering. These libraries are automatically inlined into the generated HTML files by the html_renderer.py component. This ensures that reports are fully functional in offline environments, load quickly without external CDN dependencies, and maintain consistent behavior through fixed versions.

  4. Project Directory Structure Overview

    main

    The repository is organized into specialized engines and supporting modules:

    • QueryEngine/: News and web search logic.
    • MediaEngine/: Multimodal (video/image) understanding.
    • InsightEngine/: Private database mining and sentiment analysis.
    • ReportEngine/: The core orchestration engine for template selection, layout design, and rendering (HTML/PDF).
    • ForumEngine/: The collaboration mechanism for agent communication.
    • MindSpider/: Social media crawling system for topic extraction and sentiment crawling.
    • SentimentAnalysisModel/: Collection of fine-tuned models (BERT, GPT-2, Qwen) for sentiment tasks.
    • SingleEngineApp/: Streamlit applications for testing individual engines independently.
    • app.py: The main Flask application entry point.

  5. Project Code Structure Overview

    main

    The repository is organized into specialized engines and supporting modules:

    • QueryEngine/: Logic for news breadth search and international/domestic toolkits.
    • MediaEngine/: Multimodal understanding and search toolkits.
    • InsightEngine/: Private database mining, including sentiment_analyzer.py and search.py tools.
    • ReportEngine/: The orchestration engine for report generation. Key sub-modules include:
      • core/: Template parsing and document stitching.
      • ir/: The Intermediate Representation (IR) schema and validation.
      • nodes/: Workflow reasoning (template selection, layout, budget, chapter generation).
      • renderers/: HTML and PDF (via WeasyPrint) output engines.
    • ForumEngine/: The collaboration mechanism (monitor.py and llm_host.py).
    • MindSpider/: The social media crawler system used for data acquisition.
    • SentimentAnalysisModel/: A collection of fine-tuned models (BERT, GPT-2, Qwen) for sentiment tasks.
    • SingleEngineApp/: Standalone Streamlit applications for testing individual engines.

  6. Project File Structure

    main

    The repository is organized as follows:

    • adapter.py: Implementation of the Adapter layers.
    • gpt2_adapter.py: Adapter implementation specifically for GPT2 models.
    • train.py: Training script.
    • predict.py: Interactive prediction script.
    • models/gpt2-chinese/: Local storage for the pre-trained Chinese GPT2 model and configurations.
    • dataset/weibo_senti_100k.csv: The Weibo sentiment dataset (labels: 0 for negative, 1 for positive).
    • best_weibo_sentiment_model.pth: The saved weights of the best trained model.

  7. Extend BettaFish for specific domains

    main
    BettaFish is designed as a general-purpose data analysis engine. You can repurpose it for different industries (e.g., transforming it into a financial market analysis system) by modifying the Agent toolset API parameters and the system prompts.

  8. How library loading and fallback works

    main

    The html_renderer.py follows a specific loading logic:

    1. Primary Method: It attempts to load the library files directly from the ReportEngine/renderers/libs directory and inline them into the HTML.
    2. Fallback Mechanism: If a library file fails to load (e.g., the file is missing or there is a read error), the renderer automatically falls back to using the corresponding CDN links. This ensures the report remains functional even if local files are corrupted or missing.

  10. Transform BettaFish into other analysis systems

    main
    BettaFish is designed as a universal data analysis engine. While focused on public opinion, it can be repurposed for other domains (e.g., financial market analysis) by modifying the API parameters and prompts of the Agent toolset.

  11. Understand the BettaFish System Architecture

    main

    BettaFish is a multi-agent system designed for complex information analysis and automated report generation. The architecture is composed of four specialized AI agents that work in parallel and then collaborate through a forum-based mechanism:

    • Query Agent (QueryEngine): Performs precise information searches across domestic and international web sources.
    • Media Agent (MediaEngine): Provides multimodal content analysis (images, video, etc.).
    • Insight Agent (InsightEngine): Conducts in-depth mining of private public opinion databases.
    • Report Agent (ReportEngine): Orchestrates the final report generation process, transforming analysis results into structured, interactive HTML or PDF reports.

    These agents interact via a ForumEngine, which acts as a collaboration layer where agents discuss findings and receive guidance from an LLM-based host.

  12. Model storage and hardware acceleration

    main

    Model Storage

    • On the first run, the model is automatically downloaded to a local model folder in the current directory.
    • Subsequent runs load the model from this local folder without needing a network connection.
    • The model size is approximately 400MB.
    • To remove the model files, simply delete the model folder.

    Hardware Acceleration

    • The implementation supports GPU acceleration and will automatically detect available devices.