GPT Researcher

repository·main·Indexed 12 days ago

https://github.com/assafelovic/gpt-researcher

An open-source autonomous deep research agent designed to produce detailed, factual, and unbiased research reports with citations. It utilizes a planner-and-execution architecture to perform web and local document research, featuring a multi-agent system with a Chief Editor and specialized Researcher subagents. Version 0.14.7 supports real-time progress tracking via an on_progress callback and provides tools for conducting research, retrieving sources, and generating formatted reports.

Tokens
113.1K
Snippets
309
Records
445
Agent score
97%

What's inside GPT Researcher

  1. Overview of GPT Researcher

    main

    GPT Researcher is an autonomous agent designed for comprehensive online research. It produces detailed, factual, and unbiased research reports by addressing common LLM limitations such as outdated training data, hallucination risks, and short token output limits.

    Key capabilities include:

    • Generating long-form research reports (2,000+ words).
    • Aggregating information from over 20 web sources per research task to ensure objectivity.
    • Scraping web sources with JavaScript support.
    • Exporting reports to formats like PDF and Word.
    • Providing a web interface for interaction.
  2. What is GPT Researcher?

    main

    GPT Researcher is an autonomous agent designed for comprehensive online research. It generates detailed, formal, and objective research reports by utilizing a 'Plan-and-Solve' and RAG-inspired architecture.

    Unlike standard LLM web plugins that may provide superficial answers, GPT Researcher uses parallelized agents to research multiple questions simultaneously, ensuring higher speed, stability, and reliability. It typically aggregates information from over 20 web resources per research task to minimize bias and hallucinations.

  3. Roadmap for Social Media Data Acquisition integration

    main

    The integration of social media data acquisition is planned across three phases:

    • Phase 1 (v4.1) - Basic Integration: Focuses on Apify integration, URL platform detection, and scrapers for LinkedIn and Twitter.
    • Phase 2 (v4.2) - Platform Expansion: Adds support for Facebook and Instagram, data format unification, and improved error handling.
    • Phase 3 (v4.3) - Enterprise Features: Introduces Bright Data integration, a provider router, cost monitoring, and cache optimization.
  4. Overview of the Claude Skills directory

    main

    The .claude/skills/ directory acts as a specialized knowledge base for AI assistants. It consists of two primary files:

    • SKILL.md: A comprehensive development guide containing architecture deep dives, core component method signatures (e.g., GPTResearcher, ResearchConductor), end-to-end data flows, prompt system details, retriever implementation guides, and the "8-Step Feature Pattern".
    • REFERENCE.md: A quick-lookup guide for technical specifications, including all environment variables, REST API endpoints, WebSocket message types, and Python client parameters.
  5. How multi-agent assistants work with LangGraph

    main

    GPT Researcher utilizes a multi-agent system built with LangGraph to improve the depth and quality of research. Inspired by the STORM paper, the system employs a team of AI agents that collaborate to handle the entire research lifecycle, from initial planning to final publication.

    An average execution generates a research report of approximately 5-6 pages, which can be exported in PDF, Docx, or Markdown formats.

    For implementation details, refer to the multi_agents directory in the repository or the official documentation.

  6. How the AG2 multi-agent research team works

    main

    This implementation uses AG2 to orchestrate a team of 8 specialized agents through a multi-stage workflow.

    The Agent Team

    • Human: Oversees the process and provides feedback.
    • Chief Editor: Manages the team and oversees the research process.
    • Researcher (gpt-researcher): Conducts in-depth research on the topic.
    • Editor: Plans the research outline and structure.
    • Reviewer: Validates correctness against specific criteria.
    • Revisor: Revises results based on reviewer feedback.
    • Writer: Compiles and writes the final report.
    • Publisher: Publishes the final report in various formats.

    Workflow Stages

    1. Planning stage
    2. Data collection and analysis
    3. Review and revision
    4. Writing and submission
    5. Publication
  7. Compare MCP Retriever Strategies

    main

    Choose your RETRIEVER configuration based on your research requirements:

    StrategyUse CasePerformanceCoverage
    mcpSpecialized domains, structured data⚡ Fast🎯 Focused
    tavily,mcpGeneral research with specialized tools⚖️ Balanced🌐 Comprehensive
    google,arxiv,tavily,mcpMaximum coverage, redundancy🐌 Slower🌍 Extensive
    arxiv,mcpAcademic + specialized research⚡ Fast🎓 Academic-focused
  8. How the Deep Agents research team works

    main

    The system uses a multi-agent architecture to automate in-depth research. The team consists of:

    • Chief Editor (Main Agent): Oversees the entire process. It plans the outline using the write_todos tool, delegates research tasks to subagents, reviews drafts, and assembles the final report.md.
    • Researcher (Subagent): Specialized agents that conduct in-depth research on specific sections. They operate with isolated contexts to prevent bloating the Chief Editor's context.
    • GPT Researcher (Core Engine): Exposed to the agents as two primary tools:
      • quick_search: Fast search returning ranked results, snippets, and URLs for scoping.
      • deep_research: The full pipeline (conduct_research + write_report) that returns a detailed markdown report with citations.

    Workflow Stages:

    1. Planning: Chief Editor runs a quick search and creates a plan.
    2. Data Collection: Researchers run deep_research on sections in parallel and write drafts to sections/*.md.
    3. Review/Revision: Chief Editor reviews drafts against guidelines and requests revisions if necessary.
    4. Writing/Publication: Chief Editor assembles the final report and references to report.md.
  9. Implement Human-in-the-Loop (HITL) in AG2 pipelines

    main

    To implement human review within an AG2 group chat pipeline, use the RevertToUserTarget() mechanism. This pauses the agent orchestration and waits for user input.

    Workflow Pattern

    1. Pause the Pipeline: A tool function (e.g., present_for_review) returns a ReplyResult with target=RevertToUserTarget().
    2. Handle Input: The backend listens for a response via an endpoint (e.g., /research/respond).
    3. Route Based on Feedback: Use LLMCondition on the human agent to decide where to go next based on the user's text:
      • If the user is unsatisfied $\rightarrow$ route back to the researcher_agent.
      • If the user approves $\rightarrow$ route forward to the writer_agent.

    Example Implementation

    # 1. Define the review tool that pauses the pipeline
    async def present_for_review(context_variables: ContextVariables) -> ReplyResult:
        # ... emit summary data to UI ...
        return ReplyResult(
            message="Research summary ready. Please review.",
            context_variables=context_variables,
            target=RevertToUserTarget(),
        )
    
    # 2. Add conditional handoffs for the human agent
    human_review_agent.handoffs.add_llm_condition(
        OnCondition(
            target=AgentTarget(researcher_agent),
            condition=StringLLMCondition("The user wants changes, more research, or is not satisfied."),
        )
    )
    human_review_agent.handoffs.add_llm_condition(
        OnCondition(
            target=AgentTarget(writer_agent),
            condition=StringLLMCondition("The user approves or wants to proceed with writing."),
        )
    )
    from autogen.agentchat import a_run_group_chat
    from autogen.agentchat.group import (
        AgentNameTarget, AgentTarget, ContextVariables, OnCondition,
        ReplyResult, RevertToUserTarget, StringLLMCondition,
    )
    
    # Example of a tool that triggers HITL
    async def present_for_review(context_variables: ContextVariables) -> ReplyResult:
        return ReplyResult(
            message="Research summary ready. Please review.",
            context_variables=context_variables,
            target=RevertToUserTarget(),
        )
    
    # Example of routing logic after human input
    human_review_agent.handoffs.add_llm_condition(
        OnCondition(
            target=AgentTarget(researcher_agent),
            condition=StringLLMCondition("The user wants changes."),
        )
    )
  10. Understand content quality differences between scrapers

    main

    Scrapers produce different levels of content quality, which directly impacts LLM performance:

    • Low Quality (e.g., BeautifulSoup): Output often contains significant noise, including navigation menus, headers, footers, and social sharing buttons (e.g., "Home About Contact Login Article Title Author | Date Share..."). This requires more tokens and can confuse the LLM.
    • High Quality (e.g., Tavily Extract / FireCrawl): Provides clean, well-formatted text that has been intelligently extracted to remove all navigation, ads, and irrelevant elements. This is highly LLM-friendly.
  11. Explore Multi-Agent Research with LangGraph and AG2

    main

    GPT Researcher supports multi-agent systems built with LangGraph and AG2. This approach uses specialized agents to conduct research from planning to publication, typically generating 5-6 page reports in PDF, Docx, or Markdown formats.

    Examples of multi-agent implementations can be found in the multi_agents directory of this repository.