Open Deep Research

repository·main·Indexed 12 days ago

https://github.com/dzhng/deep-research

An AI-powered research assistant that performs iterative, deep research by combining search engines, web scraping, and LLMs. It features a lightweight implementation (<500 LoC) capable of recursive exploration to refine research directions. Supports OpenAI (defaulting to o3-mini), DeepSeek R1 via Fireworks, and custom OpenAI-compatible endpoints. Integrates with Firecrawl for web search and extraction, providing tools for recursive research via the deepResearch() function and report generation via writeFinalReport().

Tokens
3.1K
Snippets
13
Records
15
Agent score
98%

What's inside Open Deep Research

  1. Install and Setup Open Deep Research via Node.js

    main

    To run the research assistant locally using Node.js, follow these steps:

    1. Clone the repository.
    2. Install dependencies using npm install.
    3. Create a .env.local file in the root directory and configure your API keys.

    Required environment variables:

    • FIRECRAWL_KEY: Your Firecrawl API key for web search and extraction.
    • OPENAI_KEY: Your OpenAI API key (defaults to using the o3-mini model).

    If you are using a self-hosted Firecrawl instance, add:

    • FIRECRAWL_BASE_URL: The URL of your instance (e.g., http://localhost:3002).
    npm install
    
    # Create .env.local with:
    FIRECRAWL_KEY="your_firecrawl_key"
    OPENAI_KEY="your_openai_key"
  2. Install and Setup Open Deep Research via Docker

    main

    To run the research assistant using Docker:

    1. Clone the repository.
    2. Rename .env.example to .env.local and populate it with your API keys.
    3. Build the image: docker build -f Dockerfile.
    4. Start the container: docker compose up -d.
    5. Execute the research command within the container using: docker exec -it deep-research npm run docker.
    docker build -f Dockerfile
    docker compose up -d
    docker exec -it deep-research npm run docker
  3. Use DeepSeek R1 via Fireworks

    main

    The system can automatically switch from o3-mini to DeepSeek R1 if a Fireworks API key is detected. Simply add the following to your environment variables:

    • FIREWORKS_KEY: Your Fireworks API key.
    FIREWORKS_KEY="api_key"
  4. Configure Local LLMs or Custom OpenAI-Compatible Endpoints

    main

    You can bypass OpenAI by using a local LLM server or other OpenAI-compatible providers (like OpenRouter or Gemini) by configuring the following environment variables in .env.local:

    • OPENAI_ENDPOINT: The address of your local server or custom provider (e.g., http://localhost:1234/v1).
    • OPENAI_MODEL or CUSTOM_MODEL: The name of the model to be used. (Note: The README mentions both OPENAI_MODEL and CUSTOM_MODEL for tweaking the model string).

    To use a local LLM, comment out OPENAI_KEY and set these values.

    # Example for local LLM
    # OPENAI_KEY=""
    OPENAI_ENDPOINT="http://localhost:1234/v1"
    OPENAI_MODEL="model-name-here"
  5. Configure Concurrency Limits

    main

    You can control the speed of research by adjusting the CONCURRENCY_LIMIT environment variable:

    • High Performance: If using a paid Firecrawl account or a local version, increase CONCURRENCY_LIMIT to run searches and processing in parallel.
    • Rate Limit Avoidance: If using a free version of Firecrawl, set CONCURRENCY_LIMIT=1 to avoid rate limit errors.
    # To run faster
    CONCURRENCY_LIMIT=10
    
    # To avoid rate limits
    CONCURRENCY_LIMIT=1
  6. Run Deep Research using Docker Compose

    main

    You can deploy the Deep Research service using Docker Compose. The configuration uses a local .env.local file for environment variables and mounts the current directory to /app/ inside the container to allow for live development or persistent access to local files.

    To run the service, ensure you have a docker-compose.yml file in your project root and execute:

    docker-compose up
    services:
      deep-research:
        container_name: deep-research
        build: .
        env_file:
          - .env.local
        volumes:
          -  ./:/app/
        tty: true
        stdin_open: true
  7. Configure AI models via environment variables

    main

    The project selects models based on the presence of specific environment variables. The selection priority is:

    1. Custom Model: If CUSTOM_MODEL is set, it uses the OpenAI provider with that model name and structuredOutputs: true enabled.
    2. DeepSeek R1: If FIREWORKS_KEY is set, it uses the deepseek-r1 model via Fireworks, wrapped with reasoning middleware to handle <think> tags.
    3. o3-mini: If no custom model or Fireworks key is provided, it defaults to OpenAI's o3-mini with reasoningEffort: 'medium' and structuredOutputs: true.

    Required Environment Variables:

    • OPENAI_KEY: Required for OpenAI models (including o3-mini and CUSTOM_MODEL).
    • OPENAI_ENDPOINT: (Optional) Overrides the default OpenAI API URL. Defaults to https://api.openai.com/v1.
    • FIREWORKS_KEY: Required to use the DeepSeek R1 model via Fireworks.
    • CUSTOM_MODEL: (Optional) Specifies a specific model name to use via the OpenAI provider.
    # Example for using a custom model
    export OPENAI_KEY="sk-..."
    export CUSTOM_MODEL="gpt-4o"
    
    # Example for using DeepSeek R1 via Fireworks
    export FIREWORKS_KEY="..."
  8. Configure Firecrawl via environment variables

    main

    The agent uses Firecrawl for web searching and scraping. You can configure its behavior and connection using the following environment variables:

    • FIRECRAWL_KEY: Your Firecrawl API key.
    • FIRECRAWL_BASE_URL: The base URL for your Firecrawl instance (useful for self-hosted setups, e.g., http://localhost:3002).
    • FIRECRAWL_CONCURRENCY: Controls the number of parallel requests made to Firecrawl. Defaults to 2. Increase this if your API rate limits allow.
  9. Run the Research Assistant

    main

    Start the interactive research process by running npm start. The CLI will prompt you for:

    1. Research Query: The topic you want to investigate.
    2. Research Breadth: How many search results to process (recommended: 3-10, default: 4).
    3. Research Depth: How many recursive iterations to perform (recommended: 1-5, default: 2).
    4. Follow-up Questions: Answers to refine the research direction.

    Upon completion, the system generates a markdown report saved as report.md or answer.md in your working directory.

    npm start
  10. Generate a concise answer with writeFinalAnswer()

    main

    Use writeFinalAnswer when you need a short, direct answer to a specific question based on research, rather than a long report. This is ideal for tasks like answering multiple-choice questions or extracting a single specific metric.

    Parameters

    • prompt: The specific question or format instructions (e.g., "Answer in Latex").
    • learnings: The research findings to use for the answer.
    import { writeFinalAnswer } from './deep-research';
    
    const answer = await writeFinalAnswer({
      prompt: "What is the melting point of Gold? Answer in Celsius.",
      learnings: ["Gold melts at 1064 degrees Celsius."]
    });
    
    console.log(answer); // "1064°C"
  11. Perform deep research with deepResearch()

    main

    The deepResearch function is the primary entrypoint for conducting recursive, multi-step research on a given topic. It uses a search engine (via Firecrawl) to find information, extracts learnings, and then recursively performs follow-up searches based on those learnings until the specified depth is reached.

    Parameters

    • query: The initial research topic or question.
    • breadth: The number of parallel search queries to perform at each level of depth.
    • depth: How many levels of recursive research to perform.
    • learnings (optional): An array of existing learnings to seed the research (useful for resuming or refining).
    • visitedUrls (optional): An array of URLs already visited.
    • onProgress (optional): A callback function that receives a ResearchProgress object whenever the state changes.

    Returns

    A Promise resolving to a ResearchResult object containing:

    • learnings: A unique list of all information points gathered.
    • visitedUrls: A unique list of all URLs visited during the process.
    import { deepResearch } from './deep-research';
    
    const result = await deepResearch({
      query: "What are the latest advancements in solid-state batteries?",
      breadth: 3,
      depth: 2,
      onProgress: (progress) => {
        console.log(`Progress: ${progress.completedQueries}/${progress.totalQueries} queries done.`);
      }
    });
    
    console.log(result.learnings);
  12. Retrieve the active LanguageModelV1 using getModel()

    main

    The getModel() function returns the currently configured LanguageModelV1 instance based on the environment variables provided. If no valid configuration is found (i.e., no keys are set), it throws an error: No model found.

    This is the primary way to access the model instance used by the application.

    import { getModel } from './ai/providers';
    
    const model = getModel();
    // model is a LanguageModelV1 instance ready for use with AI SDK