Open Deep Research

repository·main·Indexed 27 days ago

https://github.com/nickscamara/open-deep-research

An open-source implementation of deep research capabilities that combines Firecrawl for web searching and data extraction with reasoning models to perform complex research tasks. It supports various providers including OpenAI, TogetherAI, and OpenRouter, with specific support for reasoning models like o1, o3-mini, and DeepSeek-R1.

Tokens
3.8K
Snippets
9
Records
24
Agent score
91%

What's inside open-deep-research

  1. Run Open Deep Research locally

    main

    To run the project on your local machine, follow these steps in order:

    1. Install Vercel CLI: npm i -g vercel
    2. Link project: Run vercel link to link your local instance with your Vercel and GitHub accounts.
    3. Pull environment variables: Run vercel env pull to download your environment variables from Vercel.
    4. Install dependencies: Use pnpm install.
    5. Run migrations: Execute pnpm db:migrate to set up your database.
    6. Start development server: Run pnpm dev to start the app at http://localhost:3000/.
    npm i -g vercel
    vercel link
    vercel env pull
    pnpm install
    pnpm db:migrate
    pnpm dev
  2. Deploy to Vercel

    main
    You can deploy a version of this project to Vercel using the one-click deploy button. The deployment requires several environment variables including AUTH_SECRET, OPENAI_API_KEY, OPENROUTER_API_KEY, FIRECRAWL_API_KEY, BLOB_READ_WRITE_TOKEN, POSTGRES_URL, UPSTASH_REDIS_REST_URL, UPSTASH_REDIS_REST_TOKEN, REASONING_MODEL, BYPASS_JSON_VALIDATION, TOGETHER_API_KEY, and MAX_DURATION.
  3. Configure the Reasoning Model

    main

    The application uses a specific model for reasoning tasks (research analysis, structured outputs, etc.) via the REASONING_MODEL environment variable. If no model is specified, it defaults to o1-mini. If an invalid model is provided, it falls back to o1-mini.

    Supported Models

    • OpenAI: gpt-4o, o1, o3-mini (These have native JSON schema support).
    • TogetherAI: deepseek-ai/DeepSeek-R1 (Requires BYPASS_JSON_VALIDATION=true).

    Configuration via .env

    To use a non-OpenAI model like DeepSeek, you must also set BYPASS_JSON_VALIDATION=true to allow the model to function without native JSON schema support.

    # Example for TogetherAI DeepSeek
    REASONING_MODEL=deepseek-ai/DeepSeek-R1
    BYPASS_JSON_VALIDATION=true
  4. Configure environment variables for Docker deployment

    main

    When running the application via Docker Compose, the app service requires several environment variables to connect to the supporting infrastructure (PostgreSQL, Redis, and MinIO). Ensure your .env file or Docker environment includes the following keys:

    PostgreSQL

    • POSTGRES_USER: Database username (default: postgres)
    • POSTGRES_PASSWORD: Database password (default: postgres)
    • POSTGRES_DB: Database name (default: open_deep_research)
    • POSTGRES_URL: Connection string in the format postgresql://<user>:<password>@postgres:5432/<db>

    Redis (Upstash compatible)

    • UPSTASH_REDIS_REST_URL: The URL for the Redis instance (default: http://redis:6379)
    • UPSTASH_REDIS_REST_TOKEN: The token for Redis access (default: local_development_token)

    MinIO (Object Storage)

    • MINIO_ROOT_USER: Root user for MinIO (default: minioadmin)
    • MINIO_ROOT_PASSWORD: Root password for MinIO (default: minioadmin)
    • BLOB_READ_WRITE_TOKEN: Token for blob read/write access (default: minioadmin)

    Authentication

    • NEXTAUTH_URL: The public URL of the application (e.g., http://localhost:3000)
    • NEXTAUTH_URL_INTERNAL: The internal service URL used within the Docker network (e.g., http://app:3000)
    • NEXTAUTH_SECRET: A secret key for NextAuth (must be provided via ${AUTH_SECRET} in the environment)
  5. Configure NextAuth authentication settings

    main

    The authConfig object defines the core authentication behavior for the application using NextAuthConfig.

    Key configurations include:

    • pages.newUser: Specifies the redirect path for new users (set to /).
    • callbacks.authorized: A middleware-level callback that controls access to routes. It currently implements logic to redirect authenticated users away from /login and /register pages back to the root / to prevent redundant authentication attempts.
    export const authConfig = {
      pages: {
        newUser: '/',
      },
      providers: [
        // added later in auth.ts since it requires bcrypt which is only compatible with Node.js
        // while this file is also used in non-Node.js environments
      ],
      callbacks: {
        authorized({ auth, request: { nextUrl } }) {
          const isLoggedIn = !!auth?.user;
          const isOnRegister = nextUrl.pathname.startsWith('/register');
          const isOnLogin = nextUrl.pathname.startsWith('/login');
    
    // Redirect authenticated users away from auth pages
          if (isLoggedIn && (isOnLogin || isOnRegister)) {
            return Response.redirect(new URL('/', nextUrl as unknown as URL));
          }
    
    // Allow access to everything
          return true;
        },
      },
    } satisfies NextAuthConfig;
  6. Configure Drizzle ORM for PostgreSQL

    main

    The project uses Drizzle ORM with a PostgreSQL dialect. The configuration is managed via drizzle.config.ts and relies on environment variables defined in .env.local.

    To configure the database connection, ensure the POSTGRES_URL environment variable is set in your .env.local file. The schema is located at ./lib/db/schema.ts and migrations are output to ./lib/db/migrations.

  7. Configure available AI models

    main

    The AI engine uses a Model interface to define available models. You can extend or modify the models array to include different LLMs for standard tasks. Each model requires an id, label, apiIdentifier, and description.

    export interface Model {
      id: string;
      label: string;
      apiIdentifier: string;
      description: string;
    }
    
    export const models: Array<Model> = [
      {
        id: 'gpt-4o',
        label: 'GPT 4o',
        apiIdentifier: 'gpt-4o',
        description: 'For complex, multi-step tasks',
      },
      // ...
    ]
  8. Manage chats with saveChat, getChatsByUserId, and deleteChatById

    main

    The following functions allow for chat lifecycle management:

    • saveChat({ id, userId, title }): Creates a new chat record.
    • getChatsByUserId({ id }): Retrieves all chats for a specific user, ordered by creation date descending.
    • getChatById({ id }): Retrieves a single chat record by its ID.
    • deleteChatById({ id }): Deletes a chat and all associated votes and messages.
    • updateChatVisiblityById({ chatId, visibility }): Updates a chat's visibility to either 'private' or 'public'.
  9. Manage documents and suggestions

    main

    The system uses documents and suggestions for research data:

    • saveDocument({ id, title, kind, content, userId }): Saves a document. kind must be a valid BlockKind.
    • getDocumentsById({ id }): Retrieves all documents matching an ID, ordered by creation date ascending.
    • getDocumentById({ id }): Retrieves the most recent document for a given ID.
    • saveSuggestions({ suggestions }): Bulk inserts an array of Suggestion objects.
    • getSuggestionsByDocumentId({ documentId }): Retrieves all suggestions associated with a specific document ID.
    • deleteDocumentsByIdAfterTimestamp({ id, timestamp }): Deletes documents and their associated suggestions that were created after the provided timestamp.