Firesearch Documentation

When initial search results are insufficient to meet the MIN_ANSWER_CONFIDENCE threshold, Firesearch automatically employs several strategies to find missing information:

• Broaden Keywords: Removes specific terms to get wider results.
• Narrow Focus: Adds specific terms to target missing aspects.
• Synonyms: Uses alternative terms and phrases.
• Rephrase: Completely reformulates the query.
• Decompose: Breaks complex queries into sub-questions.
• Academic: Adds scholarly terms for research-oriented results.
• Practical: Focuses on tutorials and how-to guides.

To run Firesearch locally, follow these steps:

1. Clone the repository.
2. Configure Environment Variables: Create a .env.local file in the root directory and add your API keys:

   FIRECRAWL_API_KEY=your_firecrawl_key
   OPENAI_API_KEY=your_openai_key

3. Install Dependencies: Run npm install or yarn install.
4. Start Development Server: Run npm run dev or yarn dev.

Required API Keys

After an answer is provided, the engine generates exactly 3 relevant follow-up questions to encourage deeper exploration.

Follow-up questions are designed to:

• Explore different aspects of the topic.
• Dig deeper into technical terms or details.
• Be natural, conversational, and actionable.
• Stay under 80 characters.
• Consider the full conversation context to ensure they build upon the current topic.

The search process moves through a specific sequence of phases. Understanding these helps in managing UI states or debugging where a search might be stalling:

1. understanding: Analyzing the user's intent.
2. planning: Breaking the query into sub-queries and search strategies.
3. searching: Executing web searches via Firecrawl.
4. scraping: Deep-diving into specific URLs to extract content.
5. analyzing: Evaluating gathered information against sub-queries to check for completeness.
6. synthesizing: Generating the final comprehensive answer and follow-up questions.
7. complete: Finalizing the process.
8. error: A terminal state if the process fails after retries.

type SearchPhase = 
  | 'understanding'
  | 'planning' 
  | 'searching'
  | 'analyzing'
  | 'synthesizing'
  | 'complete'
  | 'error';

The search engine breaks down complex user queries into multiple factual sub-queries. Each sub-query consists of a question and a searchQuery. After search results are retrieved, the engine updates these sub-queries by matching them against results.

Answering a sub-query is determined by a confidence threshold defined in SEARCH_CONFIG.MIN_ANSWER_CONFIDENCE. If a result's confidence meets this threshold, the sub-query is marked as answered: true, the answer is populated, and sources are aggregated.

The engine supports streaming answers to provide a better user experience. It uses a streamingLlm to generate a response that includes:

• Citations: Uses markdown-style citations like [1], [2] to link findings to specific sources.
• Context Awareness: If a conversation history is provided, the engine incorporates previous queries and responses to maintain continuity.
• Streaming: The onChunk callback is used to emit text fragments as they are generated by the LLM.

When a sub-query remains unanswered or fails to meet the MIN_ANSWER_CONFIDENCE threshold, the engine can generate alternative search queries to broaden the search.

Strategies used for generating alternatives include:

1. Using broader or more general terms.
2. Trying different phrasings or synonyms.
3. Removing specific qualifiers (like years or versions) that might be too restrictive.
4. Searching for related concepts.
5. Searching for the base product or company name if a specific version search fails.

If a query contains a specific version pattern (e.g., v1.2, 2024.05) and fails multiple times (specifically after 2 attempts), the engine prioritizes stripping the version to find the base product.

To keep the context window manageable and focused, the engine summarizes content retrieved from search results. The summarization process is constrained by:

• Relevance: It only extracts findings specifically relevant to the original search query.
• Length: Summaries are kept under the limit defined by SEARCH_CONFIG.SUMMARY_CHAR_LIMIT.
• Format: It returns a single sentence containing specific details like numbers or dates.

To enable persistence and the ability to resume searches using a checkpointId, you must pass enableCheckpointing: true in the constructor options. This initializes a MemorySaver internally. When calling .search(), providing a checkpointId will allow the engine to use a specific thread_id for state management.

// Enable checkpointing during initialization
const engine = new LangGraphSearchEngine(firecrawl, { enableCheckpointing: true });

// Use a checkpointId to resume or track a specific thread
await engine.search("my query", onEvent, undefined, "my-unique-session-id");

Firesearch uses the Firecrawl /search endpoint to find relevant URLs and extract markdown content in a single call. This is achieved by passing scrapeOptions in the request body.

API Usage

• Endpoint: POST /search
• Purpose: Finds relevant URLs AND extracts markdown content.
• Key Feature: The scrapeOptions parameter enables content extraction during the search process.
• Response: Returns URLs with titles, snippets, and full markdown content.

Example Request

POST /search
{
  "query": "iPhone 16 specs pricing",
  "limit": 8,
  "scrapeOptions": {
    "formats": ["markdown"]
  }
}