Fire Enrich Documentation

repository·main·Indexed 22 days ago

https://github.com/firecrawl/fire-enrich

An AI-powered data enrichment tool that converts simple inputs, such as email addresses, into comprehensive datasets including company profiles, funding details, and industry information. It utilizes a multi-agent architecture featuring specialized agents for company research, fundraising intelligence, executive research, product and technology, and contact information, powered by Firecrawl and OpenAI GPT-4.

Tokens
9.3K
Snippets
21
Records
40
Agent score
78%

What's inside Fire Enrich

  1. How Fire Enrich agents and orchestration work

    main

    Fire Enrich uses a sequential multi-agent orchestration system to enrich data. Instead of running all searches at once, it executes specialized agents in phases so that each phase can use the context discovered by the previous one.

    The Agent Sequence

    1. Discovery Agent: Establishes the foundation (company name, website, domain).
    2. Company Profile Agent: Uses the verified name to find industry, sub-category, and business model.
    3. Financial Intel Agent: Uses company and industry context to search for funding stages, total raised, and investors.
    4. Tech Stack Agent: Analyzes HTML, GitHub, and documentation to identify programming languages and infrastructure.
    5. General Purpose Agent: Handles custom fields (e.g., CEO, competitors) using the full context gathered in previous steps.

    Key Architectural Principles

    • Context Building: Each agent adds data that makes subsequent searches more accurate (e.g., industry knowledge helps the Financial agent find the right databases).
    • Parallelism within Phases: While agents run sequentially, each agent performs multiple searches in parallel via the Firecrawl API to maximize speed.
    • AI Synthesis: A final GPT-4o pass combines all findings, resolves conflicts, and validates the structured data.
  2. How Fire Enrich's Multi-Agent System works

    main

    Fire Enrich uses a specialized multi-agent architecture to perform data enrichment. Instead of a single generic prompt, it employs five specialized AI agents, each optimized for specific domains:

    • Company Research Agent: Extracts company fundamentals (name, description, industry, employee count).
    • Fundraising Intelligence Agent: Discovers funding rounds, investors, and valuation data.
    • Executive Research Agent: Identifies leadership teams, founders, and key personnel.
    • Product & Technology Agent: Uncovers product offerings, tech stack, and competitive landscape.
    • Contact Information Agent: Finds emails, phone numbers, and social media profiles.

    Agent Mode vs. Traditional Mode

    When configuring your enrichment, you can choose between two modes:

    • Agent Mode (Recommended): Uses the specialized agents listed above. This provides higher accuracy and better quality extraction for specific fields, though it is slightly slower.
    • Traditional Mode: Uses direct GPT-4 extraction. This is faster and uses fewer tokens, making it suitable for simple, non-specialized fields.
  3. Install and Setup Fire Enrich

    main

    To run Fire Enrich locally, follow these steps:

    1. Clone the repository:
      git clone <repository-url>
      cd hostedTools
    2. Install dependencies using pnpm:
      pnpm install
    3. Configure environment variables by creating a .env.local file in the root directory with your API keys:
      FIRECRAWL_API_KEY=your_firecrawl_key
      OPENAI_API_KEY=your_openai_key
    4. Start the development server:
      pnpm dev
    5. Access the application at http://localhost:3000/fire-enrich.

    Prerequisites:

    git clone <repository-url>
    cd hostedTools
    pnpm install
    # Create .env.local
    FIRECRAWL_API_KEY=your_firecrawl_key
    OPENAI_API_KEY=your_openai_key
    pnpm dev
  4. Configure enrichment fields and custom queries

    main

    Fire Enrich allows you to select preset fields or define custom fields using natural language.

    Preset Fields

    You can choose from a variety of standard business intelligence fields, including:

    • Company Name
    • Industry & Description
    • Employee Count
    • Revenue
    • Headquarters Location
    • Social Media Profiles
    • Leadership Team

    Custom Fields

    You can use natural language to instruct the AI to find specific data points. The AI interprets your requirements to generate the necessary extraction logic.

    Examples of effective custom fields:

    • "Find the CEO's email and LinkedIn"
    • "Get their main product pricing"
    • "Find recent news mentions"

    Tips for High-Quality Extraction

    1. Be Specific: Use "CEO name and email" instead of "contact info".
    2. Separate Concerns: Request one data type per field.
    3. Use Examples: Provide format hints like "Revenue (e.g., $10M ARR)".
    4. Leverage Context: Mention your specific use case to help the AI understand the intent.
  5. Configure API Keys for Fire Enrich

    main

    Fire Enrich requires two primary API keys to function:

    1. Firecrawl API Key: Used for web scraping and search capabilities.
    2. OpenAI API Key: Used for intelligent data extraction via GPT-4.

    Configuration Methods

    1. Environment Variables (Server-side)

    Create a .env.local file in your project root and add:

    FIRECRAWL_API_KEY=your_firecrawl_key
    OPENAI_API_KEY=your_openai_key

    2. Browser-based Entry (Client-side)

    If you do not want to use environment variables, you can enter keys directly in the UI:

    1. Visit the Fire Enrich page.
    2. Click "Enter API Keys" when prompted.
    3. Keys are stored securely in your browser's localStorage and are not stored on the server.
  6. Extend Fire Enrich with new data extraction capabilities

    main

    Fire Enrich is extensible through type-safe Zod schemas. You can add new data points by modifying existing agents or creating entirely new ones.

    1. Add a field to an existing agent

    Modify the Zod schema in the corresponding agent file (e.g., /lib/agent-architecture/agents/[agent-name].ts).

    // Example: Adding a new field to the FundingAgent
    const FundingResult = z.object({
      fundingStage: z.string().optional(),
      totalRaised: z.string().optional(),
      lastRoundAmount: z.string().optional(),
      investors: z.array(z.string()).optional(),
      // Add your new field here:
      debtFinancing: z.string().optional(),
    });

    2. Create a new agent

    • Define a new Zod schema.
    • Implement the AgentBase interface.
    • Update the orchestrator's routing logic to direct relevant fields to your new agent.

    Field Routing Logic

    The system automatically routes fields to agents based on keywords:

    • industry or headquarter $\rightarrow$ Company Profile Agent
    • fund or invest $\rightarrow$ Financial Intel Agent
    • employee or revenue $\rightarrow$ Metrics Agent
    • tech and stack $\rightarrow$ Tech Stack Agent
    • Everything else $\rightarrow$ General Purpose Agent
  7. Setup Fire Enrich

    main

    To set up Fire Enrich locally, follow these steps:

    1. Clone the repository to your local machine.
    2. Configure environment variables: Create a .env.local file in the root directory and add your API keys.
    3. Install dependencies: Run npm install or yarn install.
    4. Start the development server: Run npm run dev or yarn dev.
    5. Access the application: Open http://localhost:3000 in your browser.
    # Create .env.local with these keys
    FIRECRAWL_API_KEY=your_firecrawl_key
    OPENAI_API_KEY=your_openai_key
    
    # Install and run
    npm install
    npm run dev
  8. Configure Unlimited Mode and limits

    main

    When running the repository locally, Fire Enrich enables Unlimited Mode by default. You can control the constraints for CSV rows, columns, and enrichment fields in app/fire-enrich/config.ts.

    Use the FIRE_ENRICH_UNLIMITED environment variable or set NODE_ENV to development to bypass limits.

    const isUnlimitedMode = process.env.FIRE_ENRICH_UNLIMITED === 'true' || 
                           process.env.NODE_ENV === 'development';
    
    export const FIRE_ENRICH_CONFIG = {
      CSV_LIMITS: {
        MAX_ROWS: isUnlimitedMode ? Infinity : 15,
        MAX_COLUMNS: isUnlimitedMode ? Infinity : 5,
      },
      REQUEST_LIMITS: {
        MAX_FIELDS_PER_ENRICHMENT: isUnlimitedMode ? 50 : 10,
      },
    } as const;
  9. How the Enrichment Coordinator works

    main

    The createEnrichmentCoordinator function creates a master Agent that manages multiple specialized agents. It uses a handoff pattern to delegate research tasks based on the requested EnrichmentFields.

    Workflow:

    1. Analysis: It parses the requested fields and their descriptions.
    2. Delegation: It selects the appropriate specialized agents (Company, Fundraising, People, Product, or Contact) based on keywords in the field names and descriptions.
    3. Execution: It uses the selected agents to gather data.
    4. Consolidation: It compiles results from all delegated agents.
    5. Mapping: It maps the agent outputs to the exact field names requested by the user, using a dynamically generated Zod schema.
  10. Troubleshoot Fire Enrich issues

    main

    Common Issues and Solutions

    IssueSolution
    "No API Keys Found"Check your .env.local file or use the browser-based "Enter API Keys" option. Verify your keys are valid.
    Slow EnrichmentNormal processing takes ~5-15 seconds per row. If it is excessively slow, check your API rate limits or switch to Traditional Mode.
    Missing DataSome companies have limited online presence. Check the Confidence Scores and review the provided Source URLs to verify if data exists.
    Export IssuesEnsure the enrichment process is fully complete. Check the browser console for errors and try switching between CSV and JSON formats.

    Performance & Limits

    • Default Processing Rate: 1 row per second (to protect APIs).
    • Max Fields: 10 fields per enrichment task.
    • Rate Limits: Subject to your Firecrawl and OpenAI plan limits.
  11. Example of Data Enrichment Output

    main

    Fire Enrich transforms simple input (like an email address) into a rich JSON dataset containing company profiles, funding data, and tech stack information.

    Input Example:

    {
      "email": "erez@wiz.io"
    }

    Output Example:

    {
      "email": "erez@wiz.io",
      "companyName": "Wiz",
      "industry": "Cybersecurity",
      "employeeCount": "1001-5000",
      "yearFounded": 2020,
      "headquarters": "New York, NY",
      "fundingStage": "Series D",
      "totalRaised": "$900M",
      "website": "https://www.wiz.io",
      "sources": [
        "https://www.wiz.io/about",
        "https://techcrunch.com/2023/02/27/wiz-confirms-300m-at-a-10b-valuation-to-build-out-its-cloud-security-platform/"
      ]
    }
    {
      "email": "erez@wiz.io",
      "companyName": "Wiz",
      "industry": "Cybersecurity",
      "employeeCount": "1001-5000",
      "yearFounded": 2020,
      "headquarters": "New York, NY",
      "fundingStage": "Series D",
      "totalRaised": "$900M",
      "website": "https://www.wiz.io",
      "sources": [
        "https://www.wiz.io/about",
        "https://techcrunch.com/2023/02/27/wiz-confirms-300m-at-a-10b-valuation-to-build-out-its-cloud-security-platform/"
      ]
    }
  12. Configure enrichment concurrency and batching

    main

    The ENRICHMENT_CONFIG object controls the throughput and rate-limiting behavior of the enrichment process. You can tune these settings to balance processing speed against API rate limits.

    • CONCURRENT_ROWS: The number of rows to process simultaneously. Higher values increase speed but also increase API consumption and the risk of rate limiting. A recommended value for most use cases is 2-5, though the default is 10.
    • BATCH_DELAY_MS: The delay in milliseconds between processing batches. Increasing this value helps prevent hitting API rate limits.
    export const ENRICHMENT_CONFIG = {
      CONCURRENT_ROWS: 10,
      BATCH_DELAY_MS: 1000,
    } as const;