DeepSearcher Documentation

DeepSearcher can be integrated into your workflow using one of three primary methods:

1. Python API: Best for programmatic integration and custom logic.
2. Command Line Interface (CLI): Best for quick tasks and terminal-based workflows.
3. Web Service Deployment: Best for serving DeepSearcher as a scalable service.

Refer to the specific guides for quick_start.md, cli.md, or deployment.md to begin.

DeepSearcher provides the following capabilities:

• Private Data Search: Utilizes enterprise internal data securely, with optional integration of online content.
• Vector Database Management: Supports Milvus and other vector databases with data partitioning for efficient retrieval.
• Flexible Embedding Options: Compatible with multiple embedding models.
• Multiple LLM Support: Supports various models including DeepSeek and OpenAI for Q&A and content generation.
• Document Loader: Supports loading local files and (under development) web crawling.

DeepSearcher allows customization of several core components through its configuration system. You can configure the following components:

• LLM: Large Language Models used for query processing.
• Embedding Models: Models used for generating text embeddings for vector retrieval.
• Vector Database: The storage and retrieval system for vector embeddings.
• File Loader: Handles loading and processing of various file formats.
• Web Crawler: Gathers information from web sources.

Run the built deepsearcher:latest image as a container. This command maps port 8000, sets the OPENAI_API_KEY environment variable, and mounts local directories for data, logs, and the config.yaml file to ensure persistence and configuration application.

docker run -p 8000:8000 \
  -e OPENAI_API_KEY=your_openai_api_key \
  -v $(pwd)/data:/app/data \
  -v $(pwd)/logs:/app/logs \
  -v $(pwd)/deepsearcher/config.yaml:/app/deepsearcher/config.yaml \
  deepsearcher:latest

To run a basic DeepSearcher workflow, configure your LLM and Embedding providers, load data (local files or websites), and then execute a query. Ensure you have the necessary API keys (e.g., OPENAI_API_KEY) set in your environment variables.

Note: If using web crawling, the FIRECRAWL_API_KEY environment variable is required.

To use a local LLM via Ollama, set the provider to Ollama.

1. Install and run Ollama locally.
2. Pull the desired model (e.g., ollama pull qwq).
3. Ensure the Ollama REST API is running (default: http://localhost:11434).

config.set_provider_config("llm", "Ollama", {"model": "qwq"})

To set up the local documentation environment using MkDocs, install the necessary dependencies and clone the repository.

1. Install MkDocs and required plugins:

pip install mkdocs mkdocs-material mkdocs-jupyter pymdown-extensions

2. Clone the repository:

git clone https://github.com/zilliztech/deep-searcher.git
cd deep-searcher

To use the managed Zilliz Cloud service, provide your specific instance endpoint as the uri and your API key as the token.

config.set_provider_config("vector_db", "Milvus", {
    "uri": "https://your-instance-id.api.gcp-us-west1.zillizcloud.com", 
    "token": "your_api_key"
})

DeepSearcher uses embedding models to convert text into vector representations for semantic search. You can configure the embedding provider using the config.set_provider_config method, specifying the provider name and a dictionary of arguments (such as the model name).

config.set_provider_config("embedding", "(EmbeddingModelName)", "(Arguments dict)")

Use the evaluate.py script to compare the performance of DeepSearcher against naive RAG using the Recall@K metric. The evaluation currently supports the 2wikimultihopqa dataset.

You can provide a configuration YAML file to specify your LLM, embedding model, and other provider parameters. To avoid re-loading the dataset into the vector database on subsequent runs, use the --skip_load flag.

python evaluate.py \
--dataset 2wikimultihopqa \
--config_yaml ./eval_config.yaml \
--pre_num 5 \
--output_dir ./eval_output

After installation, you can verify that the package is correctly set up by initializing a Configuration object and importing the query function from deepsearcher.online_query.

from deepsearcher.configuration import Configuration
from deepsearcher.online_query import query

# Initialize with default configuration
config = Configuration()
print("DeepSearcher installed successfully!")