DOCSEARCHbyAlgolia
Submit Docs

AgentSearch Documentation

repository·main·Indexed 19 days ago

https://github.com/sciphi-ai/agent-search

An open source framework and dataset for webscale local search. AgentSearch integrates LLM providers with search engines to power search agents and complex RAG (Retrieval-Augmented Generation) tasks, such as summarizing search results and generating follow-up queries. It features a multi-stage search pipeline including vectorization, URL deduplication, hierarchical reranking, and PageRank reranking.

Tokens
7.1K
Snippets
30
Records
35
Agent score
67%

What's inside AgentSearch

  1. View the built documentation locally

    main

    After building the documentation, you can serve the generated HTML files using a local Python HTTP server. This allows you to view the documentation in your web browser.

    ```bash
python -m http.server -d build/html/

    Once the server is running, open your browser and navigate to http://localhost:8000.

  3. Local Setup and Initialization

    main

    To run AgentSearch locally with a full stack (Postgres and Qdrant), follow these steps from the project root:

    1. Launch Postgres Database sudo service postgresql start

    2. Populate Postgres Database python -m agent_search.scripts.populate_postgres_from_hf run

    3. Start Qdrant Service with Docker docker run -p 6333:6333 -p 6334:6334 -v $(pwd)/qdrant_storage:/qdrant/storage:z qdrant/qdrant

    4. Populate Vector Database (Qdrant) python -m agent_search.scripts.populate_qdrant_from_postgres run --delete_existing=True

    5. Run the Server python -m agent_search.app.server

    # 1. Start Postgres
sudo service postgresql start

# 2. Populate Postgres
python -m agent_search.scripts.populate_postgres_from_hf run

# 3. Start Qdrant
docker run -p 6333:6333 -p 6334:6334 -v $(pwd)/qdrant_storage:/qdrant/storage:z qdrant/qdrant

# 4. Populate Qdrant
python -m agent_search.scripts.populate_qdrant_from_postgres run --delete_existing=True

# 5. Run Server
python -m agent_search.app.server

  5. Use the pre-configured search agent endpoint

    main

    Use client.get_search_rag_response() to perform a Retrieval-Augmented Generation (RAG) task. This method searches for information, summarizes the results, and generates related queries in a single call.

    Parameters:

    • query: The search query string.
    • search_provider: The engine to use (e.g., 'bing').
    • llm_model: The model to use for summarization (e.g., 'SciPhi/Sensei-7B-V1').

    Returns: A dictionary containing:

    • response: The summarized answer.
    • related_queries: A list of suggested follow-up queries.
    • search_results: The raw search results used.
    # Requires SCIPHI_API_KEY in the environment
from agent_search import SciPhi

client = SciPhi()

# Search, then summarize result and generate related queries
agent_summary = client.get_search_rag_response(query='latest news', search_provider='bing', llm_model='SciPhi/Sensei-7B-V1')
print(agent_summary)
# {'response': "The latest news encompasses ...", 'related_queries': ['Details on the...', ...], 'search_results' : [...]}

  7. Build the AgentSearch documentation

    main

    To build the documentation locally, you need to install the required dependencies and use make to generate the HTML files. This process requires a Python environment with pip and a system with make installed.

    # Install dependencies.
pip install -r requirements-docs.txt

# Build the docs.
make clean
make html

  9. Obtain and use a SciPhi API Key

    main

    To access the AgentSearch API, you must sign up at https://www.sciphi.ai/signup to obtain an API key. All requests must include this key in the Authorization header as a Bearer token.

    Set your key in your environment as SCIPHI_API_KEY for convenience in CLI tools.

    export SCIPHI_API_KEY=${MY_API_KEY}