DocSearch Documentation

The Ask AI API allows developers to build custom chat interfaces powered by Algolia's AI assistant. It enables creating conversational experiences that search an Algolia index and generate contextual responses using a user-provided LLM provider.

Note for DocSearch users: If you are using the standard DocSearch package, the Ask AI API integration is handled automatically and you typically do not need to interact with these endpoints directly.

Migration Warning: Ask AI is being migrated from a standalone DocSearch feature into Algolia's Agent Studio. DocSearch v5 will transition to using Agent Studio fully.

DocSearch v4 has migrated its infrastructure to leverage the Algolia Crawler. Key changes include:

• Managed Scraping: Instead of running a local scraper, DocSearch now uses the Algolia Crawler. You can manage crawls, schedule them, and monitor them via the Algolia Crawler dashboard (https://dashboard.algolia.com/crawler).
• No Local Tooling Required: You no longer need to install and maintain local scraping tools to update your index.
• Live Configuration: You can edit your configuration file using a live editor in the crawler dashboard.
• Direct Algolia Integration: DocSearch now operates within your own Algolia application, allowing you to manage team members, browse indexed records, and access other Algolia features directly.

The DocSearch scraper (Legacy versions 1.x - 2.x) is a Python-based tool inspired by the Scrapy framework. It crawls a website by following all internal links and extracts content from the HTML structure to populate an Algolia index.

By default, the scraper uses HTML semantics to build the index hierarchy: h1, h2, etc., are used as titles to establish hierarchy, and <p> tags are treated as potential search results. These default CSS selectors can be customized via a JSON configuration file to match a specific website's structure.

DocSearch is a search engine purpose-built for documentation and blogs. It is designed to index and surface technical content such as API references, how-to guides, code snippets, tables, and markdown structures.

Key features include:

• AI-Powered Search: Uses Algolia Ask AI to interpret natural-language queries and suggest synonyms.
• High Performance: Built on Algolia's global infrastructure, providing sub-20 ms response times.
• Customizable UI: Supports branding via CSS variables and JS hooks for colors, fonts, and layouts.
• Accessibility: Follows WAI-ARIA best practices and is tested against WCAG 2.1 standards for keyboard and screen-reader support.
• MCP Server Support: Allows connecting documentation to AI assistants (like Claude and Cursor) via the Model Context Protocol.

The Algolia Crawler Visual UI allows you to control the crawling lifecycle and monitor performance:

• Control Crawls: Use the Overview section to start, restart, or pause crawls and view real-time summaries.
• Monitor Errors: Use the Monitoring section to identify crawl errors and troubleshoot search result quality.
• Review Suggestions: The Suggestions section provides actionable feedback after each crawl, including recommendations for fixing redirects/domain issues, addressing ignored or failed URLs, and adding missing sitemaps.

The DocSearch tab in the Algolia Crawler Visual UI (v4.x) provides the necessary assets to integrate DocSearch into your frontend. It contains:

• Implementation code: Ready-to-use code snippets for frontend integration.
• API keys: Your unique Application ID and Search API Key required to connect your site to your Algolia index.
• Quick links: Links to review records, documentation, and support.

The Algolia Crawler Visual UI allows you to control and observe the indexing process through several sections:

• Overview: Use this section to start, restart, or pause crawls and view a real-time summary of the process.
• Monitoring: Use this section to identify crawl errors and troubleshoot issues affecting search results.
• Suggestions: Provides actionable feedback after each crawl, including recommendations for fixing redirects/domain issues, addressing ignored or failed URLs, and adding missing sitemaps.

The DocSearch tab in the Algolia Crawler Visual UI (Legacy v3.x) provides the necessary assets to integrate DocSearch into your frontend. It contains:

• Implementation code: Ready-to-use code snippets for frontend integration.
• API keys: Your unique Application ID and Search API Key required to connect your site to your Algolia index.
• Quick links: Access to record reviews, documentation, and support.

In DocSearch Legacy, the crawler discovers pages using two primary methods:

1. Recursive Crawling: The crawler starts at the URLs defined in the start_urls configuration value. It reads those pages and recursively follows every link found until all compliant pages have been browsed.
2. Sitemap Scraping: If a sitemap.xml is explicitly defined in the configuration, the crawler will scrape every provided and compliant URL in the sitemap. Using a sitemap is recommended to ensure all pages are discovered, even those not linked internally.

DocSearch (Legacy v3.x) consists of two main components:

1. Crawler: Managed by the Algolia Crawler. It is scheduled to run once a week by default, though crawls can be triggered manually. Configuration and monitoring are handled via the Crawler interface.
2. Frontend Library: Built on top of Algolia Autocomplete, providing a search experience via a modal interface.