Migrating from the legacy scraper
Introduction
This guide covers the historical migration from the self-hosted DocSearch scraper to the Algolia Crawler. It is separate from upgrading the frontend UI. For the current frontend, choose a DocSearch v5 package.
What's new?
Indexing infrastructure
DocSearch indexing now uses the Algolia Crawler. The DocSearch helper extracts records in place of the legacy DocSearch scraper.
You don't need to install local scraper tooling to maintain or update your index.
Use the legacy interface or the current Crawler interface to:
- Start, schedule and monitor your crawls
- Edit your config file from our live editor
- Test results with the historical DocSearch v3 or DocSearch v4 previews
These preview versions don't set the crawler record schema or the frontend package installed on your website. Likewise, crawler recordVersion: 'v3' names a record schema, not DocSearch UI v3.
Algolia application and credentials
We've received a lot of requests asking for:
- A way to manage team members
- Browse and see how Algolia records are indexed
- See and subscribe to other Algolia features
These features are available in your Algolia application.
FAQ
You can find answers related to the DocSearch migration in our Crawler FAQ page.
Useful links
Config file key mapping
Below are the keys that can be found in the legacy DocSearch configs and their translation to an Algolia Crawler config. For more detailed information on the Algolia Crawler, see the official documentation.
legacy | current | description |
|---|---|---|
start_urls | startUrls | Now accepts URLs only, see helpers.docsearch to handle custom variables |
page_rank | pageRank | Can be added to the recordProps in helpers.docsearch, should be passed as a string |
js_render | renderJavaScript | Unchanged |
js_wait | renderJavascript.waitTime | See documentation of renderJavaScript |
index_name | removed, see actions | Handled directly in the actions |
sitemap_urls | sitemaps | Unchanged |
stop_urls | exclusionPatterns | Supports micromatch |
selectors_exclude | removed | Should be handled in the recordExtractor and helpers.docsearch |
custom_settings | initialIndexSettings | Unchanged |
scrape_start_urls | removed | Can be handled with exclusionPatterns |
strip_chars | removed | # are removed automatically from anchor links, edge cases should be handled in the recordExtractor and helpers.docsearch |
conversation_id | removed | Not needed anymore |
nb_hits | removed | Not needed anymore |
sitemap_alternate_links | removed | Not needed anymore |
stop_content | removed | Should be handled in the recordExtractor and helpers.docsearch |