Skip to main content
Version: Beta (v4.x)

Improving Answer Quality with Markdown Indexing

To deliver more accurate, context-rich answers at scale, AskAI benefits from cleanly structured content. One of the most effective ways to achieve this is by using a Markdown-based indexing helper in your Algolia Crawler configuration. This setup ensures AskAI can access well-formed, content-focused records—especially important for larger documentation sites where metadata, navigation elements, or layout artifacts might otherwise dilute the quality of generative responses.

info

Setting up markdown indexing can be automated through the Crawler UI for most use cases. For advanced customization or understanding the underlying configuration, manual setup options are also available.

Note: For more integration examples (Docusaurus, VitePress, Astro/Starlight, and generic setups), see the section at the bottom of this page.

Overview​

To maximize the quality of AskAI responses, configure your Crawler to create a dedicated index for Markdown content. This approach enables AskAI to work with structured, chunked records sourced from your documentation, support content, or any Markdown-based material—resulting in significantly more relevant and precise answers.

You can set up markdown indexing in two ways:

  1. Automated Setup (Recommended): Use the Crawler UI to automatically create and configure your markdown index
  2. Manual Configuration: Manually configure your Crawler for advanced customization needs

The easiest way to set up markdown indexing is through the Crawler UI, which automatically creates and configures an optimized markdown index for AskAI.

Step 1: Access Markdown Indexing in Crawler Configuration​

  1. Navigate to your Crawler dashboard
  2. Go to Configuration → Markdown for LLMs tab
  3. You'll see the Markdown Indexing section where you can create a dedicated markdown index

Crawler Configuration - Markdown for LLMs

Step 2: Add a New Markdown Index​

  1. Click "+ Add Index" to create a new markdown index
  2. Fill in the required fields:
    • Index Name: Enter a descriptive name (e.g., my-docs-markdown)
    • Content Tag: Specify the HTML content selector (typically main)
    • Template: Choose the template that matches your documentation framework:
      • Docusaurus - For Docusaurus sites
      • VitePress - For VitePress sites
      • Astro/Starlight - For Astro/Starlight sites
      • Non-DocSearch (Generic) - For custom sites or other frameworks

Add Markdown Index Dialog

  1. Click "Add Index" to create the index

The Crawler will automatically configure the optimal settings for your chosen template, including:

  • Proper record extraction and chunking
  • Framework-specific metadata extraction (language, version, tags)
  • Optimized index settings for AskAI

Step 3: Run the Crawler​

Once your markdown index is configured:

  1. Click "Start Crawling" to begin indexing your content
  2. Monitor the crawl progress in the dashboard
  3. Your new markdown index will be populated with clean, structured records optimized for AskAI

Crawler Status

Step 4: Integrate with AskAI​

After crawling completes, configure DocSearch to use your new markdown index for AskAI responses. See the Integration section below for detailed setup instructions.

Manual Configuration (Advanced)​

For users who need advanced customization or want to understand the underlying configuration, you can manually set up markdown indexing by modifying your Crawler configuration directly.

Step 1: Update your existing DocSearch Crawler configuration​

  • In your Crawler config, add the following to your actions: [ ... ] array:
// actions: [ ...,
{
  indexName: "my-markdown-index",
  pathsToMatch: ["https://example.com/docs/**"],
  recordExtractor: ({ $, url, helpers }) => {
    const text = helpers.markdown("main"); // Change "main" to match your content tag (e.g., "main", "article", etc.)
    if (text === "") return [];

    // Extract language or other attributes as needed. Optional
    const language = $("html").attr("lang") || "en";

    return helpers.splitTextIntoRecords({
      text,
      baseRecord: {
        url,
        objectID: url,
        title: $("head > title").text(),
        lang: language, // Add more attributes as needed
      },
      maxRecordBytes: 100000, // Higher = fewer, larger records. Lower = more, smaller records.
      // Note: Increasing this value may increase the token count for LLMs, which can affect context size and cost.
      orderingAttributeName: "part",
    });
  },
},
// ...],
  • Then, add the following to your initialIndexSettings: { ... } object:
// initialIndexSettings: { ...,
"my-markdown-index": {
  attributesForFaceting: ["lang"], // Add more if you extract more attributes
  ignorePlurals: true,
  minProximity: 4,
  removeStopWords: false,
  searchableAttributes: ["unordered(title)", "unordered(text)"],
  removeWordsIfNoResults: "allOptional" // This will help if the LLM finds no results. A graceful fallback.
},
// ...},

Step 2: Run the DocSearch crawler to create a new Ask AI optimized index​

After updating your Crawler configuration:

  1. Publish your configuration in the Algolia Crawler dashboard to save and activate it.
  2. Run the Crawler to index your markdown content and create the new index.

The Crawler will process your content using the markdown extraction helper and populate your new index with clean, structured records optimized for AskAI.

Tip: Monitor the crawl progress in your dashboard to ensure all pages are processed correctly. You can view the indexed records in your Algolia index to verify the structure and content.

Integrate your new index with Ask AI​

Once your Crawler has created your optimized index, you can integrate it with Ask AI in two ways: using DocSearch (recommended for most users) or building a custom integration using the Ask AI API.

Using DocSearch​

Configure DocSearch to use both your main keyword index and your markdown index for Ask AI:

docsearch({
  indexName: 'YOUR_INDEX_NAME', // Main DocSearch keyword index
  apiKey: 'YOUR_SEARCH_API_KEY',
  appId: 'YOUR_APP_ID',
  askAi: {
    indexName: 'YOUR_INDEX_NAME-markdown', // Markdown index for Ask AI
    apiKey: 'YOUR_SEARCH_API_KEY', // (or a different key if needed)
    appId: 'YOUR_APP_ID',
    assistantId: 'YOUR_ALGOLIA_ASSISTANT_ID',
    searchParameters: {
      facetFilters: ['language:en'], // Optional: filter to specific language/version
    },
  },
});
  • indexName: Your main DocSearch index for keyword search.
  • askAi.indexName: The markdown index you created for Ask AI context.
  • assistantId: The ID of your configured Ask AI assistant.
  • searchParameters.facetFilters: Optional filters to limit Ask AI context (useful for multi-language sites).

Tip: Keep both indexes updated as your documentation evolves to ensure the best search and AI answer quality.

Best Practices & Tips​

  • Use clear, consistent titles in your markdown files for better searchability.
  • Test your index with AskAI to ensure relevant answers are returned.
  • Adjust maxRecordBytes if you notice answers are too broad or too fragmented.
    • Note: Increasing maxRecordBytes may increase the token count for LLMs, which can affect the size of the context window and the cost of each AskAI response.
  • Keep your markdown well-structured (use headings, lists, etc.) for optimal chunking.
  • Add attributes like lang, version, or tags to your records and attributesForFaceting if you want to filter or facet in your search UI or AskAI.

FAQ​

Q: Why use a separate markdown index? A: It allows AskAI to access content in a format optimized for LLMs, improving answer quality.

Q: Can I use this with other content types? A: Yes, but markdown is especially well-suited for chunking and context extraction.

Q: What if I have very large markdown files? A: Lower the maxRecordBytes value to split content into smaller, more focused records.

For more details, see the AskAI documentation or contact support if you need help configuring your Crawler.

Crawler Configuration Examples by Integration​

Below are example configurations for setting up your markdown index with different documentation platforms. Each shows how to extract facets (like language, version, tags) and configure the Crawler for your specific integration:

Generic Example:

// In your Crawler config:

// actions: [ ...,
{
  indexName: "my-markdown-index",
  pathsToMatch: ["https://example.com/**"],
  recordExtractor: ({ $, url, helpers }) => {
    const text = helpers.markdown("main"); // Change "main" to match your content tag (e.g., "main", "article", etc.)
    if (text === "") return [];

    // Customize selectors or meta extraction as needed. Optional
    const language = $("html").attr("lang") || "en";

    return helpers.splitTextIntoRecords({
      text,
      baseRecord: {
        url,
        objectID: url,
        title: $("head > title").text(),
        // Add more optional attributes to the record
        lang: language
      },
      maxRecordBytes: 100000, // Higher = fewer, larger records. Lower = more, smaller records.
      // Note: Increasing this value may increase the token count for LLMs, which can affect context size and cost.
      orderingAttributeName: "part",
    });
  },
},
// ...],

// initialIndexSettings: { ...,
"my-markdown-index": {
  attributesForFaceting: ["lang"], // Recommended if you add more attributes outside of objectID
  ignorePlurals: true,
  minProximity: 4,
  removeStopWords: false,
  searchableAttributes: ["unordered(title)", "unordered(text)"],
  removeWordsIfNoResults: "allOptional" // This will help if the LLM finds no results. A graceful fallback.
},
// ...},

Each example shows how to extract common facets and configure your markdown index for AskAI. Adjust selectors and meta tag names as needed for your site.