DOCSEARCHbyAlgolia
Submit Docs

MarkdownDB

repository·main·Indexed 19 days ago

https://github.com/flowershow/markdowndb

A JavaScript library and CLI tool that converts Markdown files into a structured, queryable database. It extracts metadata, tags, links, and tasks from YAML frontmatter and supports multiple backends including SQLite, MySQL, PostgreSQL, and JSON. It features Zod-based schema validation for frontmatter and provides a Node.js API for indexing folders and retrieving file data.

Tokens
18.9K
Snippets
79
Records
94
Agent score
66%

What's inside mddb

  1. What is MarkdownDB

    main

    MarkdownDB is a JavaScript library designed to transform a folder of markdown files into a queryable SQLite database. It extracts structured data from your plain text files—including frontmatter, tags, links, and tasks—and indexes them so you can perform standard SQL queries across your content.

    Key characteristics:

    • Stack-agnostic: It can be used with any JavaScript framework or in a standalone environment.
    • SQL-native: It uses a real SQLite database, allowing you to use standard SQL instead of a proprietary query language.
    • Decoupled: It focuses exclusively on indexing and providing an API; it does not manage your rendering pipeline or tie you to a specific framework (unlike Contentlayer or Astro content collections).
    • Rich Extraction: Beyond frontmatter, it extracts inline tags, wikilinks, backlinks, and tasks.

  3. Overview of MarkdownDB

    main
    MarkdownDB is a JavaScript library that transforms Markdown files into a structured, queryable database. It parses Markdown files to extract metadata (frontmatter, tags, links, tasks) and builds an index in either JSON files or a local SQL database. This allows developers to build content-driven sites (blogs, wikis, docs) using standard SQL or a lightweight Node.js API.

  6. How MarkdownDB processes and stores data

    main

    MarkdownDB follows a pipeline to transform raw markdown into a queryable database:

    1. Parsing: Markdown files are parsed using remark-parse into a syntax tree.
    2. Extraction: Features (Metadata, Tags, Links) are extracted from the syntax tree to create TypeScript objects.
    3. Computation: These objects are processed and computed.
    4. Storage: The processed data is converted to SQL and stored in a markdown.db (SQLite) file, and also written to disk as JSON files within a .markdowndb folder.

  7. Extract metadata using Markdown frontmatter

    main

    MarkdownDB automatically extracts metadata from the frontmatter section of Markdown files. Frontmatter is defined at the beginning of a file, typically enclosed by triple dashes (---).

    When a file is indexed, the extracted fields are nested within a metadata.frontmatter object in the resulting SQL database or JSON output. This allows you to query and filter content based on custom attributes like author, date, or tags.

    ---
title: Introduction to Frontmatter
author: John Doe
date: 2023-01-15
tags: markdown, frontmatter
---

# Content starts here

    Resulting Data Structure:

    {
  "metadata": {
    "frontmatter": {
      "title": "Introduction to Frontmatter",
      "author": "John Doe",
      "date": "2023-01-15",
      "tags": ["markdown", "frontmatter"]
    }
  }
}

  8. Supported Database Backends

    main

    MarkdownDB uses Knex.js to support multiple database backends. All backends provide the same API and features:

    • SQLite (default): No additional setup required. Ideal for local development and small/medium sites.
    • MySQL: Best for larger sites requiring a separate server. Requires the mysql2 package.
    • PostgreSQL: Enterprise-grade support. Requires the pg package.

  9. Extract metadata and tasks from Markdown

    main

    MarkdownDB automatically extracts several types of structured data from your Markdown files:

    • Frontmatter: Extracts YAML fields (e.g., title, date) into a metadata object.
    • Tasks: Extracts markdown task lists (e.g., - [x] task) into a tasks array within metadata.
    • Tags: Extracts tags from both the tags frontmatter field and from the Markdown body (using #tag syntax).

  10. How Computed Fields work

    main

    Computed fields allow you to dynamically calculate and add new metadata properties to your files during the indexing process. This is useful for generating slugs from titles, extracting H1 headings, or assigning types based on folder structures.

    Implementation Steps

    1. Define the function: Create a function that accepts fileInfo (the metadata object) and ast (the Markdown Abstract Syntax Tree). Modify the fileInfo object directly.
    const addTitle = (fileInfo, ast) => {
  const headerNode = ast.children.find((node) => node.type === "heading");
  const title = headerNode
    ? headerNode.children.map((child) => child.value).join("")
    : "";

  fileInfo.title = title;
};
    1. Apply during indexing: Pass the function in the computedFields array when calling client.indexFolder.
    client.indexFolder("PATH_TO_FOLDER", { 
  computedFields: [addTitle] 
});
    const addTitle = (fileInfo, ast) => {
  const headerNode = ast.children.find((node) => node.type === "heading");
  const title = headerNode
    ? headerNode.children.map((child) => child.value).join("")
    : "";
  fileInfo.title = title;
};

  11. Understand the MarkdownDB database schema and relationships

    main

    The database uses a relational structure to connect files, tags, links, and tasks.

    Key Relationships:

    • Files & Tags: A many-to-many relationship via the file_tags table.
    • Files & Links: A self-referential relationship where links connect a source file (from) to a target file (to).
    • Files & Tasks: A one-to-many relationship where each task belongs to a file.

    Data Integrity: All foreign keys use CASCADE on delete. This means deleting a file in the files table will automatically remove all associated entries in file_tags, links, and tasks.

    files ──< file_tags >── tags
  │
  └──< links >── files (self-referential)

files ──< tasks

  12. How MarkdownDB validation works

    main

    When MarkdownDB loads a Markdown file, it automatically validates the file's frontmatter against the schema defined in markdowndb.config.js.

    If validation fails, MarkdownDB throws an error containing the filename, the schema name, and the specific validation error message.

    Example Error Formats:

    • Invalid format: Error: In 'blog.md' for the 'post' schema. Invalid date format. Please use YYYY-MM-DD format for the 'date' field.
    • Missing field: Error: Missing 'date' field in 'blog.md' for the 'post' schema.