DOCSEARCHbyAlgolia
Submit Docs

mdq

repository·main·Indexed 23 days ago

https://github.com/yshavit/mdq

A command-line tool and Rust library providing jq-like querying capabilities for Markdown documents. It allows users to select, filter, and extract specific elements such as headers, lists, tasks, links, and table rows. The library supports parsing Markdown into a hierarchical tree structure (MdElem), performing text transformations via FlattenedText while preserving formatting, and rendering selected nodes back to Markdown.

Tokens
14.3K
Snippets
32
Records
78
Agent score
80%

What's inside mdq

  1. Markdown features supported by mdq

    main

    Based on the example documentation, mdq (jq for Markdown) is designed to parse and query various Markdown constructs. The following elements are supported for extraction and querying:

    • Headers: #, ##, ###, etc.
    • Text Styles: Italics, bold, and inline code.
    • Links:
      • Inline links: [text](url)
      • Links with titles: [text](url "title")
      • Reference links: [text][id]
      • Collapsed/Shortcut links: [text][] or [text]
    • Lists:
      • Ordered lists: 1. item
      • Unordered lists: - item
      • Nested lists
      • Task lists: [x] checked and [ ] unchecked
    • Tables: Standard Markdown pipe tables.
    • Blockquotes: > quote and nested blockquotes.
    • Code Blocks:
      • Fenced code blocks with language identifiers (e.g., ```types).
      • Code blocks with metadata (e.g., ```text title="...").
    • Footnotes: [^1] and [^1]: definition.
    • HTML: Inline HTML (e.g., <span>) and block-level HTML (e.g., <div>).

  2. Use mdq to select and render Markdown elements

    main

    The mdq CLI tool allows you to select and render specific elements from a Markdown document using selectors. You can provide selectors and a list of Markdown file paths, or pipe content via standard input.

    Basic Usage:

    mdq [OPTIONS] [selectors] [MARKDOWN_FILE_PATHS]...

    Input Behavior:

    • If no file paths are provided, mdq reads from standard input.
    • If file paths are provided, they are processed in order and treated as if concatenated into a single file.
    • To use standard input while also providing file paths, use - as a path.
    • A path of - represents standard input. Only the first occurrence of - is honored; subsequent - paths are ignored.
    mdq [OPTIONS] [selectors] [MARKDOWN_FILE_PATHS]...

  3. Install mdq

    main

    You can install mdq using several methods depending on your environment:

    • Homebrew (Mac/Linux): brew install mdq
    • Docker: docker pull yshavit/mdq
    • Cargo (Rust): cargo install --git https://github.com/yshavit/mdq (Requires rustc >= 1.85.1)
    • Manual: Download binaries from the latest release.

    Note for Mac users: If you encounter a quarantine error when using downloaded binaries, run xattr -d com.apple.quarantine mdq on the binary.

  4. Basic Usage of mdq

    main

    mdq allows you to select specific parts of a Markdown document using a syntax that mirrors Markdown itself. You can pipe results into further mdq commands to chain filters.

    Selection Syntax Reference

    ElementSyntax
    Sections# title text
    Lists- unordered list item text
    Ordered Lists1. ordered list item text
    Uncompleted Task- [ ] uncompleted task
    Completed Task- [x] completed task
    Any Task- [?] any task
    Links[display text](url)
    Images![alt text](url)
    Block quotes> block quote text
    Code blocks ```language <code block text>
    Raw HTML</> html_tag
    Plain paragraphsP: paragraph text
    Tables:-: header text :-: row text
    Front matter+++[toml|yaml] front matter text

    Text Matching Rules

    • Unquoted string: Starts with a letter; case-insensitive.
    • Quoted string: 'single' or "double"; case-sensitive.
    • Anchors: Use ^ for start of string and $ for end of string.
    • Regex: Use /regex/ for matching or !s/regex/replace/ for replacement.
    • Wildcard: Use * to match any text.

  5. Representing Sections and Hierarchical Headers

    main

    Unlike some parsers that treat headers as flat siblings to the body, mdq treats headers as part of a Section. A Section contains its depth, a title (as a Vec<Inline>), and a body (as a Vec<MdElem>).

    Nested headers (e.g., ## Subheader) are contained within the body of the parent Section, creating a true hierarchical tree.

  6. Use FlattenedText for text transformations with formatting preservation

    main

    The FlattenedText struct provides a way to perform text operations (like regex replacements) on Markdown inline content while preserving the underlying formatting. It separates the plain text from the formatting metadata.

    Core Workflow

    1. Flatten: Convert a collection of Inline elements into a FlattenedText instance using from_inlines. This extracts all plain text into a single string and records FormattingEvents.
    2. Transform: Use replace_range to modify the plain text. This method automatically adjusts the formatting_events to account for changes in text length and position, ensuring formatting stays aligned with the new text.
    3. Unflatten: Reconstruct the original Inline tree structure from the modified FlattenedText using unflatten.

  7. How link transformation works

    main

    The LinkTransformer orchestrates how links are processed during output generation. It uses a strategy pattern to decide how to handle different types of link references.

    Link Reference Types

    • Inline: [text](url)
    • Full(id): [text][id]
    • Collapsed: [text][]
    • Shortcut: [text]

    Transformation Logic

    Depending on the selected LinkTransform mode, the transformer applies the following logic via the apply method:

    1. Keep Strategy: Returns the link exactly as it was provided.
    2. Inline Strategy: Converts all link types (except potentially those that cannot be inlined) into LinkReference::Inline.
    3. NeverInline Strategy:
      • Converts Inline links into Full links with a new auto-incremented numeric ID.
      • Renumbers existing numeric Full links if they need to be reordered to accommodate new assignments.
      • Leaves Collapsed and Shortcut links untouched to avoid confusing the user with unexpected renumbering.

  8. Control placement of link and footnote references

    main

    When rendering Markdown, you can specify whether link and footnote definitions should appear at the end of the document or within their respective sections using MdWriterOptions.

    ReferencePlacement variants:

    • ReferencePlacement::Section: Definitions are placed at the end of the current section.
    • ReferencePlacement::Doc: Definitions are placed at the end of the entire document.

    This is useful for controlling the visual structure and flow of the generated Markdown output.

  9. Handle Inline elements and Text variants

    main

    The Inline enum represents elements that appear within text flows.

    Inline::Text is a terminal node that uses a TextVariant to distinguish between different types of content:

    • TextVariant::Plain: Standard text.
    • TextVariant::Code: Inline code (e.g., `code`).
    • TextVariant::Math: Inline math (e.g., $math$).
    • TextVariant::InlineHtml: Inline HTML tags (e.g., <span>).

    Other Inline variants include Span (for formatted text like strong or emphasis), Image, Link, and Footnote.

  10. Handle HTML in Markdown output

    main

    The library supports including raw HTML within Markdown via MdElem::BlockHtml and Inline::Text with TextVariant::InlineHtml.

    • Block HTML: Use MdElem::BlockHtml(String) to insert multi-line or single-line HTML blocks.
    • Inline HTML: Use Inline::Text with TextVariant::InlineHtml to insert HTML tags (like <span>) directly within text flows.

  11. How Markdown elements are structured and nested

    main

    The mdq library transforms a linear stream of Markdown elements into a hierarchical tree structure using MdElem::all_from_iter.

    Nesting Logic

    • Sections: Elements like MdElem::Section (headers) act as containers. Subsequent elements (like Paragraph or other Sections) are nested within the body of the current section until a header of equal or higher depth is encountered.
    • Header Depth: Nesting is determined by the header depth. A depth: 1 header will contain all subsequent elements until another depth: 1 header appears. A depth: 2 header will be nested inside a depth: 1 header.
    • Linear to Tree: If you provide a linear list of elements, all_from_iter will group them into a tree where headers own the content following them.

  12. Supported Markdown elements and parsing behavior

    main

    The mdq parser supports a wide range of Markdown elements, including block-level elements (Paragraph, Blockquote, List, Html, etc.) and inline elements (Text, Emphasis, Strong, Delete, Link, Image, etc.).

    Key parsing behaviors include:

    • GFM Support: Using ParseOptions::gfm() enables GitHub Flavored Markdown features like autolinks for bare URLs/emails, task lists in lists, and footnote support.
    • Footnotes: Supports standard footnote references [^a] and definitions [^a]: .... Footnotes can be part of a cycle.
    • Links and Images: Supports standard inline links, autolinks (bracketed or bare), and various reference types: Full, Collapsed, and Shortcut.
    • HTML: Supports both block-level and inline HTML tags.
    • Math: If enabled in ParseOptions, supports inline math using $ delimiters.