DOCSEARCHbyAlgolia
Submit Docs

harehare/mq

repository·main·Indexed 21 days ago

https://github.com/harehare/mq

A command-line tool for querying, filtering, and transforming Markdown files using a jq-like syntax. The ecosystem includes mq-crawl for converting HTML to Markdown with support for headless Chrome and WebDriver, mq-check for type inference and checking of mq code, and mq-dap for debugging via the Debug Adapter Protocol.

Tokens
132.5K
Snippets
471
Records
625
Agent score
75%

What's inside harehare-mq

  1. What is mq?

    main
    mq is a command-line tool designed for Markdown processing. It functions similarly to jq but is specifically optimized for Markdown structures, allowing you to query, filter, and transform Markdown content.

  2. Features of mq-lsp

    main

    The mq-lsp implementation provides the following IDE capabilities for mq development:

    • Diagnostics & Linting: Real-time syntax/semantic error reporting and optional mq-lint diagnostics (toggled via --enable-lint).
    • Code Intelligence: Intelligent code completion (selectors, functions, variables), hover information (documentation/types), and signature help.
    • Navigation: Go To Definition and Find References.
    • Symbol Management: Document Symbols (outline view) and Workspace Symbols (cross-file search).
    • Code Manipulation: Automatic formatting, Code Actions (e.g., adding missing include/import), Refactoring (extract/inline), and Rename (symbol updates across files).
    • Editor UI Support: Folding ranges for blocks/comments and Semantic Tokens for enhanced syntax highlighting.

  3. Use the mq MCP tools

    main

    The mq MCP server exposes four primary tools for processing content via mq queries:

    1. html_to_markdown: Converts HTML to Markdown and applies an optional mq query.
    2. extract_markdown: Extracts content from existing Markdown using mq queries.
    3. available_functions: Lists all available mq functions (returns names, descriptions, parameters, and examples).
    4. available_selectors: Lists all available mq selectors (returns names, descriptions, and parameters).

  4. Web Scraping with mq toolchain

    main

    The mq ecosystem provides three primary tools for web scraping and data extraction, depending on your requirements:

    1. mq-crawl: Best for fetching or crawling URLs to Markdown. It supports JavaScript rendering, multi-page crawling, and respects robots.txt.
    2. mq: Best for filtering or transforming Markdown or HTML using jq-like selectors. Use -I html to process HTML.
    3. mq's http() builtin: Best for making specific HTTP requests (like POST or custom headers) directly inside a query using the --allow-net flag. Use this when mq-crawl cannot satisfy the specific request requirements.

    Note: mq-crawl sends logs and statistics to stderr. When scripting, redirect these to avoid polluting your output: 2>/dev/null.

  5. Use quote and unquote for metaprogramming

    main

    For advanced metaprogramming, mq provides quote and unquote to control when code is evaluated:

    • quote(expr): Delays evaluation, treating the content as raw code to be generated rather than executing it immediately.
    • unquote(expr): Evaluates the expression immediately and injects the resulting value into the quoted block.

    This allows you to perform pre-computation or conditional code generation during the expansion phase.

    # Basic injection
macro make_expr(x) do
  quote: unquote(x) + 1
end

| make_expr(5)  # Returns 6

  6. Use Shift Operators (`<<` and `>>`) for type-specific transformations

    main

    Shift operators perform different operations based on the operand type. They map to the shift_left(value, amount) and shift_right(value, amount) builtin functions respectively.

    Left Shift (<<)

    • Number: Bitwise left shift (multiplies by 2^amount).
    • String: Removes amount characters from the start of the string.
    • Array: Appends the value to the end of the array.
    • Markdown Heading: Decreases heading depth (promotes heading, e.g., ## → #), minimum depth is 1.

    Right Shift (>>)

    • Number: Bitwise right shift on the truncated integer value.
    • String: Removes amount characters from the end of the string.
    • Array: Adds the value to the beginning of the array.
    • Markdown Heading: Increases heading depth (demotes heading, e.g., # → ##), maximum depth is 6.
    # Bitwise left shift on numbers
1 << 2
# => 4

# Remove characters from the start of a string
"hello" << 2
# => "llo"

# Promote a heading
let md = do to_markdown("## Heading 2") | first; |
md << 1
# => # Heading 2

# Bitwise right shift on numbers
4 >> 2
# => 1

# Remove characters from the end of a string
"hello" >> 2
# => "hel"

# Demote a heading
let md = do to_markdown("# Heading 1") | first; |
md >> 1
# => ## Heading 1

  8. Extending mq with external subcommands

    main

    You can extend mq functionality by adding custom subcommands. mq will look for executables starting with the prefix mq- in your PATH or in ~/.local/bin/.

    This allows you to build your own specialized tools and integrate them into mq workflows seamlessly.

  9. How mq-test discovers tests

    main

    The mq-test runner uses the CST (Concrete Syntax Tree) to identify test functions in .mq files. A function is treated as a test if it meets any of these criteria:

    1. Its name starts with test_.
    2. It is immediately preceded by a # @test or # [test] annotation comment.
    3. It is immediately preceded by a # @parametrize(...) annotation comment.

  10. Supported Types in mq

    main

    mq supports several core data types for representing values:

    TypeDescriptionExamples
    NumberNumeric values.1, 3.14, -42
    StringUnicode sequences using \{0x000} escapes."hello", "😊", "\u{1F600}"
    BytesRaw byte sequences prefixed with b. Only ASCII allowed unescaped.b"abc", b"\xf0\x9f\x99\x82"
    SymbolImmutable, interned identifiers prefixed with :.:value, :success, :ok
    BooleanTruth values.true, false
    ArrayOrdered collections.[1, 2, 3], array(1, 2, 3)
    DictKey-value mappings.{"a": 1}, dict(["a", 1])
    FunctionExecutable code.def foo(): 42;

  11. Import modules via HTTP/HTTPS

    main

    If the http-import feature is enabled, import and include support URLs.

    Security: By default, only github.com/harehare (via raw.githubusercontent.com) is allowed. Use --allowed-domain <domain> to permit others.

    GitHub Shorthand: You can omit the scheme for GitHub. The path is mapped to raw.githubusercontent.com using the following logic:

    • github.com/{owner}/{path}[@{version}]

    Caching and Reproducibility:

    • Fetched modules are cached in {system_cache_dir}/mq/.
    • An mq.lock file is created to record SHA-256 hashes of fetched content, ensuring reproducible builds. If remote content changes, mq will error unless you use --refresh-modules.
    # Plain URL
import "https://example.com/mymod.mq"

# GitHub shorthand
import "github.com/harehare/kdl.mq"
| kdl::kdl_parse("title \"Hello, World!\"")

  12. How the section module works

    main

    The section module allows splitting and filtering Markdown documents by their heading structure.

    Critical Requirement: Aggregate Mode

    Section functions require access to all document nodes at once. If you call a section function on a single node, mq will issue a warning and treat the node as a one-element array. To use section functions correctly, you must:

    1. Use the -A flag on the command line.
    2. Or use the nodes function in your query to pipe the full document into the section functions.

    Access Styles

    StyleSyntaxNotes
    importimport "section" then section::fn()Namespaced (Recommended)
    includeinclude "section" then fn()No namespace prefix
    -A flagmq -A 'section::fn()'Aggregate mode (processes all nodes)

    Programmatic Usage (Rust/Other)

    When using the section module outside the CLI, section objects are returned as plain dicts. You must explicitly convert them using section::collect() to get Markdown nodes.

    # Programmatic conversion example
import "section"
| nodes
| section::section("Installation")
| section::collect()