Arborium Documentation

Arborium is a batteries-included collection of tree-sitter grammars. Key capabilities include:

• Extensive Language Support: ~70 grammars included out of the box.
• Permissive Licensing: Default grammars use MIT, Apache-2.0, CC0, or Unlicense.
• WASM Support: Optimized for WebAssembly with a custom allocator fix.
• HTML Rendering: Includes 32 built-in themes for web-based syntax highlighting.
• Flexible Deployment: Supports CLI (terminal/HTML), Browser (script tag/ESM), and Rust library usage.
• Granular Control: Use feature flags to include only the specific language grammars required for your project.

Arborium organizes languages into semantic groups. When adding a new language, place it in the appropriate group directory:

Some languages require multiple grammars within a single crate (e.g., XML and DTD). To support this, use the grammar_path field in arborium.yaml to point to the specific subdirectory containing that grammar's files.

grammars:
  - id: xml
    name: XML
    # ...

  - id: dtd
    name: DTD
    grammar_path: dtd
    # ...

The injections.scm file (optional) allows you to define how one language is embedded within another. This is useful for highlighting code inside template literals, HTML tags, or specific string patterns.

Examples:

• Highlighting JavaScript inside <script> tags in HTML.
• Highlighting SQL inside Python string literals.
• Highlighting Markdown inside documentation comments.

; Highlight template literal strings with embedded expressions
((template_string) @injection.content
  (#set! injection.language "javascript"))

; Highlight JSX/TSX
((jsx_element) @injection.content
  (#set! injection.language "jsx"))

Each grammar in arborium follows a specific structure. Some files are the 'source of truth' and are committed, while others are generated by xtask gen and should be gitignored.

Committed Files (Source of Truth):

• arborium.kdl: Grammar configuration.
• grammar/grammar.js: Tree-sitter grammar definition.
• grammar/scanner.c: Custom scanner (if applicable).
• queries/*.scm: Syntax highlighting and injection queries.
• samples/*: Test samples.

Generated Files (Gitignored):

• Cargo.toml
• build.rs
• src/lib.rs
• grammar/src/* (Tree-sitter generated code)

When using the arborium IIFE script via --html-in-header, the script performs the following logic automatically:

1. Detection: It scans the document for code blocks identified by the class="language-*" attribute.
2. Rust Exclusion: It automatically skips Rust code blocks. This is necessary because rustdoc already provides semantic <a links within Rust blocks that would be broken by standard highlighting engines.
3. Highlighting: It applies tree-sitter grammars to highlight all other detected language blocks (e.g., JSON, YAML, SQL, Shell, TOML).

Arborium uses a prioritized detection strategy to determine the language of the input:

1. Explicit --lang flag: The highest priority; if provided, this overrides all other methods.
2. File extension: If the input is a file path, the extension is used to detect the language.
3. Shebang line: For input provided via stdin or as a literal string, the tool checks for a shebang (e.g., #!/usr/bin/env python3).

To include a language in WASM builds (for browser environments), set generate_plugin: true in the arborium.yaml grammar definition.

When to enable:

• Popular languages (tier 1-2).
• Languages needed in browser environments.

When to avoid:

• Rarely used languages (to reduce bundle size).
• Very large grammars (>500KB WASM output).

You can check the resulting WASM size using: ls -lh target/wasm32-unknown-unknown/release/*.wasm

Languages are organized into semantic groups. Choosing the right group determines the directory structure and categorization.

• group-cedar (JVM Languages): For languages running on the Java Virtual Machine (e.g., Java, Kotlin, Scala, Clojure, Groovy).
• group-hazel (Scripting Languages): For languages used for scripting or automation (e.g., Python, Ruby, Bash, PHP, Lua).
• group-pine (Modern Languages): For languages designed recently with modern features (e.g., Rust, Swift, Dart, WIT).
• group-acorn (Web): For web-centric languages (e.g., TypeScript).

Note: Groups are semantic. If a language fits multiple groups (like Groovy), choose the one representing its primary use case (e.g., JVM interop).

Rule: Do not create new groups without discussion.

When adding a language, you must follow this directory layout within your chosen language group:

langs/group-{name}/{language}/def/
├── arborium.yaml          # REQUIRED: Configuration file
├── grammar/               # Grammar source files
│   ├── grammar.js         # REQUIRED: Tree-sitter grammar definition
│   └── scanner.c          # Optional: External scanner
├── queries/               # Tree-sitter query files
│   ├── highlights.scm     # REQUIRED: Syntax highlighting rules
│   ├── injections.scm     # Optional: Language injection rules
│   └── locals.scm         # Optional: Scope/symbol tracking
└── samples/               # Example code files
    └── example.{ext}      # REQUIRED: At least one sample file