unified

repository·main·Indexed 26 days ago

https://github.com/unifiedjs/unified

A core engine for parsing, inspecting, transforming, and serializing content through syntax trees (ASTs) using a plugin-based architecture. It serves as the foundation for ecosystems such as remark (Markdown), rehype (HTML), and retext (natural language), allowing users to create processing pipelines that convert content between different formats.

Tokens
6.1K
Snippets
20
Records
33
Agent score
88%

What's inside unified

  1. Overview of unified

    main

    unified is a core package used to process content with plugins. It works with content as structured data (Abstract Syntax Trees or ASTs).

    It is part of a larger ecosystem of 500+ packages, including specialized ecosystems like:

    • remark: for Markdown
    • rehype: for HTML
    • retext: for natural language

    Use unified when you need to connect different types of content (e.g., converting Markdown to HTML) by picking and choosing specific plugins from these ecosystems.

  2. Understand the unified processing workflow

    main

    The unified ecosystem processes content using syntax trees through a three-step pipeline:

    1. Parse: A parser converts raw text into a syntax tree.
    2. Run: Plugins (transformers) inspect and modify the syntax tree.
    3. Stringify: A compiler converts the modified syntax tree back into text.

    A processor orchestrates this entire flow. On its own, the root unified() processor does nothing; it must be configured with plugins to perform specific tasks like parsing markdown or compiling HTML.

  3. Transform between ecosystems using Bridge and Mutate modes

    main

    When combining different syntax tree ecosystems (e.g., markdown to HTML), you can use two modes of transformation:

    • Bridge mode: Transforms the tree from an origin format to a destination format. A different processor runs on the destination tree, but the original processor continues working on the origin tree after the transformation is complete.
    • Mutate mode: Transforms the syntax tree from one format to another, but the original processor continues transforming the destination tree, effectively discarding the origin tree.

    Common plugins for these transformations include:

    • remark-retext: Markdown $\rightarrow$ Natural Language
    • remark-rehype: Markdown $\rightarrow$ HTML
    • rehype-retext: HTML $\rightarrow$ Natural Language
    • rehype-remark: HTML $\rightarrow$ Markdown
  4. Type your unified plugins with TypeScript

    main

    The unified package is fully typed. To ensure type safety, especially when working with different syntax trees (like hast, mdast, or nlcst), you should use the Plugin type with its generics.

    Common patterns for typing plugins include:

    • Plugins with options: Use Plugin<[(Options | null | undefined)?]>.
    • Plugins working on a specific tree: Use Plugin<[], TreeType>.
    • Plugins transforming one tree to another: Use Plugin<[], SourceTreeType, TargetTreeType>.
    • Parser plugins: Use Plugin<[], string, TreeType>.
    • Compiler plugins: Use Plugin<[], TreeType, string>.

    It is highly recommended to use the official node types for the syntax trees provided by related packages (e.g., @types/hast, @types/mdast, @types/nlcst).

    /**
     * @import {Root as HastRoot} from 'hast'
     * @import {Root as MdastRoot} from 'mdast'
     * @import {Plugin} from 'unified'
     */
    
    /**
     * @typedef Options
     *   Configuration (optional).
     * @property {boolean | null | undefined} [someField]
     *   Some option (optional).
     */
    
    // To type options:
    /** @type {Plugin<[(Options | null | undefined)?]>} */
    export function myPluginAcceptingOptions(options) {
      const settings = options || {}
      // `settings` is now `Options`.
    }
    
    // To type a plugin that works on a certain tree, without options:
    /** @type {Plugin<[], MdastRoot>} */
    export function myRemarkPlugin() {
      return function (tree, file) {
        // `tree` is `MdastRoot`.
      }
    }
    
    // To type a plugin that transforms one tree into another:
    /** @type {Plugin<[], MdastRoot, HastRoot>} */
    export function remarkRehype() {
      return function (tree) {
        // `tree` is `MdastRoot`.
        // Result must be `HastRoot`.
      }
    }
    
    // To type a plugin that defines a parser:
    /** @type {Plugin<[], string, MdastRoot>} */
    export function remarkParse(options) {}
    
    // To type a plugin that defines a compiler:
    /** @type {Plugin<[], HastRoot, string>} */
    export function rehypeStringify(options) {}
  5. Install unified

    main

    The unified package is ESM only. Install it based on your environment:

    Node.js (version 16+): Use npm to install the package.

    Deno: Import directly from esm.sh.

    Browsers: Import via a module script using esm.sh with the ?bundle query parameter.

    npm install unified
  6. Use unified to transform content

    main

    You can use unified to create a processing pipeline. By chaining .use() calls, you can add parsers, transformers (plugins), and compilers to convert content from one format to another (e.g., Markdown to HTML).

    In the example below, a Markdown string is parsed, converted to an HTML AST, wrapped in a document structure, formatted, and finally stringified into an HTML string.

    import rehypeDocument from 'rehype-document'
    import rehypeFormat from 'rehype-format'
    import rehypeStringify from 'rehype-stringify'
    import remarkParse from 'remark-parse'
    import remarkRehype from 'remark-rehype'
    import {unified} from 'unified'
    import {reporter} from 'vfile-reporter'
    
    const file = await unified()
      .use(remarkParse)
      .use(remarkRehype)
      .use(rehypeDocument, {title: '👋🌍'})
      .use(rehypeFormat)
      .use(rehypeStringify)
      .process('# Hello world!')
    
    console.error(reporter(file))
    console.log(String(file))
  7. Configure a unified processor with plugins

    main

    To create a functional processor, use the .use() method to chain plugins. Each plugin can perform parsing, transformation, or compilation.

    Example of a pipeline that parses markdown, converts it to HTML, adds a document title, formats the HTML, and then stringifies it to a string:

    const processor = unified()
      .use(remarkParse)
      .use(remarkRehype)
      .use(rehypeDocument, {title: '👋🌍'})
      .use(rehypeFormat)
      .use(rehypeStringify)
  8. Use the unified programming interface to process content and metadata

    main

    The unified API allows you to process content while gathering metadata (such as linting messages) into a vfile object. You can chain multiple ecosystems (like remark for markdown and retext for natural language) to perform complex transformations.

    import rehypeStringify from 'rehype-stringify'
    import remarkParse from 'remark-parse'
    import remarkPresetLintMarkdownStyleGuide from 'remark-preset-lint-markdown-style-guide'
    import remarkRehype from 'remark-rehype'
    import remarkRetext from 'remark-retext'
    import retextEnglish from 'retext-english'
    import retextEquality from 'retext-equality'
    import {unified} from 'unified'
    import {reporter} from 'vfile-reporter'
    
    const file = await unified()
      .use(remarkParse)
      .use(remarkPresetLintMarkdownStyleGuide)
      .use(remarkRetext, unified().use(retextEnglish).use(retextEquality))
      .use(remarkRehype)
      .use(rehypeStringify)
      .process('*Emphasis* and _stress_, you guys!')
    
    console.error(reporter(file))
    console.log(String(file))
  9. Parse text to a syntax tree with `processor.parse(file)`

    main

    Converts text (string or VFile) into a syntax tree. This method performs the parse phase and automatically freezes the processor if it is not already frozen.

    import remarkParse from 'remark-parse'
    import {unified} from 'unified'
    
    const tree = unified().use(remarkParse).parse('# Hello world!')
    
    console.log(tree)
  10. Serialize a tree with `processor.stringify(tree, file)`

    main

    Compiles a syntax tree into its textual representation (typically a string or Uint8Array). This performs the stringify phase. This method freezes the processor if it is not already frozen.

    import {h} from 'hastscript'
    import rehypeStringify from 'rehype-stringify'
    import {unified} from 'unified'
    
    const tree = h('h1', 'Hello world!')
    
    const document = unified().use(rehypeStringify).stringify(tree)
    
    console.log(document)
  11. Configure shared data with `processor.data()`

    main

    Use processor.data() to store information in an object that is accessible to all plugins in the processor. This is useful for sharing configuration (like a list of self-closing HTML elements) across multiple plugins.

    Signatures:

    • processor = processor.data(key, value) (Set a value)
    • processor = processor.data(dataset) (Set a whole dataset)
    • value = processor.data(key) (Get a value)
    • dataset = processor.data() (Get the entire dataset)

    Note: You cannot set data on a frozen processor. Call the processor first to create a new unfrozen instance.

    import {unified} from 'unified'
    
    const processor = unified().data('alpha', 'bravo')
    
    processor.data('alpha') // => 'bravo'
    processor.data() // => {alpha: 'bravo'}
    
    processor.data({charlie: 'delta'})
    processor.data() // => {charlie: 'delta'}
  12. Freeze a processor with `processor.freeze()`

    main

    A frozen processor is meant to be extended and not configured directly. Once frozen, it cannot be unfrozen. To use a frozen processor, you must call it to create a new unfrozen instance.

    Processors freeze automatically when calling .parse(), .run(), .runSync(), .stringify(), .process(), or .processSync().

    import rehypeParse from 'rehype-parse'
    import rehypeStringify from 'rehype-stringify'
    import {unified} from 'unified'
    
    // Create a frozen base processor
    export const rehype = unified().use(rehypeParse).use(rehypeStringify).freeze()
    
    // Use it by calling it to get a new unfrozen processor
    import rehypeFormat from 'rehype-format'
    rehype()
      .use(rehypeFormat)