llm-ui

llm-ui is a React library specifically designed for rendering Large Language Model (LLM) outputs. It focuses on providing a smooth, high-performance streaming experience for text and structured data generated by AI models.

Key capabilities include:

• Robust Markdown Rendering: Automatically handles and removes broken markdown syntax common in streamed LLM outputs.
• Custom Component Integration: Allows you to map specific LLM output patterns to your own custom React components.
• Smooth Streaming: Uses throttling to smooth out the visual pauses often seen during LLM token streaming.
• High Performance: Renders output at native frame rates.
• Advanced Code Highlighting: Provides code blocks for all languages using Shiki.
• Headless Design: The library is headless, meaning it provides the logic and structure while allowing you to bring your own styles and CSS.

The useLLMOutput hook accepts a throttle argument to control how the UI renders the streamed output. Throttling allows the UI to lag behind the actual LLM output to improve the visual experience.

Benefits of Throttling

• Smoothing: It can smooth out jittery or uneven pauses in the LLM's streamed output.
• Visual Polish: It allows blocks to hide 'non-user' characters (like the ## syntax in a markdown header) before they are rendered, creating a cleaner transition.

Trade-offs

• Latency: The primary disadvantage is that the LLM output is delayed in reaching the user.

Configuration Example

Throttling can be configured using modes like buffer and adjusted via options such as delayMultiplier.

<ExampleSideBySide
  client:load
  example="# H1\nHi Docs\n\n```typescript\nconsole.log('hello')\n```"
  throttle="buffer"
  options={{ delayMultiplier: 2.5 }}
/>

A block object defines how a specific pattern in the LLM output is identified and rendered. It requires a component and matching logic.

Key properties for a block:

• component: A React component that receives blockMatch as a prop.
• findCompleteMatch: Logic to identify when a block is fully contained within the llmOutput.
• findPartialMatch: Logic to identify when a block has started but hasn't finished yet.
• lookBack: A function used for smooth rendering. It allows you to modify the output or the visibleText to prevent flickering or awkward partial renders during streaming.

const block = {
  // The component to render when the block is matched.
  component: ({ blockMatch }) => <div>{blockMatch.block.match}</div>,

  // The logic to find a 'complete' match
  findCompleteMatch: (llmOutput: string) => ({
    startIndex: 1,
    endIndex: 10,
    outputRaw: "some llm output"
  }),

  // The logic to find a 'partial' match
  findPartialMatch: (llmOutput: string) => ({
    startIndex,
    endIndex,
    outputRaw
  }),

  // A lookback function for smooth rendering
  lookBack: ({
    output: '【{type:"buttons",buttons:[{text:"my bu"',
    isComplete: false,
    visibleTextLengthTarget: 10,
    isStreamFinished: true
  }) => {
    return {
      output: '{type:"buttons",buttons:[{text:"my bu"}]}',
      visibleText: "my bu"
    }
  }
}