Neon

RangeState is the foundation of Neon, designed for efficient, on-demand processing of range-based content. It uses a HybridSyncAsyncValueProvider to allow both low-latency synchronous paths for small documents and asynchronous paths for large ones.

Key types include:

• RangeProcessor: Performs on-demand processing (e.g., parsing).
• RangeValidator: Manages validation of content.
• RangeInvalidationBuffer: Consolidates invalidations to be applied optimally.
• SinglePhaseRangeValidator: Uses a single data source.
• ThreePhaseRangeValidator: Uses primary, fallback, and secondary data sources for multi-layered highlighting.

Neon supports overlaying token data from multiple sources to balance latency and quality. This is typically implemented using a ThreePhaseTextSystemStyler.

A common pattern is:

1. First pass (Fallback): A fast pattern-matching system (e.g., regex) for guaranteed low latency.
2. Second pass (Primary): A high-quality parser like tree-sitter which provides better accuracy.
3. Third pass (Secondary): High-latency, high-accuracy data like Language Server Protocol (LSP) semantic tokens to augment existing highlighting.

To determine which styles apply to specific parts of the text, Neon uses the TokenProvider type. This is typically driven by a parser that assigns semantic meaning to text ranges.

If you are performing semantic analysis on source code, you can use Tree-sitter via the TreeSitterClient type. This integration is available and can be used within TextViewHighlighter to automate the mapping of tokens to styles.

Neon is composed of three distinct layers:

1. RangeState: The lowest-level component. It provides the core building blocks for range-based content processing (parsing, validation, and invalidation) using a hybrid synchronous/asynchronous execution model. It is content-independent.
2. Neon: The top-level module for managing text styling. It is text-system agnostic and manages how semantic tokens are applied to text. It includes integration components for AppKit and UIKit (like TextViewHighlighter).
3. TreeSitterClient: A hybrid sync/async interface to SwiftTreeSitter. It handles the actual parsing of code using tree-sitter grammars and provides an API for queries and edits.

Neon provides several components for integrating with standard Apple text systems:

• TextViewHighlighter: A simple bridge between NSTextView/UITextView and TreeSitterClient.
• TextViewSystemInterface: Implements the TextSystemInterface protocol for standard text views.
• LayoutManagerSystemInterface, TextLayoutManagerSystemInterface, and TextStorageSystemInterface: Specialized implementations for TextKit 1 and 2.

Note: For flicker-free highlighting on keystrokes in TextKit 1, it is recommended to use an NSTextStorage subclass like TSYTextStorage (from the TextStory library) to precisely control the timing of invalidation and styling via RangeInvalidationBuffer.

Add Neon to your Package.swift dependencies. Note that you may need to explicitly include the TreeSitterClient and RangeState products depending on your target requirements.

dependencies: [
    .package(url: "https://github.com/ChimeHQ/Neon", branch: "main")
],
targets: [
    .target(
        name: "MyTarget",
        dependencies: [
            "Neon",
            .product(name: "TreeSitterClient", package: "Neon"),
            .product(name: "RangeState", package: "Neon"),
        ]
    ),
]

Neon is text system-independent and does not provide its own AppKit, UIKit, or SwiftUI views. To use Neon, you must provide your own text view and implement an interface for Neon to interact with it.

Integration requires two parts:

1. A TextSystemInterface protocol implementation to apply styles to the text.
2. Manual notification to Neon's components regarding changes to text content and visibility.

For a quick setup with NS/UITextView, you can use TextViewHighlighter, which handles most of these integration details automatically, though it offers less flexibility and performance than a custom implementation.

``TextViewHighlighter``

You can use TreeSitterClient.highlight to asynchronously produce an AttributedString for a given string using a specific language configuration and a TokenAttributeProvider.

let languageConfig = try LanguageConfiguration(
    tree_sitter_swift(),
    name: "Swift"
)

let attrProvider: TokenAttributeProvider = { token in
    return [.foregroundColor: NSColor.red]
}

let highlightedSource = try await TreeSitterClient.highlight(
    string: source,
    attributeProvider: attrProvider,
    rootLanguageConfig: languageConfig,
    languageProvider: { _ in nil }
)