syntect

For text editors requiring high performance, syntect's API allows for a caching strategy to achieve near-instantaneous re-renders during edits.

Recommended Strategy:

1. Initial Parse: Every ~1000 lines, copy the current parse state into a side-buffer.
2. On Edit:
   • Search backwards in the parse state cache for the last state recorded before the edit location.
   • Start a background task to re-highlight from that cached state forward.
   • Render the new changes once the background task reaches the current editor viewport.
3. Concurrency: When a new edit occurs, stop any existing background highlighting jobs to ensure the thread is always working on the most up-to-date text state.

To add syntect to your Rust project, use the following command to update your Cargo.toml:

cargo add syntect

If you have cloned the repository locally, ensure you initialize submodules to fetch required dependencies for tests:

git submodule update --init

The repository includes several examples demonstrating different integration patterns:

• syncat: A simple file highlighting workflow that prints to the terminal using 24-bit color ANSI escape codes.
• synhtml: Demonstrates how to generate HTML output for highlighted code, suitable for web servers or static site generators.
• synstats: Shows how to use syntect APIs for code analysis and semantic tokenization (e.g., counting functions, lines, or comments).
• parsyncat: Demonstrates parallel highlighting of multiple files using multiple threads.

Detailed implementations can be found in the examples/ directory of the repository.

Syntaxes can inherit from other syntaxes using the extends field. When a child syntax extends a parent:

1. Variables: Child variables override parent variables.
2. Context Merging: Contexts can be merged using ContextMergeMode:
   • Replace: The child's version of the context wins entirely.
   • Prepend: Child patterns are placed before parent patterns.
   • Append: Parent patterns are placed before child patterns.
3. Scope Inheritance: If a child context is empty, it inherits metadata like meta_scope, meta_content_scope, and clear_scopes from the parent.
4. Regex Resolution: After extending, all regexes in the child syntax are re-resolved to ensure they work within the new context.

The syntect library is architecturally divided into two main stages for text processing:

1. Parsing: Using the parsing module to turn raw text into text annotated with scopes (syntax information).
2. Highlighting: Using the highlighting module to turn that annotated text into styled or colored text (e.g., HTML or terminal escapes).

For common use cases that combine these steps, refer to the easy module.

To perform incremental highlighting (e.g., highlighting a large file in chunks or resuming highlighting after a pause), you can capture and restore the internal state of HighlightLines.

1. Use HighlightLines::state(self) to consume the highlighter and retrieve the current (HighlightState, ParseState).
2. Use HighlightLines::from_state(theme, highlight_state, parse_state) to create a new HighlightLines instance starting from that exact state.

```rust
// 1. Capture state
let (highlight_state, parse_state) = highlighter.state();

// 2. Resume later with a new instance
let mut other_highlighter = HighlightLines::from_state(
    &ts.themes["base16-ocean.dark"],
    highlight_state,
    parse_state,
);

In version 2 (version: 2) syntax definitions, the set and clear_scopes commands have specific interactions with the scope stack:

• set: <context>: When switching contexts via set, the parser must correctly handle the meta_scope of the current context. The old meta scope is popped, and the new target context's meta scope is pushed.
• clear_scopes: <n>: This command removes n scopes from the stack. When used in conjunction with set: [ctx1, ctx2, ...] (where multiple contexts are targeted), the clear_scopes command applies at the specific position of each target context in the stack.
• meta_content_scope: This is a specialized scope used in V2. When a context is exited, its meta_content_scope is managed to ensure it doesn't leak into subsequent contexts, especially when clear_scopes or set are used.

When extending a parent syntax, you can control the order of patterns within a context using these flags:

• meta_prepend: true: Places the child's patterns at the beginning of the parent's context list. The child's patterns will be matched before the parent's.
• meta_append: true: Places the child's patterns at the end of the parent's context list. The parent's patterns will be matched first.

contexts:
  main:
    - meta_prepend: true
    - match: 'keyword'
      scope: keyword.child

A ScopeStack represents a sequence of scopes used to describe the hierarchy of a token of text. This is the same model used by Sublime Text and TextMate scope selectors.

To check if a selector (a ScopeStack) matches a given stack of scopes, use the does_match method. If it matches, it returns a MatchPower value.

MatchPower is a wrapper around an f64 that represents the strength of the match. Higher scores indicate stronger matches, where deeper and longer matches are prioritized. The scoring follows the logic that deeper matches in the stack carry more weight.

Note: The matching algorithm is guaranteed to be perfectly accurate up to stack depths of 17, after which it remains a very good approximation.