tree-sitter-kotlin

The project uses cross-validation to compare the tree-sitter-kotlin parse trees against the official JetBrains Kotlin compiler's PSI reference parser. This ensures the grammar accurately reproduces the official AST structure.

The process involves:

1. Vendoring: Fetching JetBrains PSI test fixtures (.kt source + .txt expected PSI tree) at a pinned commit.
2. Corpus Generation: Parsing these files with tree-sitter; files that parse cleanly (and aren't in excluded.txt) become corpus tests.
3. Structural Comparison: Comparing tree-sitter output against JetBrains' expected PSI tree using 110+ node type mappings.
4. Issue Tracking: Files that produce incorrect ASTs are tracked in tools/cross-validation/excluded.txt and categorized in tools/cross-validation/TODO.md.

To update the grammar's reference to a newer version of the JetBrains test fixtures, follow these steps. Note that the commit hash is stored in tools/cross-validation/.fixtures-version for reproducibility; you must commit this file when updating.

# Update to a specific commit
npm run vendor-fixtures -- <new-commit-hash>

# Regenerate corpus tests
npm run vendor-jetbrains

# Run tests
npm test

# Check for new matches or regressions
npm run cross-validate

To use the grammar in a browser-based environment, you must first compile it to WebAssembly using Emscripten.

Requirements:

• Emscripten installed (e.g., via brew install emscripten).
• Node.js 18+
• tree-sitter-cli

Steps:

1. Compile to WASM: npm run build-wasm
2. Launch interactive playground: npm run playground

# Compile WASM
npm run build-wasm

# Launch playground
npm run playground

To use the Kotlin grammar in a Rust project, add tree-sitter and tree-sitter-kotlin to your Cargo.toml dependencies. You can then use the tree_sitter_kotlin::language() function to provide the grammar to a tree_sitter::Parser instance via set_language().

tree-sitter = "0.23"
tree-sitter-kotlin = "0.3.9"

The grammar is defined in grammar.js. To (re-)compile the grammar and generate the parser files in the src directory, run the generate script.

npm run generate

The vendor-jetbrains.js tool generates tree-sitter corpus tests from the vendored JetBrains fixtures located in cross-validation/fixtures/.

Process:

1. Iterates through .kt files in the fixtures directory.
2. Skips *_ERR.kt files and files listed in cross-validation/excluded.txt.
3. Parses files using the tree-sitter parse CLI.
4. Filters out files that produce ERROR nodes or MISSING keyword tokens.
5. Converts zero-length named nodes into (MISSING _alpha_identifier) syntax.
6. Outputs one corpus test file per fixture to test/corpus/jetbrains/<Name>.txt.

Requirements:

• tree-sitter CLI (must be installed via npm install)

# Uses default fixtures path (tools/cross-validation/fixtures/)
npm run vendor-jetbrains

# Pass custom fixtures path as argument
npm run vendor-jetbrains -- /path/to/fixtures

You can run the unit tests using the test script. For manual verification, you can also run the tree-sitter CLI directly on Kotlin source files to inspect the parse tree and statistics.

# Run unit tests
npm run test

# Manually parse a project file
./node_modules/.bin/tree-sitter parse "/path/to/some/project/**/*.kt" --quiet --stat

The vendor-fixtures.js tool fetches JetBrains Kotlin PSI test fixtures using a sparse checkout at a specific commit hash. Fixtures are not checked into git; they are fetched on demand to ensure reproducibility via the .fixtures-version file.

Workflow:

1. Reads the pinned commit from cross-validation/.fixtures-version.
2. Sparse-clones compiler/testData/psi/ from the JetBrains/kotlin repository.
3. Copies .kt and .txt files into cross-validation/fixtures/.
4. Updates .fixtures-version if a new commit hash is provided.

Requirements:

• Git (for sparse checkout)
• Node.js

# Fetch at pinned commit (from .fixtures-version)
npm run vendor-fixtures

# Fetch at a specific commit (updates .fixtures-version)
npm run vendor-fixtures -- <commit-hash>

To set up the development environment for the Kotlin grammar, install the dependencies using npm.

npm install

To parse Kotlin source code, initialize a tree_sitter::Parser, set its language using tree_sitter_kotlin::language(), and then call parse().

let code = r"#
  data class Point(
    val x: Int,
    val y: Int
  )
"#;
let mut parser = Parser::new();
parser.set_language(&tree_sitter_kotlin::language()).expect("Error loading Kotlin grammar");
let parsed = parser.parse(code, None);