Tree Sitter NG

The files CVTUTF7.C and CVTUTF7.H are intended for archival and historical purposes only. They have not been updated to Unicode 3.0 or later and are considered obsolescent.

Specifically, CVTUTF7.C contains functions to convert between UCS2 (BMP characters only) and UTF-7. This implementation:

• Does not support surrogates.
• Has not been tested.
• Is unsuitable for general purpose use.

The ConvertUTF.c and ConvertUTF.h files provide utilities for converting between different Unicode transformation formats.

Supported conversions include:

• UTF-32 to UTF-16
• UTF-32 to UTF-8
• UTF-16 to UTF-32
• UTF-16 to UTF-8
• UTF-8 to UTF-16
• UTF-8 to UTF-32

A test harness is included to run various conversion tests.

To use Tree Sitter NG in your Java project, add the core library and the specific language parsers you need as dependencies. Replace $VERSION with the desired version number.

Gradle:

dependencies {
    implementation 'io.github.bonede:tree-sitter:$VERSION'
    implementation 'io.github.bonede:tree-sitter-json:$VERSION'
}

Maven:

<dependencies>
    <dependency>
        <groupId>io.github.bonede</groupId>
        <artifactId>tree-sitter</artifactId>
        <version>$VERSION</version>
    </dependency>
    <dependency>
        <groupId>io.github.bonede</groupId>
        <artifactId>tree-sitter-json</artifactId>
        <version>$VERSION</version>
    </dependency>
</dependencies>

You can add new or community parsers using the provided gen Gradle task, which handles JNI bindings and boilerplate.

1. Generate the subproject: Run the gen task with the parser name, version, and the URL to the source code zip.

./gradlew gen --parser-name=kotlin --parser-version=0.3.8 --parser-zip=https://github.com/fwcd/tree-sitter-kotlin/archive/refs/tags/0.3.8.zip

This creates a new directory (e.g., tree-sitter-kotlin) and updates settings.gradle.

2. Build and test: Use Zig (built into the build system) to cross-compile the native libraries.

./gradlew :tree-sitter-kotlin:buildNative
./gradlew :tree-sitter-kotlin:test

To parse a string, instantiate a TSParser, set the language using a TSLanguage implementation (like TreeSitterJson), and call parseString. You can then traverse the resulting TSTree using TSNode methods.

TSParser parser = new TSParser();
TSLanguage json = new TreeSitterJson();
parser.setLanguage(json);
TSTree tree = parser.parseString(null, "[1, null]");
TSNode rootNode = tree.getRootNode();
TSNode arrayNode = rootNode.getNamedChild(0);

Tree Sitter NG supports parsing with specific encodings or via a custom TSReader for manual buffer management.

Parsing with encoding:

parser.parseStringEncoding(null, JSON_SRC, TSInputEncoding.TSInputEncodingUTF8);

Parsing with a custom TSReader:

TSReader reader = (buf, offset, position) -> {
    // implementation logic to fill buf
    return length;
};
TSTree tree = parser.parse(buffer, null, reader, TSInputEncoding.TSInputEncodingUTF8);

To visualize the AST, you can output it as a DOT graph file.

File dotFile = File.createTempFile("json", ".dot");
parser.printDotGraphs(dotFile);

Instead of using built-in Java classes for languages, you can load parsers from disk as shared objects using TSLanguage.load.

TSLanguage.load("path/to/language/shared/object", "tree_sitter_some_lang");

You can intercept parser messages by providing a TSLogger implementation to the parser.

TSLogger logger = (type, message) -> {
    System.out.println(message);
};
parser.setLogger(logger);

You can perform pattern matching on the AST using TSQuery and TSQueryCursor.

TreeSitterQuery query = new TSQuery(json, "((document) @root)");
TSQueryCursor cursor = new TSQueryCursor();
cursor.exec(query, rootNode);
SQueryMatch match = new SQueryMatch();
while(cursor.nextMatch(match)){
    // handle match
}

TreeSitterQuery query = new TSQuery(json, "((document) @root)");
TSQueryCursor cursor = new TSQueryCursor();
cursor.exec(query, rootNode);
SQueryMatch match = new SQueryMatch();
while(cursor.nextMatch(match)){
    // do something with the match
}