DOCSEARCHbyAlgolia
Submit Docs

antlr4-c3

repository·main·Indexed 19 days ago

https://github.com/mike-lischke/antlr4-c3

A grammar-agnostic code completion engine for ANTLR4 and antlr-ng based parsers. It provides a core implementation (CodeCompletionCore) that analyzes the Augmented Transition Network (ATN) of a grammar to suggest valid tokens and rules at a given caret position. The project includes a TypeScript original implementation and ports for Java, C#, C++, and Dart.

Tokens
11.8K
Snippets
41
Records
57
Agent score
65%

What's inside antlr4-c3

  1. Overview of antlr4-c3

    main

    antlr4-c3 is a grammar-agnostic code completion engine designed for antlr-ng and ANTLR4 based parsers. It provides a common infrastructure for implementing code completion in editors by walking the internal ATN (Augmented Transition Network) to find all possible valid paths from a given caret position.

    The original implementation is a TypeScript package compatible with both Node.js and browsers. Ports to Java, C#, C++, and Dart are available in the ports subfolder of the repository.

  2. Use the Java port of antlr4-c3

    main

    The antlr4-c3 Java port is a port of the original TypeScript implementation of the ANTLR4 Code Completion Core. It is designed to provide code completion logic for ANTLR-based languages.

    Key differences from the TypeScript implementation include:

    • Integration with the Java logging framework for debug messages.
    • The CandidatesCollection class includes an additional field that returns information about encountered preferred rules.

  3. Handle caret position and token index mapping

    main

    Mapping a visual caret position to a token index is not always a 1:1 relationship. To find the correct candidates, you must often adjust the token index based on the type of the token immediately preceding the caret.

    Common Scenarios:

    • Mid-word completion: If the caret is at the end of a word, it may visually belong to the current token index, but logically you are still completing that token.
    • Post-token position: After certain tokens (like an = sign), the position may map directly to the index.

    Warning on Whitespace: If your grammar uses the skip lexer action for whitespace, you will not have token indexes for those positions. This makes it impossible to distinguish between completing a keyword (e.g., var) and starting a new identifier.

    Best Practice: Instead of using skip for whitespace, place whitespaces on a hidden channel. This preserves token indexes and allows for unambiguous position mapping.

  5. How code completion works with antlr4-c3

    main

    The engine works by combining grammar knowledge with symbol information. A complete implementation typically involves two main components:

    1. Symbol Table: A source (derived from the current source code via a parser/listener or loaded from disk) that provides available symbols (variables, classes, etc.) visible at a specific position.
    2. c3 Engine: Determines which types of symbols are actually required by the grammar at the current position.

    Getting non-keyword symbols (Variables, Functions, etc.)

    By default, the engine only returns lexer tokens (keywords). To retrieve domain-specific entities like variables or class names, you must follow these steps:

    • Define explicit parser rules: Instead of using a generic lexer rule like ID directly in your expressions, wrap it in a specific parser rule. For example, instead of assignment: ID EQUAL expression;, use assignment: variableRef EQUAL expression; where variableRef: ID;.
    • Register preferred rules: Tell the engine which parser rules you are interested in by setting the CodeCompletionCore.preferredRules field. The engine will then return these rule indices instead of just lexer tokens.
    // Instead of this:
dropTable: DROP TABLE ID;

// Do this:
dropTable: DROP TABLE tableRef;
tableRef: ID;

  6. Getting Started with the c3 engine

    main

    To use the engine, you need a parser instance with a fully set up token stream. While the parser doesn't need to have successfully parsed the input previously, it must contain the ATN, vocabulary, and rule names. Note that predicates will only work if they are written for the JavaScript/TypeScript target.

    Basic Setup Example

    let inputStream = new CharStream.fromString("var c = a + b()");
let lexer = new ExprLexer(inputStream);
let tokenStream = new CommonTokenStream(lexer);

let parser = new ExprParser(tokenStream);
let errorListener = new ErrorListener();
parser.addErrorListener(errorListener);
let tree = parser.expression();

let core = new c3.CodeCompletionCore(parser);
let candidates = core.collectCandidates(0);

  7. Generate Dart code from ANTLR4 grammar

    main

    To use antlr4_c3 in a Dart project, you must first generate the Dart lexer and parser from an ANTLR4 grammar file (e.g., Expr.g4).

    Ensure the ANTLR4 version used for generation matches the version specified in your pubspec.yaml (e.g., 4.13.2).

    Use the following command structure to generate the code:

    antlr4 -v <VERSION> -Dlanguage=Dart <GRAMMAR_FILE> -o <OUTPUT_DIRECTORY>

    # Example: Generating Dart code from Expr.g4 into ../example/gen
# Ensure the version (4.13.2) matches your pubspec.yaml
antlr4 -v 4.13.2 -Dlanguage=Dart Expr.g4 -o ../example/gen

  8. Add the ANTLRv4 C3 C++ Port as a dependency

    main

    Currently, the C++ port does not support package managers or external dependency resolution. To use it in your project, you must copy and paste the entire source code directory into your own project structure.

    Source directory: ./source/antlr4-c3

    cp -r antlr4-c3/ports/cpp/source/antlr4-c3 /path/to/your/project/source/antlr4-c3

  9. Generate Dart code from ANTLR4 grammars

    main

    To use antlr4-c3 in a Dart project, you must first generate Dart source files from an ANTLR4 grammar file (e.g., Expr.g4). Use the antlr4 CLI tool with the -Dlanguage=Dart flag.

    Ensure the version passed to the -v flag matches the ANTLR4 version specified in your pubspec.yaml.

    # Expr.g4 is your grammar file
# 4.13.2 should match your pubspec.yaml version
# ../example/gen is your target output directory
antlr4 -v 4.13.2 -Dlanguage=Dart Expr.g4 -o ../example/gen

  11. Build the ANTLRv4 C3 C++ Port

    main

    To build the project, use CMake. The build process will automatically download the ANTLRv4 Runtime, ANTLRv4 Tool, and other necessary dependencies during the configuration stage.

    CMake Configuration Options

    • ANTLR4C3_CONAN: Should be set to OFF.
    • ANTLR4C3_DEVELOPER: Set to ON if you intend to run tests.
    • CMAKE_BUILD_TYPE: Supports Release, as well as Asan (AddressSanitizer) and Tsan (ThreadSanitizer).

    Build Requirements

    • C++ 20 standard
    • ANTLRv4 C++ Runtime
    • CMake 3.7+
    • ANTLRv4 Tool (for tests)
    • Google Test (for tests)
    git clone git@github.com:mike-lischke/antlr4-c3.git
cd antlr4-c3/ports/cpp

mkdir build && cd build

cmake \
    -DANTLR4C3_CONAN=OFF \
    -DANTLR4C3_DEVELOPER=ON \
    -DCMAKE_BUILD_TYPE=Release \
    ..

make

  12. Handle duplicate symbols in ScopedSymbol

    main

    When calling addSymbol(symbol), the ScopedSymbol checks for name collisions.

    By default, if a symbol with the same name already exists in the scope, a DuplicateSymbolError is thrown. This behavior is controlled by the allowDuplicateSymbols option in the owning symbolTable. If duplicates are allowed, the symbol is added normally, and the internal name counter is incremented.