cobol-rekt

Goal

Reduce the che-che4z-lsp-for-cobol-integration fork's patched-upstream surface from 20 files / 514 lines to approximately 14 files / 364 lines.

Strategy

Replace the _DIALECT_ <guid> marker with positional fragment correlation. This involves reparenting the three copied dialect visitors onto their upstream counterparts. Because upstream dialect visitors perform length-preserving blanking of dialect fragments, the extended-document position of a fragment remains identical before and after substitution, making the injected marker redundant.

Tech Stack

• Language: Java 17
• Build Tool: Maven (multi-module reactor)
• Parser: ANTLR 4
• Libraries: Lombok, JUnit 5, Mockito, Guice

The Db2SqlSubstitutingVisitor follows a similar pattern to the CICS implementation. It extends Db2SqlVisitor and records positional fragments for substituted EXEC SQL blocks using PersistentData.record(ctx, LocalisedDialect.DB2_SQL).

Key Requirements:

• It must produce the same nodes and errors as the original Db2SqlVisitor.
• It must record exactly one positional fragment per substituted EXEC SQL block.
• The Db2SqlVisitorBuilder.visitor(...) method must return a Db2SqlVisitor type.

Do not call PersistentData.reset() immediately after the COBOL parse() method returns.

PersistentData has a three-phase lifetime that extends beyond the parsing phase. Downstream consumers (such as IdmsTransferFlowNode or DialectStatementFlowNode) perform program analysis by calling NodeText.dialectOriginalText(), which relies on PersistentData.getDialectNode() to recover the original IDMS text. Resetting the state too early will cause these analysis steps to fail or return null.

The PersistentData class is used to hold a static list of fragments and a static set of claimed positions.

Critical Usage Note: PersistentData is not thread-safe by design. To prevent data corruption or positional correlation errors (where a sequential parse claims a fragment left over from a previous parse), PersistentData.reset() must be called at the entry point of the ParsePipeline.

The PersistentData class uses unprotected static mutable state (e.g., public static int counter, static List<AnnotatedParserRuleContext> trees).

Constraint: smojol is designed to parse files sequentially. Never parallelize calls to ParsePipeline in the same JVM without replacing PersistentData with a scoped, thread-local equivalent. This is a deliberate architectural choice to support dialect node re-injection.

The marker-free design relies on a specific reordering of the dialectStatement rule in the ANTLR grammar to ensure that dialectNodeFiller (which handles whitespace/markers) is correctly matched. The dialectStatement must be defined such that the filler is prioritized or correctly absorbed to prevent bare tokens from matching incorrectly.

conditionalStatementCall : statement | dialectStatement ;
dialectStatement         : dialectIfStatment | dialectNodeFiller ;
dialectNodeFiller        : ZERO_WIDTH_SPACE+ DOT_FS? EOF? ;

When working with PersistentData in the COBOL-LSP integration, be aware of the following constraints:

• Thread Safety: PersistentData holds a static fragment list and a static claimed set; it is not thread-safe by design.
• Lifecycle: PersistentData.reset() must be called at the entry point of the ParsePipeline.

Failure to call reset() can result in a sequential parse claiming a positionally identical fragment left over from a previous parse.

The SmojolSymbolTable builds two internal indexes from a CobolDataStructure tree to enable efficient and unambiguous lookups:

• bareIndex: A Map<String, List<CobolDataStructure>> that maps a bare name to all matching nodes. This is used to detect ambiguity when multiple fields have the same name.
• qualifiedIndex: A Map<QualifiedName, CobolDataStructure> used for unique, unambiguous lookups.

When reference(QualifiedName) is called, the table first checks the bareIndex. If multiple candidates exist, it filters them using isSuffixMatchedBy against the candidate's full ancestor path to find the specific match.

The smojol project uses a customized fork of the Eclipse Che4z COBOL Language Server (che-che4z-lsp-for-cobol-integration) as an embedded parser library rather than a running LSP server.

Because the fork contains custom logic to expose the parser as a Java library, upgrading to a new upstream version is a rebase/merge exercise, not a simple version bump. You must re-apply customisations (such as the poc package, grammar modifications, and CliModule) on top of every new upstream base to ensure the parser remains embeddable.

Architecture: The fork is a git submodule at `che-che4z-lsp-for-cobol-integration/` that was customised to act as an embeddable parser library. The customisations (the `poc` package, grammar modifications, `CliModule`) must be re-applied on top of each new upstream base.

Before committing a grammar upgrade, perform the following validation steps to ensure compatibility with the smojol core and toolkit:

1. Inventory Contexts: Search for all context types currently in use: grep -rn 'CobolParser\.[A-Z][A-Za-z]*Context' smojol-core/ smojol-toolkit/
2. Verify Rules: For every context type found, verify that the corresponding grammar rule still exists (e.g., rule fooBar must generate FooBarContext).
3. Identify Structural Changes: Watch for rules that have been split (introducing new intermediate wrappers) or merged (requiring token-based discrimination to distinguish types).
4. Run Tests: Execute the test suite for both modules: mvn test -pl smojol-core,smojol-toolkit