Darklang Documentation

LibParser is a hand-written recursive-descent parser for Darklang source code. It transforms raw source text into a syntax tree through a multi-stage pipeline. The pipeline consists of:

1. Lexing: Lexer.tokenize converts source into tokens and trivia (comments and positions), preserving whitespace text only for position tracking.
2. Syntax Parsing: Parser.parseSyntax performs a shared private syntax pass.
3. Lowering/Branching:
   • Tooling Path: Parser.parse produces a WrittenTypes tree along with syntax and structural diagnostics used for IDE/tooling support.
   • Execution Path: Parser.parseFor Mode performs structural and file-purpose validation, resulting in a ValidatedSourceFile. This is further lowered via WrittenTypesToProgramTypes into executable ProgramTypes.

To ensure consistency, the two lowering paths are verified via differential testing to ensure their ProgramTypes output is identical.

source
  │  Lexer.tokenize                  tokens + trivia
  ▼
  │  Parser.parseSyntax              shared private syntax pass
  ▼
  ├─ Parser.parse                    WrittenTypes tree + diagnostics
  │  └─ Builtins.Language.WrittenTypesToDarkTypes
  └─ Parser.parseFor Mode             ValidatedSourceFile
     └─ WrittenTypesToProgramTypes → executable ProgramTypes

The WASM REPL has specific behaviors regarding state and available builtins:

State Management

• Package Items: Declarations of fn, type, or val become in-memory package items under the Repl owner. These are callable without qualification in subsequent entries, and redefinitions will take precedence.
• Session Variables: A bare let x = ... declaration persists its bindings as session variables. These are injected into later entries as pre-loaded VM registers.
• Scope Limitation: Declarations cannot close over session variables; package items are considered static.

Builtins and Environment

• Standard Library: Stdlib.* is provided via the package snapshot.
• Core Builtins: Includes Builtins.Pure and Builtins.Http.Client (which uses fetch and is subject to CORS).
• Browser I/O: Supports printLine and print (buffered into results) and pmGetLocationsBy* lookups used by the pretty-printer.
• Output: The REPL output matches the behavior of dark eval.

Dark uses a "Vanilla" JSON serializer to handle arbitrary types. To prevent accidental breaking changes to serialization formats, Dark enforces strict allow-listing and testing requirements for any type intended for serialization.

Requirements for using Vanilla serialization:

1. Explicit Allow-listing: You must explicitly allow a type using Json.Vanilla.allow<type>(). Attempting to serialize or deserialize a type that has not been explicitly allowed will result in an exception.

   • Placement: The allow call should be placed in an init function within the same file that uses the serializer, rather than in the file where the type is defined.
   • Nesting: You only need to call allow for the top-level type being serialized; you do not need to explicitly allow all nested types within that structure.

2. Mandatory Test Cases: Every top-level type that is serialized must have a corresponding test case. This ensures that any changes to the type's structure are caught via git diffs in the serialization artifacts.

   • Tests will fail if an allowed type lacks a test case.
   • Tests will fail if a test case exists for a type that hasn't been explicitly allowed.