stream-json

The stream-json library is migrating from a class-based OOP approach (extending Node.js Transform, Writable, or Duplex classes) to a functional composition approach using stream-chain patterns.

Why this matters:

• Portability: Functional logic (using flushable or generator functions) is decoupled from the Node.js stream lifecycle.
• Web Streams Readiness: Pure functions can be easily wrapped in a TransformStream for the WHATWG Streams API, whereas class-based Node streams require significant restructuring to migrate to Web Streams.
• Usage Patterns: Most modules now export a factory function for use with chain() and a .asStream() method for use with .pipe().

The core of stream-json is a SAX-inspired token protocol. The parser produces a stream of {name, value} tokens that serve as the universal interchange format for all downstream components like filters, streamers, and stringers.

Common tokens include:

• Structural: startObject, endObject, startArray, endArray, startKey, endKey.
• Values: keyValue (packed key), startString/endString/stringChunk/stringValue (for strings), startNumber/endNumber/numberChunk/numberValue (for numbers), nullValue, trueValue, falseValue.

Downstream components use specific type aliases for these stages:

• TokenSource: text → tokens
• TokenTransform: tokens → tokens
• TokenConsumer<Item>: tokens → items
• TokenStringer: tokens → text

The project is migrating from a class-based OOP approach (Node Streams) to a functional composition approach (Web Streams) using stream-chain as a bridge. The migration follows a tiered complexity model:

• Tier 1 (Trivial): Direct delegation to stream-chain primitives.
• Tier 2 (Medium): Rewritten using gen() pipelines or specific stream-chain primitives.
• Tier 3 (Complex): Rewritten using patterns like flushable + asStream() or complex generator compositions.

Modules like emitter.js are exceptions because they require a stream reference to handle events, making a purely functional form difficult.

The JsonlParser and JsonlStringer modules in stream-json have been re-implemented using stream-chain internals, but their public interfaces remain identical.

• JsonlParser: No action is required. It maintains its existing API, including errorIndicator, checkErrors, and the checkedParse() static method.
• JsonlStringer: No action is required. It maintains its existing API, though it is now powered by the more feature-rich stream-chain/jsonl/stringerStream.js (which supports additional options like prefix, suffix, space, and emptyValue).

The JSONC implementation is designed for seamless integration with the rest of the stream-json ecosystem.

• Downstream Compatibility: All existing components such as filters, streamers, and the assembler are fully compatible with the JSONC parser. Because these components are designed to ignore tokens they do not recognize, they will simply skip whitespace and comment tokens without requiring any code changes.
• Parser/Stringer Relationship: The JSONC parser is a fork of the standard parser, and the JSONC stringer is a fork of the standard stringer. This separation ensures that the performance of the standard JSON parser is not impacted by the additional logic required for JSONC.

stream-json provides several core modules that are unique to its token-based architecture and do not have direct equivalents in stream-chain. These include:

• parser.js: A SAX-like JSON tokenizer.
• assembler.js: Reconstructs tokens into objects (EventEmitter).
• disassembler.js: Converts objects into a token stream.
• stringer.js: Converts tokens into complex JSON text.
• emitter.js: Converts tokens into events (Writable stream).
• filters/*: Token-stream editors for manipulating the token protocol.
• streamers/*: Token-stream assemblers.
• utils/verifier.js: Standalone JSON validation.

The library uses a pipeline architecture where each component acts as a single stage. You compose these stages using stream-chain to process massive JSON files with a minimal memory footprint.

Typical pipeline flow:

1. Parser: Turns raw text/bytes into a token stream.
2. Filters: Trims or reshapes the stream (e.g., using pick to descend into specific keys or ignore to drop unwanted parts).
3. Streamers: Assemble the surviving tokens back into usable JavaScript objects.

This approach allows you to process documents far larger than available RAM because bytes that are skipped by filters are never assembled into memory.

Most rewritten modules in stream-json follow a consistent dual-interface pattern to support both functional chaining and standard Node.js piping:

1. Functional Chaining: Call the factory function fn(options) to get a flushable function or a generator pipeline. This is intended for use with stream-chain's chain() method.
2. Node.js Piping: Call fn.asStream(options) to get a standard Node.js stream (e.g., Duplex, Transform, or Writable) that can be used with .pipe().
3. Convenience Aliases: Some modules provide named aliases for easier access.

Note: The .make() method has been removed in favor of this pattern.

While the standard Assembler creates plain objects {} and arrays [], FlexAssembler allows you to substitute custom containers (like Map, Set, or custom classes) at specific JSON paths. This is useful when you want to change how data is structured during the assembly process without manual post-processing.

Rules are applied based on a filter. The first matching rule in your rules array takes precedence. If no rule matches a path, the assembler falls back to standard {} or [] behavior.

// Example: Using a Map for all objects
const asm = new FlexAssembler({
  objectRules: [
    {
      filter: () => true,
      create: () => new Map(),
      add: (map, key, value) => map.set(key, value)
    }
  ]
});