WHATWG Streams Standard

An abort signal is the dual to a cancel signal. It is sent down the pipe chain from a source that can no longer produce data (usually due to an error).

Unlike a standard "close" signal, an abort signal implies:

• The data written so far is not meaningful or valid.
• Any currently queued writes should be discarded.
• A cleanup operation should be performed (e.g., deleting a partially written file instead of keeping it).

A transform stream is a stream that is both readable and writable. When data is written to it, the data is transformed and then written out the other side. To use a transform stream within a pipeline, use the pipeThrough method. This allows you to insert processing steps (like compression, decoding, or data manipulation) between a source and a destination.

fs.createReadStream("source.zip")
    .pipeThrough(zlib.createGzipDecompressor(options))
    .pipeTo(fs.createWriteStream("dest/"));

fs.createReadStream("index.html")
    .pipeThrough(zlib.createGzipCompressor(options))
    .pipeTo(httpServerResponse);

fs.createReadStream("source.txt")
    .pipeThrough(new StringDecoder("utf-8"))
    .pipeThrough(database1.queryExecutor)
    .pipeTo(database2.tableWriter("table"));

A cancel signal is used when a consumer is no longer interested in data (e.g., a user navigates away from a video or an error occurs in a writable stream).

• Propagation: Cancel signals must propagate backward through the pipe chain, from the consumer up to the ultimate producer.
• Cleanup: When a cancel signal is received, the stream should stop consuming data and clean up underlying resources like sockets or file descriptors.
• Tee streams: In a tee stream, a cancel signal should only be sent upstream once both output streams have signaled they desire a cancel.

The Byte Streams API introduces three primary classes to manage byte-specific stream operations:

• ReadableStreamBYOBReader: A specialized reader vended by a byte-type ReadableStream that allows consumers to provide their own buffers for reading.
• ReadableByteStreamController: Provided to the underlying source during construction. It allows the source to control the stream's state and manage the internal queue specifically for byte streams.
• ReadableStreamBYOBRequest: Represents a request from the consumer to pull data into a specific buffer. This is accessed via the controller during the pull phase.

Backpressure is a mechanism that allows the slowest writable stream in a chain to govern the rate at which data is consumed from the source. This prevents memory usage from ballooning due to excessive queuing.

Key concepts include:

• Pull-based sources: Data is only pulled from the source when a consumer signals it is ready to read.
• Push-based sources: The stream issues a start signal to the source and a stop signal when the internal queue reaches a certain limit.
• High water mark: A threshold for in-memory queuing. Proactively pulling data up to this mark improves performance by making data immediately available to consumers.
• Low water mark: A threshold used to resume data collection before the queue is entirely empty, further optimizing throughput.

Byte streams are specialized versions of the generic stream API. They are designed to be naively substitutable where a non-byte stream is expected, meaning a byte stream API should be a superset of the generic stream API.

Key Byte Stream Features:

• BYOB (Bring Your Own Buffer): To optimize performance and memory usage, instead of the stream allocating a new ArrayBuffer for every read, consumers can supply their own pre-allocated buffer to read directly into. This allows for memory reuse.
• Upper Limits: Byte streams allow specifying a maximum number of bytes to be read (e.g., for reading file headers).
• Concurrency Safety: To prevent observable data races when reading into a buffer (potentially from another thread), the API may use techniques like detaching the array buffer.

For use cases like analytics or progress reporting, you can observe data flowing through a stream without interfering with the flow, backpressure, or queuing strategy.

To avoid the issues associated with traditional event emitters (such as having two sources of truth for the stream's state), the recommended strategy is to use AOP-style (Aspect-Oriented Programming) wrapping. This involves wrapping read or write calls to notify a separately-managed event emitter.

The Streams API is designed to be agnostic to the type of data being streamed. While many sources and sinks deal with binary data, the API supports streaming other types such as strings, objects, or video frames.

This allows for the creation of composable transform streams and the use of a single uniform interface for disparate objects like HTML elements, database records, or RPC messages.

Key characteristics:

• Composition: Different stream types can be composed together using a single interface.
• Backpressure & Queuing: The API manages automatic backpressure, queuing, and abort signals regardless of data type.
• Memory Management: Because data size varies by type, implementations should allow user-created streams to inform the system about data size (e.g., using a byte counter for ArrayBuffers, a character counter for strings, or a generic object counter).

The WritableStreamDefaultController now provides a signal property, which is an AbortSignal. This allows an underlying sink to observe when a stream is aborted and stop ongoing operations (like long-running writes or network requests) immediately, rather than waiting for the current write to complete.

To implement this, the underlying sink's write method should add an event listener to controller.signal for the 'abort' event. When the event triggers, the sink should reject the current operation or stop its work.

const ws = new WritableStream({
  write(controller) {
    return new Promise((resolve, reject) => {
      setTimeout(resolve, 1000);
      controller.signal.addEventListener('abort',
        () => reject(controller.signal.reason));
    });
  }
});
const writer = ws.getWriter();

writer.write(99);
await writer.abort();

Transferable streams allow you to offload heavy data processing to background threads to keep the main thread responsive. Common patterns include:

• Off-thread Transformations: Performing expensive tasks like transcoding media formats in a Worker.
• Service Worker Responses: Synthesizing responses in a Service Worker (e.g., generating a PDF from DOM data and streaming it to a download).
• Input Processing: Capturing data from main-thread-only APIs (like MediaRecorder for microphone/camera) and piping it through a TransformStream in a worker for processing before uploading.
• Expensive Data Generation: Downloading an experimental media format, transcoding it in a worker, and transferring the resulting stream back to the main thread for playback via <video> and MediaSource.