DOCSEARCHbyAlgolia
Submit Docs

modern-tar

repository·main·Indexed 41 days ago

https://github.com/ayuhito/modern-tar

A zero-dependency, cross-platform streaming tar archive library for JavaScript runtimes including Browsers, Node.js, and Workers. It leverages the Web Streams API for high performance and memory efficiency. Version 0.8.5 provides a core streaming API and a specialized `modern-tar/fs` entry point for Node.js filesystem utilities, supporting packing and unpacking of archives with native gzip integration via CompressionStream.

Tokens
12.6K
Snippets
39
Records
52
Agent score
51%

What's inside modern-tar

  1. Understand the two entry points of modern-tar

    main

    The package provides two distinct entry points depending on your environment and needs:

    1. modern-tar: The core, cross-platform streaming API. It uses the Web Streams API and is suitable for any modern JavaScript environment, including Browsers, Cloudflare Workers, and Node.js.
    2. modern-tar/fs: High-level filesystem utilities specifically designed for Node.js. These APIs use Node.js streams and provide advanced features like directory packing, extraction to local paths, and filesystem-specific options.

  2. Interpret modern-tar benchmark results

    main

    Benchmark results for modern-tar are primarily used to compare relative performance against other libraries like node-tar and tar-fs.

    Key Performance Characteristics:

    • Many Small Files: modern-tar shows significant performance advantages, often being 1.4x to 2.7x faster than competitors when handling large quantities of small files or nested small files.
    • Large Files: For huge files (e.g., 1GB+), performance is comparable to node-tar because the bottleneck shifts from CPU/logic to I/O.
    • Linked Files: modern-tar performs better than competitors when handling symlinks and hardlinks (e.g., in package manager workloads).

    Note on Accuracy: Results are subject to noise due to the heavy I/O-bound nature of tar workloads. Use them for relative comparison rather than absolute performance guarantees.

  3. Use Node.js filesystem utilities with modern-tar/fs

    main

    For Node.js applications, use modern-tar/fs to interact directly with the local filesystem. This allows you to pack entire directories or extract archives directly to a destination folder.

    import { packTar, unpackTar } from 'modern-tar/fs';
import { createWriteStream, createReadStream } from 'node:fs';
import { pipeline } from 'node:stream/promises';

// Pack a directory
const tarStream = packTar('./my/project');
await pipeline(tarStream, createWriteStream('./project.tar'));

// Extract to a directory
const tarReadStream = createReadStream('./project.tar', { highWaterMark: 256 * 1024 });
const extractStream = unpackTar('./output/directory');
await pipeline(tarReadStream, extractStream);

  5. Run modern-tar benchmarks

    main

    To run the informal performance benchmarks comparing modern-tar against node-tar and tar-fs, follow these steps in a pure Node environment:

    1. Navigate to the benchmarks directory.
    2. Install dependencies using npm.
    3. Execute the benchmark script via npm.

    Note: These benchmarks are intended for debugging and general reference, not for rigorous performance analysis.

    cd benchmarks
npm install
npm run bench

  6. Use the core streaming API for cross-platform tar operations

    main

    The core API works in any environment supporting Web Streams. You can pack arrays of entries or unpack existing archives.

    Important for Streaming: When using createTarDecoder to iterate over entries, you MUST read each entry body completely before advancing to the next. If you do not need an entry, you must call await entry.body.cancel() to prevent the stream from stalling.

    import { packTar, unpackTar, createTarDecoder } from 'modern-tar';

// Simple Packing
const archive = await packTar([
	{ header: { name: "file.txt", size: 5 }, body: "hello" },
	{ header: { name: "dir/", type: "directory", size: 0 } }
]);

// Simple Unpacking
const extracted = await unpackTar(archive);
for (const { header, data } of extracted) {
	// ... process entries
}

// Streaming Unpacking (e.g., from fetch)
const response = await fetch('/archive.tar');
const entries = response.body.pipeThrough(createTarDecoder());
for await (const entry of entries) {
	if (!entry.header.name.endsWith(".txt")) {
		await entry.body.cancel(); // MUST cancel if not reading
		continue;
	}
	const text = await new Response(entry.body).text();
}

  7. How streaming tar packing works

    main

    The modern-tar streaming packer follows a controller-consumer pattern using Web Streams:

    1. The Controller (TarPackController): Acts as the producer. You push TarHeader objects into it. For every header, it provides a WritableStream where you pipe the actual file content.
    2. The Sequential Constraint: The packer enforces a strict sequential workflow. You cannot start adding a new entry via controller.add() until the WritableStream of the previous entry has been closed. Attempting to do so will throw an error: "Previous entry must be completed before adding a new one".
    3. The Readable Stream: The readable stream returned by createTarPacker() is the consumer side. It emits the concatenated bytes of the headers and the file contents as they are processed, formatted as a valid tar archive.

  8. How the streaming unpacker state machine works

    main

    The modern-tar unpacker uses a state machine to process archives without loading them entirely into memory. The lifecycle of an extraction follows this pattern:

    1. STATE_HEADER: The unpacker is looking for a valid tar header. You call readHeader() to parse it. If not enough data is available, it returns null.
    2. Transition to STATE_BODY: Once readHeader() successfully parses a header, the state changes to STATE_BODY.
    3. STATE_BODY: The unpacker is now positioned at the start of the file content. You use streamBody(callback) to pull bytes out. You cannot call readHeader() while in this state.
    4. Return to STATE_HEADER: After the body is consumed, you must call skipPadding() to discard the alignment bytes and return to the header state to find the next entry.

    Error Handling: If strict: true is passed to createUnpacker, the unpacker will throw a Tar archive is truncated. error if the stream ends unexpectedly before a header or body is complete.

  9. Compress and decompress tar archives using gzip

    main

    Since modern-tar uses Web Streams, you can handle gzip compression/decompression by piping the tar stream through the native CompressionStream or DecompressionStream APIs.

    import { createTarPacker } from 'modern-tar';

// Packing with Gzip
const { readable, controller } = createTarPacker();
const gzipResponse = new Response(
  readable.pipeThrough(new CompressionStream('gzip')),
  { headers: { 'Content-Type': 'application/gzip' } }
);

// Unpacking Gzip
import { unpackTar } from 'modern-tar';
const response = await fetch('https://api.example.com/archive.tar.gz');
const entries = await unpackTar(
  response.body.pipeThrough(new DecompressionStream('gzip'))
);

  10. Compress tar streams with native gzip

    main

    Since createTarPacker() emits ArrayBuffer-backed chunks, you can pipe the resulting stream directly into the platform's native CompressionStream for gzip compression.

    const tarStream = /* ... */;
const compressedStream = tarStream.pipeThrough(new CompressionStream('gzip'));
const decompressedStream = compressedStream.pipeThrough(
  new DecompressionStream('gzip')
);

  11. Create complex archives with multiple sources in Node.js

    main

    The packTar function in modern-tar/fs accepts an array of TarSource objects, allowing you to mix files, directories, raw content, and streams into a single archive.

    import { packTar, type TarSource } from 'modern-tar/fs';
import { createWriteStream } from 'node:fs';
import { pipeline } from 'node:stream/promises';

const sources: TarSource[] = [
  { type: 'file', source: './package.json', target: 'project/package.json' },
  { type: 'directory', source: './src', target: 'project/src' },
  { type: 'content', content: 'Hello World!', target: 'project/hello.txt' },
  { type: 'content', content: '#!/bin/bash\necho "Executable"', target: 'bin/script.sh', mode: 0o755 },
  { type: 'stream', content: someReadableStream, target: 'project/data.bin', size: 1048576 }
];

const archiveStream = packTar(sources);
await pipeline(archiveStream, createWriteStream('project.tar'));

  12. Parse tar bytes using `createTarDecoder`

    main

    Use createTarDecoder() to create a stream pair that converts raw tar bytes into ParsedTarEntry objects.

    Important: When iterating over the entries stream, if you decide to skip an entry (e.g., via a filter), you MUST call entry.body.cancel() to drain the body. Failure to do so will cause the stream to stall and prevent reading subsequent entries.

    const decoder = createTarDecoder();
const entriesStream = tarStream.pipeThrough(decoder);

for await (const entry of entriesStream) {
  console.log(`Entry: ${entry.header.name}`);

  const shouldSkip = entry.header.name.endsWith('.md');
  if (shouldSkip) {
    // You MUST drain the body with cancel() to proceed to the next entry or read it fully,
    // otherwise the stream will stall.
    await entry.body.cancel();
    continue;
  }

  const reader = entry.body.getReader();
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    processChunk(value);
  }
}