DOCSEARCHbyAlgolia
Submit Docs

Hyperdrive

repository·main·Indexed 24 days ago

https://github.com/holepunchto/hyperdrive

A secure, real-time distributed file system that allows for versioned, replicable, and peer-to-peer storage of files and blobs. Operating on top of a corestore, Hyperdrive provides a file-system-like API for reading, writing, and deleting data, supporting atomic mutations via batching, symbolic links, and versioned snapshots through checkout. It includes streaming interfaces for large files and integrates with Hyperswarm for network replication.

Tokens
3.3K
Snippets
10
Records
21
Agent score
34%

What's inside hyperdrive

  2. Read and write files in Hyperdrive

    main

    Hyperdrive provides a file-system-like API for managing entries and blobs.

    Writing files

    Use put(name, buf, opts) to write a buffer to a specific path. This automatically handles the blob storage and the metadata entry.

    await drive.put('hello.txt', Buffer.from('hello world'))

    Alternatively, use createWriteStream(name, opts) to stream data into a file. If you want to avoid duplicating data for existing files, use the dedup: true option.

    const stream = drive.createWriteStream('large-file.bin', { dedup: true })
stream.write(data)
stream.end()

    Reading files

    Use get(name, opts) to retrieve the contents of a file as a Buffer.

    const buf = await drive.get('hello.txt')
console.log(buf.toString())

    For large files, use createReadStream(name, opts) to stream the content.

    const stream = drive.createReadStream('large-file.bin')
stream.on('data', (chunk) => { ... })

  3. Basic usage of Hyperdrive

    main

    Hyperdrive is a secure, real-time distributed file system that operates on top of a corestore. You initialize it by passing a Corestore instance to the Hyperdrive constructor. Once initialized, you can perform file operations such as writing (put), reading (get), deleting (del), and creating symlinks.

    const Hyperdrive = require('hyperdrive')
const Corestore = require('corestore')

const store = new Corestore('./storage')
const drive = new Hyperdrive(store)

await drive.put('/blob.txt', Buffer.from('example'))
await drive.put('/images/logo.png', Buffer.from('..'))
await drive.put('/images/old-logo.png', Buffer.from('..'))

const buffer = await drive.get('/blob.txt')
console.log(buffer) // => <Buffer ..> "example"

const entry = await drive.entry('/blob.txt')
console.log(entry) // => { seq, key, value: { executable, linkname, blob, metadata } }

await drive.del('/images/old-logo.png')

await drive.symlink('/images/logo.shortcut', '/images/logo.png')

for await (const file of drive.list('/images')) {
  console.log('list', file) // => { key, value }
}

const rs = drive.createReadStream('/blob.txt')
for await (const chunk of rs) {
  console.log('rs', chunk) // => <Buffer ..>
}

const ws = drive.createWriteStream('/blob.txt')
ws.write('new example')
ws.end()
ws.once('close', () => console.log('file saved'))

  4. List files in a directory with drive.list()

    main

    You can iterate over files in a specific directory using the drive.list(path) method. It returns an async iterator that yields objects containing the file's key and value.

    for await (const file of drive.list('/images')) {
  console.log('list', file) // => { key, value }
}

  5. Download drive contents

    main

    You can download specific parts of a drive using several methods. All return a Download object which can be awaited via .done() or cancelled via .destroy().

    • drive.download(folder, [options]): Downloads all blobs for entries prefixed with folder.
    • drive.downloadDiff(version, folder, [options]): Downloads all blobs in folder that were added between version and the current drive.version.
    • drive.downloadRange(dbRanges, blobRanges): Downloads entries and blobs within specific ranges.

    Example:

    const download = await drive.download('/my-folder')
await download.done()

  6. Write and read files in Hyperdrive

    main

    Hyperdrive allows you to store and retrieve data using paths.

    • drive.put(path, buffer, [options]): Creates a file at the specified path using the provided buffer. Options are compatible with createWriteStream.
    • drive.get(path, [options]): Returns the blob at path. Returns null if no blob exists or if the path is a symbolic link.

    get options:

    {
  wait: true, // Wait for block to be downloaded
  timeout: 0 // Wait at max some milliseconds (0 means no timeout)
}

  7. Prepare Hyperdrive for use with `ready()`

    main
    Before accessing synchronous properties like drive.discoveryKey, you must call await drive.ready(). This ensures the internal state is fully loaded. You only need to call this once.
    await drive.ready()

  8. Delete files and clear storage

    main

    Hyperdrive provides several ways to remove data:

    • drive.del(path): Deletes the file entry at path.
    • drive.clear(path, [options]): Deletes the blob from storage to free up space, but keeps the file structure reference.
    • drive.clearAll([options]): Deletes all blobs from storage to free up space.
    • drive.truncate(version, [options]): Reverts the drive to a previous version, removing both file structure and blobs.
    • drive.purge(): Completely removes all drive data (both db and blobs) from storage.

    clear options:

    {
  diff: false // Returned `cleared` bytes object is null unless enabled
}

  9. Inspect file entries with `drive.entry()`

    main

    Use drive.entry(path, [options]) to get detailed metadata about a file at a specific path. This is useful for distinguishing between regular files and symbolic links, and for accessing the underlying blob information.

    Entry object structure:

    {
  seq: Number,
  key: String,
  value: {
    executable: Boolean,
    linkname: null, // If entry is symlink, otherwise null
    blob: {
      blockOffset: Number,
      blockLength: Number,
      byteOffset: Number,
      byteLength: Number
    },
    metadata: null
  }
}

    entry options:

    {
  follow: false, // Follow symlinks, 16 max or throws an error
  wait: true, // Wait for block to be downloaded
  timeout: 0 // Wait at max some milliseconds (0 means no timeout)
}

  10. Create symbolic links in Hyperdrive

    main
    Use drive.symlink(path, linkname) to create an entry at path that points to the entry at linkname. If a blob already exists at path, it will be overwritten. drive.get(path) will return null for symlinks, but drive.entry(path) will return the symlink information.

  11. Access underlying Hyperblobs

    main

    You can access the Hyperblobs instance directly via drive.getBlobs(). This allows you to fetch blobs using the blob metadata object obtained from drive.entry().

    Example:

    await drive.put('/file.txt', Buffer.from('hi'))

const buffer1 = await drive.get('/file.txt')

const blobs = await drive.getBlobs()
const entry = await drive.entry('/file.txt')
const buffer2 = await blobs.get(entry.value.blob)

// buffer1 and buffer2 are equal
    await drive.put('/file.txt', Buffer.from('hi'))

const buffer1 = await drive.get('/file.txt')

const blobs = await drive.getBlobs()
const entry = await drive.entry('/file.txt')
const buffer2 = await blobs.get(entry.value.blob)

  12. Watch for changes in Hyperdrive

    main

    Use drive.watch([folder]) to get an iterator that yields changes in a specific folder (defaults to /). The iterator yields pairs of [current, previous] snapshots.

    Important: The snapshots yielded are automatically closed by the watcher before the next value is yielded. Do not manually close these snapshots as they are used internally.

    Watcher methods:

    • await watcher.ready(): Waits until the watcher is loaded and detecting changes.
    • await watcher.destroy(): Stops the watcher.
    const watcher = drive.watch()
await watcher.ready()

for await (const [current, previous] of watcher) {
  console.log(current.version)
  console.log(previous.version)
}