Hypercore

Hypercore is a secure, distributed append-only log designed for sharing large datasets and real-time data streams. Key features include:

• Sparse replication: Download only the specific data you need.
• Realtime: Fast and secure updates to the log.
• Performant: Uses a flat file structure to maximize I/O performance.
• Secure: Uses signed merkle trees for real-time log integrity verification.
• Modular: Focused specifically on distributing data streams.

To use Hypercore in your project, install it using npm:

npm install hypercore

When upgrading to version 11.0.0, note the following breaking changes:

• Sparse Option: The sparse option is no longer supported during Hypercore instance creation because all hypercores are now sparse by default.
• Encryption Configuration: The encryptionKey option is deprecated. Use the encryption option instead.
• Property Renaming: The property core.indexedLength has been renamed to core.signedLength.
• Storage Migration: If you provided a storage argument, Hypercore now automatically migrates storage to hypercore-storage.

Troubleshooting Storage Errors: If you encounter TypeError: db.columnFamily is not a function, you are likely attempting to use a legacy random-access-storage instance (such as random-access-memory or random-access-file) that is incompatible with the new storage requirements.

When upgrading to version 10.0.0, be aware of these changes:

• Endianness: All number encodings have switched to Little Endian (LE).
• Oplog: A new "oplog" has been introduced to atomically track local changes.
• Merkle Format: The merkle format has been updated to require only a single signature, which is stored in the oplog.

To reclaim storage by clearing unmarked blocks, use the Mark & Sweep pattern:

1. Enable marking mode: await core.startMarking().
2. Retrieve blocks you want to keep using .get(). These blocks are now 'marked'.
3. Clear all unmarked blocks: await core.sweep().

Alternatively, you can manually mark a range without loading it into memory using await core.markBlock(start, end).

Hypercore provides several streaming interfaces:

• createReadStream([options]): Reads a range of blocks. Can be consumed as an async iterator or piped.
• createByteStream([options]): Reads a range of raw bytes using byteOffset and byteLength.
• createWriteStream(): Appends chunks as blocks via a Node.js writable stream pattern.

Create a new Hypercore instance using the Hypercore constructor. You must provide a storage location, which can be a directory path, a Hypercore Storage instance, or a Corestore for efficient management of multiple cores.

Note: random-access-storage is no longer supported. If no key is provided, it will be loaded from storage or a new one will be generated if the storage is empty.

const core = new Hypercore('./directory') // store data in ./directory

Sessions allow you to create a new Hypercore instance that shares the same underlying storage. When used with an atom (created from core.state.storage.createAtom()), you can perform atomic batch changes across multiple hypercores. Changes are only persisted when await atom.flush() is called.

const core = new Hypercore('./atom-example')
await core.ready()
await core.append('block 1')

const atom = core.state.storage.createAtom()
const atomicSession = core.session({ atom })

await core.append('block 2') // Added to main core, not atom
await atomicSession.append('atom block 2') // Added to atom
await atom.flush()

// 'atom block 2' is now the latest block
console.log((await core.get(core.length - 1)).toString())

Use core.replicate(isInitiatorOrReplicationStream, opts) to create a replication stream.

• If isInitiator is true, you are the client/initiator.
• If isInitiator is false, you are the server/passive part.
• You can pass an existing stream to multiplex replication over it.

// On a server
const net = require('net')
const server = net.createServer(function (socket) {
  socket.pipe(remoteCore.replicate(false)).pipe(socket)
})

// On a client
const socket = net.connect(...)
socket.pipe(localCore.replicate(true)).pipe(socket)

Use core.get(index, [options]) to fetch a block by its index. If the data is not local, the method will wait for it to be downloaded unless wait: false is passed in options.

// get block #42
const block = await core.get(42)

// get block #43, but only wait 5s
const blockIfFast = await core.get(43, { timeout: 5000 })

// get block #44, but only if we have it locally
const blockLocal = await core.get(44, { wait: false })

Use core.append(block, options) to add a single block or an array of blocks (a batch) to the core. This returns the new length and byteLength of the core.

// simple call append with a new block of data
await core.append(Buffer.from('I am a block of data'))

// pass an array to append multiple blocks as a batch
await core.append([Buffer.from('batch block 1'), Buffer.from('batch block 2')])