DeltaCrdt Documentation

repository·master·Indexed 20 days ago

https://github.com/derekkraan/delta_crdt_ex

An Elixir implementation of a distributed key/value store using Delta CRDT (Conflict-free Replicated Data Type) concepts and MerkleMap for efficient state synchronization. It includes support for Add-Wins Last-Write-Wins Maps (AWLWWMap), causal consistency via DeltaCrdt.CausalCrdt, and telemetry metrics for tracking synchronization.

Tokens
2.9K
Snippets
14
Records
16
Agent score
69%

What's inside DeltaCrdt

  1. Basic Usage Example for DeltaCrdt

    master

    This example demonstrates how to initialize two Delta CRDT nodes, link them as neighbors to enable replication, and perform basic key/value operations like put, read, take, and get.

    # start 2 Delta CRDTs
    {:ok, crdt1} = DeltaCrdt.start_link(DeltaCrdt.AWLWWMap)
    {:ok, crdt2} = DeltaCrdt.start_link(DeltaCrdt.AWLWWMap)
    
    # make them aware of each other
    DeltaCrdt.set_neighbours(crdt1, [crdt2])
    
    # show the initial value
    DeltaCrdt.read(crdt1)
    %{}
    
    # add a key/value in crdt1
    DeltaCrdt.put(crdt1, "CRDT", "is magic!")
    DeltaCrdt.put(crdt1, "magic", "is awesome!")
    
    # read it after it has been replicated to crdt2
    DeltaCrdt.read(crdt2)
    %{"CRDT" => "is magic!", "magic" => "is awesome!"}
    
    # get only a subset of keys
    DeltaCrdt.take(crdt2, ["magic"])
    %{"magic" => "is awesome!"}
    
    # get one value
    DeltaCrdt.get(crdt2, "magic")
    "is awesome!"
  2. Read values from an AWLWWMap

    master

    Use DeltaCrdt.AWLWWMap.read/1 (or its variants) to retrieve the current resolved state of the map. The read function resolves conflicts by selecting the value with the highest timestamp for each key.

    Available signatures:

    • read(crdt): Returns a map of all resolved key/value pairs.
    • read(crdt, keys): Returns a map containing only the specified keys.
    • read(crdt, key): Returns a map containing only the single specified key.
    # Get all values
    all_values = DeltaCrdt.AWLWWMap.read(crdt)
    
    # Get specific keys
    subset = DeltaCrdt.AWLWWMap.read(crdt, ["key1", "key2"])
    
    # Get a single key
    single = DeltaCrdt.AWLWWMap.read(crdt, "key1")
  3. Read data from a CRDT with `get/3`, `take/3`, and `to_map/2`

    master

    Retrieve data from the CRDT using the following methods:

    • get(crdt, key, timeout): Returns the value for a specific key, or nil if the key does not exist.
    • take(crdt, keys, timeout): Returns a list of [{key, value}] tuples for the requested keys.
    • to_map(crdt, timeout): Returns the entire state of the CRDT as a map.
    # Get one value
    val = DeltaCrdt.get(crdt, "my_key")
    
    # Get a subset of keys
    subset = DeltaCrdt.take(crdt, ["k1", "k2"])
    
    # Get everything
    full_map = DeltaCrdt.to_map(crdt)
  4. Start a Delta CRDT node with `start_link/2`

    master

    Use DeltaCrdt.start_link/2 to initialize a new Delta CRDT process. You must provide a CRDT module (e.g., DeltaCrdt.AWLWWMap) that defines the data structure's logic. The process is linked to the calling process.

    Configuration Options

    • :sync_interval: Time in milliseconds between synchronization attempts with neighbours. Default is 200. (Note: values below 100 will trigger a warning).
    • :on_diffs: A function or module/function/args tuple invoked on every diff.
    • :max_sync_size: Maximum number of items to sync in one batch. Default is 200.
    • :name: The name of the GenServer process.
    • :storage_module: A module implementing the DeltaCrdt.Storage behaviour.
    {:ok, crdt} = DeltaCrdt.start_link(DeltaCrdt.AWLWWMap, sync_interval: 3, name: :my_crdt)
  5. Perform operations on a CausalCrdt

    master

    You can modify the state of a CausalCrdt node using GenServer.cast/2 or GenServer.call/3. The node supports bulk operations to ensure atomicity within a single state update.

    Cast an operation

    Use GenServer.cast(pid, {:operation, {function, [key | args]}}) to apply a change. The function must be a valid function defined in your crdt_module that accepts the key, the node_id, and the current crdt_state.

    Call a bulk operation

    Use GenServer.call(pid, {:bulk_operation, operations}) where operations is a list of operation tuples. This returns :ok if all operations are applied successfully.

    Read the state

    • Full read: GenServer.call(pid, :read) returns the entire CRDT state.
    • Partial read: GenServer.call(pid, {:read, keys}) returns the values for the specified keys.
    # Cast a single operation
    GenServer.cast(pid, {:operation, {:add, "my_key", "my_value"}})
    
    # Perform a bulk operation
    ops = [
      {{"set", "key1", "val1"}},
      {{"set", "key2", "val2"}}
    ]
    GenServer.call(pid, {:bulk_operation, ops})
    
    # Read specific keys
    values = GenServer.call(pid, {:read, ["key1", "key2"]})
  6. Include DeltaCrdt in a supervision tree

    master

    You can use DeltaCrdt.child_spec/1 to integrate a CRDT node into an Elixir supervision tree. You must provide the :crdt module in the options.

    children = [
      {DeltaCrdt, [crdt: DeltaCrdt.AWLWWMap, name: :my_crdt_map]}
    ]
    # Example child spec usage
    DeltaCrdt.child_spec(crdt: DeltaCrdt.AWLWWMap, name: :my_crdt_map)
  7. Configure bidirectional syncing with `set_neighbours/2`

    master

    To enable communication and state synchronization between CRDT nodes, use DeltaCrdt.set_neighbours/2.

    Important: This function sets up unidirectional synchronization. To achieve bidirectional syncing (the standard use case), you must call the function for every node in the relationship. For example, to sync c1 and c2 bidirectionally, you must call:

    DeltaCrdt.set_neighbours(c1, [c2])
    DeltaCrdt.set_neighbours(c2, [c1])
    DeltaCrdt.set_neighbours(crdt1, [crdt2, crdt3])
  8. Add a key/value pair to an AWLWWMap

    master

    Use DeltaCrdt.AWLWWMap.add(key, value, i, state) to insert or update a value.

    Parameters:

    • key: The key to associate with the value.
    • value: The value to store.
    • i: The identifier for the node/replica performing the operation.
    • state: The current DeltaCrdt.AWLWWMap struct.

    Note: This operation uses System.monotonic_time(:nanosecond) internally to handle the Last-Write-Wins resolution.

    new_state = DeltaCrdt.AWLWWMap.add("my_key", "my_value", replica_id, current_state)
  9. Configure and initialize DeltaCrdt.CausalCrdt

    master

    The DeltaCrdt.CausalCrdt module is a GenServer that manages causal consistency and delta propagation for a CRDT. When starting the process, you can provide several configuration options in a keyword list.

    Configuration Options

    • crdt_module: The module implementing the CRDT logic (must respond to .new/0, .compress_dots/1, .read/1, .read/2, .join/3, and specific operation functions).
    • name: A unique identifier used for storage and telemetry.
    • on_diffs: A callback triggered when the local state changes. Can be a function fn(diffs) -> :ok end or a tuple {Module, function, args} where the function is called as function(args..., diffs).
    • storage_module: A module used to persist the CRDT state. Must implement .read(name) and .write(name, state_tuple).
    • sync_interval: The interval (in milliseconds) at which the node attempts to sync with its neighbours.
    • max_sync_size: Limits the size of a single sync payload. Can be an integer or the atom :infinite.

    Note: The node_id is automatically generated as a random integer upon initialization.

    # Example initialization
    DeltaCrdt.CausalCrdt
    |> GenServer.start_link(
      [
        crdt_module: MyCrdtModule,
        name: "my_node",
        sync_interval: 5000,
        max_sync_size: 100,
        on_diffs: fn diffs -> IO.inspect(diffs) end
      ]
    )