DeltaCrdt Documentation
repository·master·Indexed 20 days ago
https://github.com/derekkraan/delta_crdt_exAn Elixir implementation of a distributed key/value store using Delta CRDT (Conflict-free Replicated Data Type) concepts and MerkleMap for efficient state synchronization. It includes support for Add-Wins Last-Write-Wins Maps (AWLWWMap), causal consistency via DeltaCrdt.CausalCrdt, and telemetry metrics for tracking synchronization.
What's inside DeltaCrdt
- When using DeltaCrdt, be extremely careful with atoms. Any atom used as a key or a value will be replicated across all nodes and will never be garbage collected by the BEAM, which can lead to memory exhaustion.
Install DeltaCrdt via Mix
masterTo use DeltaCrdt in your Elixir project, add the
delta_crdtpackage to yourdepslist inmix.exs.def deps do [ {:delta_crdt, "~> 0.6.3"} ] endBasic Usage Example for DeltaCrdt
masterThis example demonstrates how to initialize two Delta CRDT nodes, link them as neighbors to enable replication, and perform basic key/value operations like
put,read,take, andget.# start 2 Delta CRDTs {:ok, crdt1} = DeltaCrdt.start_link(DeltaCrdt.AWLWWMap) {:ok, crdt2} = DeltaCrdt.start_link(DeltaCrdt.AWLWWMap) # make them aware of each other DeltaCrdt.set_neighbours(crdt1, [crdt2]) # show the initial value DeltaCrdt.read(crdt1) %{} # add a key/value in crdt1 DeltaCrdt.put(crdt1, "CRDT", "is magic!") DeltaCrdt.put(crdt1, "magic", "is awesome!") # read it after it has been replicated to crdt2 DeltaCrdt.read(crdt2) %{"CRDT" => "is magic!", "magic" => "is awesome!"} # get only a subset of keys DeltaCrdt.take(crdt2, ["magic"]) %{"magic" => "is awesome!"} # get one value DeltaCrdt.get(crdt2, "magic") "is awesome!"Telemetry metrics for DeltaCrdt
masterDeltaCrdt publishes the following telemetry metric to track synchronization completion:
[:delta_crdt, :sync, :done]
[:delta_crdt, :sync, :done]Read values from an AWLWWMap
masterUse
DeltaCrdt.AWLWWMap.read/1(or its variants) to retrieve the current resolved state of the map. Thereadfunction resolves conflicts by selecting the value with the highest timestamp for each key.Available signatures:
read(crdt): Returns a map of all resolved key/value pairs.read(crdt, keys): Returns a map containing only the specifiedkeys.read(crdt, key): Returns a map containing only the single specifiedkey.
# Get all values all_values = DeltaCrdt.AWLWWMap.read(crdt) # Get specific keys subset = DeltaCrdt.AWLWWMap.read(crdt, ["key1", "key2"]) # Get a single key single = DeltaCrdt.AWLWWMap.read(crdt, "key1")Read data from a CRDT with `get/3`, `take/3`, and `to_map/2`
masterRetrieve data from the CRDT using the following methods:
get(crdt, key, timeout): Returns the value for a specific key, ornilif the key does not exist.take(crdt, keys, timeout): Returns a list of[{key, value}]tuples for the requested keys.to_map(crdt, timeout): Returns the entire state of the CRDT as a map.
# Get one value val = DeltaCrdt.get(crdt, "my_key") # Get a subset of keys subset = DeltaCrdt.take(crdt, ["k1", "k2"]) # Get everything full_map = DeltaCrdt.to_map(crdt)Start a Delta CRDT node with `start_link/2`
masterUse
DeltaCrdt.start_link/2to initialize a new Delta CRDT process. You must provide a CRDT module (e.g.,DeltaCrdt.AWLWWMap) that defines the data structure's logic. The process is linked to the calling process.Configuration Options
:sync_interval: Time in milliseconds between synchronization attempts with neighbours. Default is200. (Note: values below100will trigger a warning).:on_diffs: A function or module/function/args tuple invoked on every diff.:max_sync_size: Maximum number of items to sync in one batch. Default is200.:name: The name of the GenServer process.:storage_module: A module implementing theDeltaCrdt.Storagebehaviour.
{:ok, crdt} = DeltaCrdt.start_link(DeltaCrdt.AWLWWMap, sync_interval: 3, name: :my_crdt)Perform operations on a CausalCrdt
masterYou can modify the state of a
CausalCrdtnode usingGenServer.cast/2orGenServer.call/3. The node supports bulk operations to ensure atomicity within a single state update.Cast an operation
Use
GenServer.cast(pid, {:operation, {function, [key | args]}})to apply a change. Thefunctionmust be a valid function defined in yourcrdt_modulethat accepts the key, thenode_id, and the currentcrdt_state.Call a bulk operation
Use
GenServer.call(pid, {:bulk_operation, operations})whereoperationsis a list of operation tuples. This returns:okif all operations are applied successfully.Read the state
- Full read:
GenServer.call(pid, :read)returns the entire CRDT state. - Partial read:
GenServer.call(pid, {:read, keys})returns the values for the specified keys.
# Cast a single operation GenServer.cast(pid, {:operation, {:add, "my_key", "my_value"}}) # Perform a bulk operation ops = [ {{"set", "key1", "val1"}}, {{"set", "key2", "val2"}} ] GenServer.call(pid, {:bulk_operation, ops}) # Read specific keys values = GenServer.call(pid, {:read, ["key1", "key2"]})- Full read:
Include DeltaCrdt in a supervision tree
masterYou can use
DeltaCrdt.child_spec/1to integrate a CRDT node into an Elixir supervision tree. You must provide the:crdtmodule in the options.children = [ {DeltaCrdt, [crdt: DeltaCrdt.AWLWWMap, name: :my_crdt_map]} ]# Example child spec usage DeltaCrdt.child_spec(crdt: DeltaCrdt.AWLWWMap, name: :my_crdt_map)Configure bidirectional syncing with `set_neighbours/2`
masterTo enable communication and state synchronization between CRDT nodes, use
DeltaCrdt.set_neighbours/2.Important: This function sets up unidirectional synchronization. To achieve bidirectional syncing (the standard use case), you must call the function for every node in the relationship. For example, to sync
c1andc2bidirectionally, you must call:DeltaCrdt.set_neighbours(c1, [c2]) DeltaCrdt.set_neighbours(c2, [c1])DeltaCrdt.set_neighbours(crdt1, [crdt2, crdt3])Add a key/value pair to an AWLWWMap
masterUse
DeltaCrdt.AWLWWMap.add(key, value, i, state)to insert or update a value.Parameters:
key: The key to associate with the value.value: The value to store.i: The identifier for the node/replica performing the operation.state: The currentDeltaCrdt.AWLWWMapstruct.
Note: This operation uses
System.monotonic_time(:nanosecond)internally to handle the Last-Write-Wins resolution.new_state = DeltaCrdt.AWLWWMap.add("my_key", "my_value", replica_id, current_state)Configure and initialize DeltaCrdt.CausalCrdt
masterThe
DeltaCrdt.CausalCrdtmodule is aGenServerthat manages causal consistency and delta propagation for a CRDT. When starting the process, you can provide several configuration options in a keyword list.Configuration Options
crdt_module: The module implementing the CRDT logic (must respond to.new/0,.compress_dots/1,.read/1,.read/2,.join/3, and specific operation functions).name: A unique identifier used for storage and telemetry.on_diffs: A callback triggered when the local state changes. Can be a functionfn(diffs) -> :ok endor a tuple{Module, function, args}where the function is called asfunction(args..., diffs).storage_module: A module used to persist the CRDT state. Must implement.read(name)and.write(name, state_tuple).sync_interval: The interval (in milliseconds) at which the node attempts to sync with its neighbours.max_sync_size: Limits the size of a single sync payload. Can be an integer or the atom:infinite.
Note: The
node_idis automatically generated as a random integer upon initialization.# Example initialization DeltaCrdt.CausalCrdt |> GenServer.start_link( [ crdt_module: MyCrdtModule, name: "my_node", sync_interval: 5000, max_sync_size: 100, on_diffs: fn diffs -> IO.inspect(diffs) end ] )