DOCSEARCHbyAlgolia
Submit Docs

Electric Clojure Documentation

repository·master·Indexed 24 days ago

https://github.com/hyperfiddle/electric

A full-stack differential dataflow framework for building interactive web products. Electric Clojure allows developers to write unified client/server code that the compiler automatically splits and synchronizes across a network boundary. Key features include a multi-tier architecture, network-transparent programming, and first-class reactivity. The documentation covers the Electric Protocol's message structure, session lifecycle consensus, and the three-stage compiler architecture consisting of the Expander, Analyzer, and Emitter.

Tokens
2.9K
Snippets
4
Records
13
Agent score
74%

What's inside Electric Clojure

  1. Core features of Electric Clojure

    master

    Electric provides several key abstractions for full-stack development:

    • Fully Reactive: Reactivity is a first-class citizen of the language (e.g., reactive-if, reactive-for, reactive lambdas). It avoids the need for manual observable management or async types.
    • Multi-tier: Frontend and backend logic are defined in the same expressions, allowing the compiler to handle code splitting rather than requiring the developer to architect around a boundary.
    • Network-transparent: Closures can close over both server and client scope bindings seamlessly.
    • Strong Composition: Because the system is built on Lisp, you have access to the full power of functional composition, including recursion, higher-order functions (HOFs), and macros, across the entire system.

  2. How the Analyzer uses the Triple Store

    master

    The Analyzer represents the program as a graph within a custom triple store. This allows for flexible, multi-pass data manipulation. The triple store consists of three parts:

    • o: An options map for arbitrary extra data.
    • eav: The main index. It maps an entity ID (:db/id) to its data map (e.g., {1 {:db/id 1, :foo :bar}}).
    • ave: A key-value index for graph traversal. It maps keys to a sorted set of entity IDs (e.g., {:foo {:bar (sorted-set 1)}}).

    Key Metadata Keys

    When working with or extending the analyzer, these keys are used to manage the graph:

    • :db/id: The internal entity key (often referred to as e). Functions returning this ID typically end with an -e suffix.
    • ::type: Categorizes the node.
    • ::parent: A universal backreference to the parent's :db/id. This allows both upward traversal (reading the key) and downward traversal (querying the :ave index).
    • ::uid: A universal, unchanging ID that survives graph rewrites, preventing stale backreferences.
    • ->id and ->uid: Used to generate monotonically-increasing integers to ensure node ordering via the triple store's sorted maps.

  3. Understand the Electric Protocol core concepts

    master

    The Electric protocol enables two peers (e.g., a JVM server and a browser client) to run a shared electric program and synchronize their states in response to local events.

    Key concepts include:

    • Peers: The two processes running the program.
    • Slots: A serializable value that uniquely identifies an expression being evaluated.
    • Transfer Sessions: Active lifecycles for synchronizing the differential state of an expression. The protocol manages these sessions to ensure both peers agree on which expressions are currently being synchronized.
    • Optimistic Transfers: An optimization where a peer anticipates the need to transfer an expression's state based on shared program knowledge, reducing latency by avoiding unnecessary round-trips.
    • Differential State Propagation: The mechanism for sending state changes (diffs) between peers. This relies on sequential delivery to prevent state corruption.

  4. How Electric's multi-tier architecture works

    master

    Electric allows you to build full-stack applications by composing client and server expressions directly within the same function or file.

    Instead of manually managing network plumbing, request/response cycles, or frontend/backend boundaries, the Electric compiler performs deep graph analysis on your unified program. It automatically infers the implied network boundary at compile time and splits your code into separate, cooperating reactive client and server target programs. This approach enables 'network-transparent' programming where closures can span across the server/client boundary through loops, recursion, and deeply nested calls.

  5. How session lifecycle consensus is reached

    master

    To prevent state corruption due to asynchronous updates, the protocol uses a consensus mechanism to manage the lifecycle of a transfer session (the 'half-port'). The protocol delays session teardown on local events to provide the remote peer a grace period, ensuring both sides agree on whether a session is active.

    Each half-port (input or output) tracks three values to determine its state:

    1. Local request count: Incremented/decremented when local events activate/deactivate a channel depending on this channel.
    2. Remote request count: Incremented/decremented when incoming messages activate/deactivate a channel depending on this channel.
    3. Pending toggle count: Incremented when the local request count is modified; decremented when the message containing the change propagation is acknowledged via acks.

    Session States:

    • Idle: A session is considered idle when the remote request flag is disabled, the inferred request flag is disabled, and the pending toggle count is zero.
    • Active: A session starts when leaving the idle state and stops when entering it again.

    State Flags (Notation):

    • P (Pending Toggle): Positive pending toggle count.
    • R (Remote Request): Positive remote request count.
    • I (Inferred Request): Positive local request count XOR pending toggle odd parity.

  6. Understand the Electric compiler architecture

    master

    The Electric compiler consists of three major stages that transform user code into electric runtime code. The compile variable orchestrates these stages in sequence:

    1. Expander: Expands all macros into electric built-ins. It handles the nuances between Clojure (clj) and ClojureScript (cljs) and ensures metadata is forwarded to support source mapping.
    2. Analyzer: The core logic engine. It takes expanded code and builds a graph representation in a triple store to determine the necessary electric code to generate. It performs multiple passes for effect ordering, dead code elimination (DCE), and optimization.
    3. Emitter: The final stage. It maps the analyzed graph (the triple store) into the final runtime code. The emit function is the primary entry point for this stage, while emit-ctor handles code generation for individual constructors.

  7. Build and deploy Electric to Clojars

    master

    To build and deploy the Electric Maven artifact to Clojars, first set the version in deps.edn under the :hyperfiddle.build/version key.

    Use the following commands to build, install locally, and deploy:

    clojure -T:build build
clojure -T:build install

# Deploy to Clojars (requires environment variables)
env $(cat .env | xargs) clojure -T:build deploy

    Required Environment Variables:

    • CLOJARS_USERNAME: Your Clojars username.
    • CLOJARS_PASSWORD: A generated Clojars token with deployment rights (do not use your account password).
    clojure -T:build build
clojure -T:build install

# To deploy:
env $(cat .env | xargs) clojure -T:build deploy

  8. Test Electric in electric-starter-app using a local installation

    master

    If you have installed a local version of Electric using clojure -T:build install, you can test it within the electric-starter-app by overriding the dependency in your clj command using the -Sdeps flag. Replace <installed version> with the version you built.

    clj -A:dev -X dev/-main -Sdeps '{:deps {com.hyperfiddle/electric {:mvn/version "<installed version>"}}}'

    Note: If you are trying to test a remote Clojars version and it is not updating, you may need to manually remove the existing version from your local Maven repository at .m2/repositories/com/hyperfiddle.

  9. Add Electric Clojure to your project

    master

    To use Electric Clojure, add the following dependency to your Clojure project using Maven coordinates. Note that the version provided is a snapshot for v3-alpha:

    `com.hyperfiddle/electric {:mvn/version "v3-alpha-SNAPSHOT"}`

  10. How the `request` map propagates local lifecycle changes

    master

    The request map is used to propagate the lifecycle of effects. When a peer performs an effect that depends on a remote expression, it must communicate this dependency.

    • Spawn: Associate the dependency slot with the integer 1.
    • Terminate: Associate the dependency slot with the integer -1.

    Multiple local events are aggregated into a single request map before being sent. The aggregation follows a monoid pattern where map values are added and zero values are elided.

    To implement this aggregation in Clojure, use a merge function like this:

    (def merge-request
  (partial reduce-kv
    (fn [r k n]
      (let [n (+ n (r k 0))]
        (if (zero? n)
          (dissoc r k)
          (assoc r k n))))))

  11. The Electric Protocol message structure

    master

    All messages sent over the wire are serializations of a 4-tuple: [acks request change freeze].

    ComponentTypeDescription
    acksNon-negative integerThe count of non-pure-ack messages received by the sender since the previous message was sent.
    requestMapAssociates slots to non-zero integers. Used to propagate local request lifecycle (e.g., spawning or terminating a process).
    changeMapAssociates slots to diffs. Carries the actual differential state updates.
    freezeSetA set of slots representing spontaneous termination of signal subscriptions.

    Note on Pure-Acks: A pure-ack is a message where request, change, and freeze are all empty. An empty message (pure-ack with acks: 0) has no effect and should not be sent.

  12. Analyzer passes and node types

    master

    The Analyzer operates through an initial analyze pass followed by analyze-electric, which performs deeper graph rewrites.

    Core Node Types

    • ::mklocal and ::bindlocal: Used to implement let and e/letfn. ::mklocal introduces a local variable, and ::bindlocal binds it. This separation enables support for circular and forward references.
    • ::localref: A reference to an electric local (e.g., the variable x in (let [x 1] x)).
    • ::lookup: A dynamic lookup for variables, supporting both symbols and non-symbolic keys (like keywords for private bindings or numbers for positional arguments).

    Optimization and Analysis Passes

    analyze-electric executes several specialized passes:

    • compute-effect-order: Assigns an ::fx-order integer to nodes to ensure side-effecting code is generated in the correct evaluation order.
    • mark-used-ctors: Performs Dead Code Elimination (DCE) by marking and ordering used constructors.
    • mark-used-calls2: Marks calls inside the previously marked constructors.
    • reroute-local-aliases: Simplifies the graph by pointing aliases directly to the original local (e.g., (let [x 1, y x] [y y]) $\rightarrow$ (let [x 1] [x x])).
    • optimize-locals & inline-locals: Decides if locals need to be runtime nodes or can be aggressively inlined.
    • order-nodes & order-frees: Orders nodes and frees based on the compute-effect-order.
    • collapse-ap-with-only-pures: Optimizes r/ap calls. If all arguments are pure, it collapses to r/pure; if an impure function is involved, it may wrap the pure calls in a single r/ap with a pure function.