repository·main·Indexed 43 days ago
https://github.com/electric-sql/phoenix_syncReal-time data synchronization for Postgres-backed Phoenix applications. Built on ElectricSQL, it enables bi-directional sync between Postgres and clients (LiveView, Web, Mobile) using 'Shapes' for partial replication. Features include sync_stream/4 for LiveView, a Writer module for atomic write-path sync via Ecto.Multi, and support for both embedded and HTTP interaction modes.
TxidMatch is a tool designed to force PostgreSQL's transaction ID (txid) to wrap around. This is used to validate transaction ID matching logic.
Note: This tool is only tested and functional on Linux.
On typical hardware, forcing a wrap-around can take several hours. You can adjust the frequency of the wrap-around by tweaking the bumper_count setting in TxidMatch.Writer.Supervisor to better suit your machine's performance.
Phoenix.Sync provides four primary APIs to sync data from Postgres to your clients (LiveView, Web, or Mobile):
Phoenix.Sync.Client.stream/2: Low-level Elixir API to convert an Ecto.Query into an Elixir Stream.Phoenix.Sync.LiveView.sync_stream/4: Used within LiveView to automatically keep a stream up-to-date with the database state.Phoenix.Sync.Router.sync/2: A macro used in your Router to expose statically defined shapes. This allows clients to connect through your existing Plug middleware (e.g., for authentication).Phoenix.Sync.Controller.sync_render/3: A view function used in standard Controllers to return dynamic shapes defined at runtime based on the request/session.Phoenix.Sync supports two primary methods for defining shapes:
Ecto.Query or an Ecto.Schema module. This is ideal for dynamic shapes and allows you to control data synchronization using standard Ecto logic. Note that sync_render/3 in a controller is required for Ecto.Query support, as router macros do not support them to avoid compile-time dependencies.Phoenix.Sync is built on top of ElectricSQL, a real-time sync engine for Postgres. Phoenix.Sync leverages Electric to manage the core complexities of partial replication, data fan-out, and data delivery.
Data is partitioned and defined using Shapes, which are the mechanism for partial replication in ElectricSQL.
To use TxidMatch, you can set up a local PostgreSQL database using an in-memory data directory via Mix tasks.
Use the following command to create the local Postgres instance with a ramdisk/in-memory directory:
mix txid.postgres upYou can test how the system handles different ID types by passing different PostgreSQL functions to the mix txid command:
To test 64-bit IDs (expected to fail):
mix txid "txid_current()"To test 32-bit IDs (expected to work):
mix txid "pg_current_xact_id()::xid"Once testing is complete, stop the Postgres server and release the ramdisk memory:
mix txid.postgres downmix txid.postgres up
mix txid "pg_current_xact_id()::xid"
mix txid.postgres downTo keep a LiveView up-to-date with Postgres, replace Phoenix.LiveView.stream/3 with Phoenix.Sync.LiveView.sync_stream/4. You must also handle the {:sync, event} message in handle_info to update the stream using sync_stream_update/2.
defmodule MyWeb.MyLive do
use Phoenix.LiveView
import Phoenix.Sync.LiveView
def mount(_params, _session, socket) do
{:ok, sync_stream(socket, :todos, Todos.Todo)}
end
def handle_info({:sync, event}, socket) do
{:noreply, sync_stream_update(socket, event)}
end
endPlugSync can be configured using the PHOENIX_SYNC_MODE environment variable. This determines how the application interacts with the sync layer:
# Example: Running in http mode
PHOENIX_SYNC_MODE="http" mix phx.server
# Example: Running in embedded mode
PHOENIX_SYNC_MODE="embedded" mix phx.serverIf you are not using Igniter, you can configure Phoenix.Sync manually. The configuration depends on whether you are using :embedded or :http mode.
In :embedded mode, Electric is an application dependency and Phoenix.Sync consumes data internally via Elixir APIs. This is the most efficient mode for local development.
In :http mode, Phoenix.Sync consumes data from an external Electric service via its HTTP API. This is useful for production environments using Electric Cloud or when running Electric as a separate service.
# Example for Embedded Mode
# mix.exs
defp deps do
[
{:electric, "~> 1.0"},
{:phoenix_sync, "~> 0.6"}
]
end
# config/config.exs
config :phoenix_sync,
env: config_env(),
mode: :embedded,
repo: MyApp.Repo
# application.ex
children = [
MyApp.Repo,
# ...
{MyApp.Endpoint, phoenix_sync: Phoenix.Sync.plug_opts()}
]
# Example for HTTP Mode
# mix.exs
defp deps do
[
{:phoenix_sync, "~> 0.6"}
]
end
# config/config.exs
config :phoenix_sync,
env: config_env(),
mode: :http,
url: "https://api.electric-sql.cloud",
credentials: [
secret: "...", # required
source_id: "..." # optional, required for Electric Cloud
]
# application.ex
children = [
MyApp.Repo,
# ...
{MyApp.Endpoint, phoenix_sync: Phoenix.Sync.plug_opts()}
]Phoenix.Sync.Sandbox module. Refer to the module documentation for specific implementation details.PHOENIX_SYNC_MODE is set to `To sync writes from the client back to Postgres, use the Phoenix.Sync.Writer module. This allows you to ingest batches of writes (e.g., from local optimistic updates) and apply them atomically using Ecto.Multi.
Workflow:
Writer.new().Writer.allow/3 to register schemas and define authorization, loading, and validation logic.Writer.apply/4.By default, Writer.allow/3 uses the schema's changeset/2 function for validation.
defmodule MutationController do
use Phoenix.Controller, formats: [:json]
alias Phoenix.Sync.Writer
alias Phoenix.Sync.Writer.Format
def mutate(conn, %{"transaction" => transaction} = _params) do
user_id = conn.assigns.user_id
{:ok, txid, _changes} =
Writer.new()
|> Writer.allow(
Projects.Project,
check: reject_invalid_params/1,
load: &Projects.load_for_user(&1, user_id),
validate: &Projects.Project.changeset/2
)
|> Writer.allow(
Projects.Issue
# Uses sensible defaults: validate: Projects.Issue.changeset/2
)
|> Writer.apply(transaction, Repo, format: Format.TanstackDB)
json(conn, %{txid: txid})
end
endYou can use Igniter to automatically configure Phoenix.Sync in your project. This handles dependency installation and sets up the Ecto.Repo with the Phoenix.Sync.Sandbox test adapter.
First, add igniter to your mix.exs dependencies:
{:igniter, "~> 0.6", only: [:dev, :test]}Then, run the appropriate installation command based on your desired sync mode.
# For embedded mode
mix igniter.install phoenix_sync --sync-mode embedded
# For embedded mode without sandbox support
mix igniter.install phoenix_sync --sync-mode embedded --no-sync-sandbox
# For HTTP mode with an external URL
mix igniter.install phoenix_sync --sync-mode http --sync-url "https://api.electric-sql.cloud"