Carbonite

Carbonite implements the Change-Data-Capture (CDC) pattern using PostgreSQL triggers to ensure a complete and consistent audit trail. It is built around the concept that a database transaction is the natural unit of auditing.

Core Components

• changes table: A central table where all mutations (INSERT, UPDATE, DELETE) of participating tables are recorded. Triggers on audited tables automatically capture data and write it here.
• transactions table: Stores information about database transactions. Developers must insert a row into this table within a transaction to record arbitrary metadata. This metadata is then associated with all changes occurring in that same transaction.
• triggers table: Stores per-table settings for the trigger procedure, such as which columns to exclude from being captured.

Consistency Guarantees

Carbonite uses foreign keys and PostgreSQL's internal transaction ID (pg_current_xact_id()) to ensure:

1. No mutation without a recorded Change: Triggers ensure every change is captured.
2. No Change without a Transaction: A changes record must reference a valid transactions record. Attempting to modify an audited table without first inserting a record into the transactions table will result in a database error.
3. Atomic Commits: Because triggers run within the same transaction as the data mutation, the mutation and its audit trail are committed as a single atomic unit (ACID).

Limitations

• TRUNCATE is not supported: Carbonite does not instrument TRUNCATE statements.
• Performance Impact: Using triggers incurs a performance penalty as additional indexes and tables are touched during mutations.

An Carbonite.Outbox acts as a named cursor on the ordered list of transactions in your audit trail. It is used to offload captured data to external stores, export data, or purge old records.

Key characteristics:

• Ordering: Transactions are processed in ascending order by their id attribute (roughly in order of insertion).
• Statefulness: The outbox remembers its last processed transaction position and can persist an arbitrary map (a "memo") to pass data between consecutive processor runs.
• Concurrency: Carbonite.process/4 does not acquire a lock or start a transaction. You must ensure only one process runs for a specific outbox at a time (e.g., using Oban's unique jobs).
• Idempotency: If the processor function raises an exception, the outbox position is not updated. The next run will re-process the same batch, so your processor must be idempotent.

# Create an outbox in a migration
Carbonite.Migrations.create_outbox("rabbit_holes")

# Process the outbox
Carbonite.process(MyApp.Repo, "rabbit_holes", fn transactions, _memo ->
  for transaction <- transactions do
    send_to_external_database(transaction)
  end

  :cont
end)

In testing environments (especially when using factories like ExMachina), you may want to temporarily disable Carbonite's change capturing to avoid the overhead of creating Carbonite.Transaction records.

There are two primary strategies:

Option 1: Selective Bypassing (Recommended for safety)

Keep the default trigger mode as :capture and use Carbonite.override_mode/2 to selectively disable it within specific test blocks. This ensures you don't accidentally forget to wrap production code in a transaction.

# Helper to wrap a function in ignored mode
def with_ignored_carbonite(fun) do
  Carbonite.override_mode(MyApp.Repo, to: :ignore)
  result = fun.()
  Carbonite.override_mode(MyApp.Repo, to: :capture)
  result
end

Option 2: Global Bypassing (Recommended for speed)

Set the trigger mode to :ignore globally for the :test environment in your configuration. You then manually enable capturing only in tests where you need to assert on audit data.

Configuration:

# config/test.exs
config :my_app, carbonite_mode: :ignore

Migration Setup:

@mode Application.compile_env!(:my_app, :carbonite_mode)

def up do
  Carbonite.Migrations.create_trigger(:rabbits)
  Carbonite.Migrations.put_trigger_config(:rabbits, :mode, @mode)
end

Carbonite allows you to attach arbitrary metadata to database transactions. This is useful for recording context (like a user_id or account_id) at a system boundary without explicitly passing it through every function call in your application logic.

Metadata is stored in the process dictionary and can be retrieved or updated using put_meta/2 and current_meta/0. When creating a new Carbonite.Transaction via changeset/1, the metadata currently in the process dictionary is automatically merged with any metadata provided in the params.

# Store metadata at a system boundary
Carbonite.Transaction.put_meta(:user_id, 123)

# Retrieve it later
meta = Carbonite.Transaction.current_meta()
# => %{user_id: 123}

# When creating a transaction changeset, the metadata is automatically included
changeset = Carbonite.Transaction.changeset(%{meta: %{reason: "update"}})
# The resulting changeset :meta field will contain %{user_id: 123, reason: "update"}