Honker Documentation

repository·main·Indexed 25 days ago

https://github.com/russellromney/honker

Honker is a SQLite extension and set of language bindings providing Postgres-style NOTIFY/LISTEN, durable pub/sub, task queues, and event streams. It enables the transactional outbox pattern within SQLite without external brokers. It includes a core Rust foundation (honker-core v0.4.0) and supports multiple languages including Rust, Python, Node.js, Go, Ruby, Elixir, Bun, C++, and .NET.

Tokens
31.6K
Snippets
87
Records
229
Agent score
82%

What's inside Honker

  1. Overview of Honker Python usage patterns

    main

    The Honker Python package supports several core patterns demonstrated in the example suite:

    • Task Queuing: Uses Huey-style @queue.task() decorators. Calling add() returns a TaskResult, and .get() can be used to wait for the worker to complete the task. It includes support for timeout-to-dead-letter paths.
    • Atomic Transactions: Supports committing database operations (e.g., INSERT INTO orders) and queue enqueues (queue.enqueue(...)) within a single transaction. If the transaction rolls back, both the database change and the queued task are dropped.
    • Pub/Sub:
      • Ephemeral: pg_notify-style pub/sub for transient notifications.
      • Durable: Pub/sub with per-consumer offset tracking and the ability to resume after a crash.
    • Scheduling: Supports time-triggered scheduling with leader election. It handles cron-style schedules, every_s(...) intervals, and 6-field cron expressions.
    • Worker Loops: Provides both high-level decorated tasks and low-level async worker loops with retry-to-dead-letter logic.
  2. Core Honker functionality provided by honker-core

    main

    While honker-core is a foundation for bindings, it provides the following core capabilities:

    • Connection Management: open_conn(path, install_notify) opens a SQLite connection with Honker's PRAGMA defaults and optional notify() installation.
    • SQL Registration:
      • attach_notify(conn): Registers the notify() SQL scalar and the _honker_notifications table.
      • attach_honker_functions(conn): Registers all honker_* SQL scalars (queues, streams, scheduler, rate limits, locks, results).
      • bootstrap_honker_schema(conn): Runs idempotent DDL for all _honker_* tables.
    • Concurrency & I/O:
      • Writer / Readers: Provides a single-writer slot and a bounded reader pool with both blocking and non-blocking acquire methods.
      • SharedUpdateWatcher: A PRAGMA-polling thread per database that fans out updates to N subscribers.
    • Scheduling: cron::next_after_unix(expr, from_unix) is a 5-field crontab parser that calculates the next fire time with local-TZ and DST handling.
  3. Use typed JS wrappers in honker-node

    main

    The honker-node binding provides typed JavaScript wrappers for high-level Honker features. Use these wrappers instead of raw SQL for better type safety and developer experience. Supported wrappers include:

    • Queues: For enqueueclaimack workflows.
    • Streams
    • Locks
    • Pub/Sub: For updateEvents() and tx.notify() patterns.
    • Scheduler

    If you need to execute raw SQL, you can still use the db.query(...) or tx.query(...) methods.

  4. Honker Core Features

    main

    Honker provides several primitives for SQLite-backed applications:

    • Pub/Sub: Ephemeral notify() / listen() semantics.
    • Streams: Durable streams with per-consumer offsets.
    • Queues: At-least-once queues with retries, delayed jobs, priority, visibility timeouts, and dead-letter rows.
    • Scheduling: Time-trigger scheduling using cron or @every <duration> expressions.
    • Concurrency Control: Named locks and rate limits.
    • Transactional Outbox: Ability to enqueue work within the same transaction as business data updates.
  5. Integrate Honker with ORMs and Frameworks

    main

    Honker does not provide specific plugins for ORMs. To integrate, load the Honker extension on your existing ORM connection, execute honker_bootstrap(), and call the Honker SQL functions within your ORM's transaction blocks.

    Supported frameworks include:

    • SQLAlchemy, SQLModel, Django, Drizzle, Kysely, sqlx, GORM, ActiveRecord, Ecto, Hibernate, jOOJ, MyBatis, and Exposed.
  6. Install Honker via various package managers

    main

    Honker is available across multiple ecosystems. Use the following commands to install the respective bindings:

    • Python: pip install honker
    • Node.js: npm install @russellthehippo/honker-node
    • Ruby: gem install honker
    • .NET / C#: dotnet add package Honker
    pip install honker
    npm install @russellthehippo/honker-node
    gem install honker
    dotnet add package Honker
  7. Configure experimental watcher backends

    main

    By default, honker-rs uses polling. You can opt into experimental core backends using OpenOptions.

    Note: Explicitly requesting a backend that is not enabled via Cargo features will cause the application to fail loudly.

    • kernel: Requires the kernel-watcher feature.
    • shm: Requires the shm-fast-path feature and SQLite WAL mode.
    let opts = honker::OpenOptions::default().watcher_backend("kernel")?;
    let db = honker::Database::open_with_options("app.db", opts)?;
  8. Quick start with honker-rs queues

    main

    You can initialize a database, create a queue, and process jobs using the following pattern. Jobs are enqueued with JSON payloads and claimed by workers using a unique identifier.

    let db = honker::Database::open("app.db", "./libhonker_ext.dylib")?;
    let q = db.queue("emails");
    
    q.enqueue(serde_json::json!({ "to": "alice@example.com" }))?;
    
    if let Some(job) = q.claim_one("worker-1")? {
        send_email(&job.payload);
        job.ack()?;
    }
  9. Install dependencies for honker-cpp

    main

    Before building or using honker-cpp, ensure you have Zig 0.15+, a C++17 compiler, SQLite development headers, nlohmann-json headers, and the Honker SQLite extension installed. The C++ binding requires SQLite to be built with loadable extension support.

    # macOS with Homebrew
    brew install sqlite nlohmann-json
    
    # macOS with MacPorts
    sudo port install sqlite3 nlohmann-json
    
    # Ubuntu / Debian
    sudo apt-get install libsqlite3-dev nlohmann-json3-dev
    
    # Fedora
    sudo dnf install sqlite-devel json-devel
    
    # Arch
    sudo pacman -S sqlite nlohmann-json