River

repository·master·Indexed 26 days ago

https://github.com/riverqueue/river

A high-performance job processing system for Go and Postgres. River leverages Postgres transactions to ensure atomic job enqueueing with application data changes. It features a flexible worker system using JobArgs and Worker structs, support for transactional inserts via InsertTx, and comprehensive client configuration for job lifecycle, retention, and queue-specific performance tuning.

Tokens
18.4K
Snippets
34
Records
142
Agent score
89%

What's inside river

  1. Understand the River job state machine

    master

    River jobs transition through several states during their lifecycle. Understanding these states helps in debugging job flows and managing job lifecycles (e.g., retries, cancellations, and manual interventions).

    Job States

    Initial/Waiting States:

    • Available: The job is ready to be fetched and run.
    • Scheduled: The job is waiting for a specific time to become available.
    • Pending: The job is waiting for preconditions to be met or for a future schedule.

    Intermediate States:

    • Running: The job is currently being executed by a worker.
    • Retryable: The job encountered an error and is waiting for its next retry attempt.

    Final States:

    • Completed: The job finished successfully.
    • Cancelled: The job was explicitly cancelled.
    • Discarded: The job failed too many times or was explicitly discarded.

    Key Transitions

    • Success/Failure: A Running job moves to Completed on success, or Retryable on error. If it exceeds error limits, it moves to Discarded.
    • Retries: Jobs in Retryable, Scheduled, Pending, Available, Running, Cancelled, or Discarded states can all be moved back to Available via a manual retry.
    • Cancellations: Jobs in Available, Running, Scheduled, Pending, or Retryable states can be moved to Cancelled via a manual cancel.
    • Rescuing: A Running job can be rescued into either Retryable or Discarded states.
  2. Start a River Client

    master

    A river.Client manages job processing and maintenance. To create one, provide a database driver (e.g., riverpgxv5.New(dbPool)), a river.Config containing your registered Workers, and queue configurations.

    To begin processing jobs, call riverClient.Start(ctx). The client will run inline and inherit the provided context.

    riverClient, err := river.NewClient(riverpgxv5.New(dbPool), &river.Config{
        Queues: map[string]river.QueueConfig{
            river.QueueDefault: {MaxWorkers: 100},
        },
        Workers: workers,
    })
    if err != nil {
        panic(err)
    }
    
    // Run the client inline. All executed jobs will inherit from ctx:
    if err := riverClient.Start(ctx); err != nil {
        panic(err)
    }
  3. Set up and migrate a development database

    master

    To run River programs locally (outside of tests), create a development database and run the migrate-up command. You can use the --database-url flag to specify the connection string and --line to specify the migration line.

    If migrations are long-running, you can override Postgres timeouts using the --statement-timeout root CLI flag.

  4. Update Go or toolchain versions in all go.mod files

    master
    To update the Go version or toolchain across the entire workspace, modify the go.work file with the desired go and/or toolchain directives, then run make update-mod-go to propagate these changes to all go.mod files.
    make update-mod-go
  5. Define Job Args and Workers

    master

    River jobs are defined using a pair of structs: one for arguments (JobArgs) and one for the processing logic (Worker).

    1. JobArgs: Must implement the JobArgs interface by providing a Kind() string method. Use json tags to define how the arguments are serialized to the database.
    2. Worker: Implements the Work method. It is recommended to embed river.WorkerDefaults[T] (where T is your args type) to simplify implementation.
    type SortArgs struct {
        // Strings is a slice of strings to sort.
        Strings []string `json:"strings"`
    }
    
    func (SortArgs) Kind() string { return "sort" }
    
    type SortWorker struct {
        // An embedded WorkerDefaults sets up default methods to fulfill the rest of
        // the Worker interface:
        river.WorkerDefaults[SortArgs]
    }
    
    func (w *SortWorker) Work(ctx context.Context, job *river.Job[SortArgs]) error {
        sort.Strings(job.Args.Strings)
        fmt.Printf("Sorted strings: %+v\n", job.Args.Strings)
        return nil
    }
  6. Stop a River Client gracefully

    master

    To shut down the client, you can either cancel the context passed to Start or call Stop(ctx) explicitly.

    When using context cancellation, it is recommended to use signal.NotifyContext for SIGINT/SIGTERM. You should also configure SoftStopTimeout in river.Config to allow active jobs time to finish. Use <-riverClient.Stopped() to wait for the client to fully shut down.

    riverClient, err := river.NewClient(riverpgxv5.New(dbPool), &river.Config{
        SoftStopTimeout: 10 * time.Second,
        ...
    })
    if err != nil {
        panic(err)
    }
    
    signalCtx, stop := signal.NotifyContext(ctx, syscall.SIGINT, syscall.SIGTERM)
    defer stop()
    
    // Stop fetching new work and wait for active jobs to finish. Cancel jobs after
    // SoftStopTimeout elapses.
    if err := riverClient.Start(signalCtx); err != nil {
        panic(err)
    }
    
    <-riverClient.Stopped()
  7. Register Workers

    master

    Before starting a client, you must register your workers so River can map job 'kinds' to the correct worker implementations. Use river.NewWorkers() to create a bundle and river.AddWorker to register them. Note that river.AddWorker will panic if the worker is already registered or invalid.

    workers := river.NewWorkers()
    // AddWorker panics if the worker is already registered or invalid:
    river.AddWorker(workers, &SortWorker{})
  8. Safely rename job kinds using JobArgsWithKindAliases

    master

    If you need to rename a job's Kind, you can implement the JobArgsWithKindAliases interface to prevent existing jobs in the database from being orphaned. This is a three-step process:

    1. Initial State: The struct implements Kind() returning the old_name.
    2. Transition State: Update Kind() to return the new_name and implement KindAliases() to return []string{"old_name"}. This allows workers to process both old and new job types.
    3. Final State: Once all jobs with the old name have been processed (including retries), remove the KindAliases() method so the worker only responds to the new_name.
    // Step 1: Original
    type jobArgsBeingRenamed struct{}
    func (a jobArgsBeingRenamed) Kind() string { return "old_name" }
    
    // Step 2: Transition
    type jobArgsBeingRenamed struct{}
    func (a jobArgsBeingRenamed) Kind() string          { return "new_name" }
    func (a jobArgsBeingRenamed) KindAliases() []string { return []string{"old_name"} }
    
    // Step 3: Final
    type jobArgsBeingRenamed struct{}
    func (a jobArgsBeingRenamed) Kind() string { return "new_name" }
  9. Initialize a River Pgx v5 Driver

    master

    To use River with a PostgreSQL database via the pgx/v5 driver, use the New function. It requires an existing *pgxpool.Pool.

    Important Notes:

    • The pool should be pre-configured to use the schema specified in your River client's Schema field.
    • The pool must remain open while any River objects (like clients or workers) are running.
    • If you pass nil as the dbPool, the driver will allow transactional inserts (InsertTx, InsertManyTx) which is useful for testing, but Start and non-transactional Insert/InsertMany calls will error.