fang

repository·master·Indexed 20 days ago

https://github.com/ayrat555/fang

A Rust background job processing library that uses SQL databases (PostgreSQL, SQLite, or MySQL) as task queues. It supports both async (Tokio-based) and threaded worker models, featuring CRON scheduling, task retries with customizable backoff, unique task enforcement, and single-purpose workers.

Tokens
13.8K
Snippets
39
Records
58
Agent score
69%

What's inside fang

  1. Overview of Fang background task processing

    master

    Fang is a background task processing library for Rust that utilizes a PostgreSQL database as its task queue. It provides several key capabilities for managing asynchronous workloads:

    • Worker Types: Supports both threaded workers (running in OS threads) and async workers (running in Tokio tasks).
    • Scheduling: Supports one-off scheduled tasks (at a specific future time) and periodic tasks using CRON expressions.
    • Task Uniqueness: Prevents task duplication in the queue if tasks are marked as unique.
    • Specialized Workers: While all tasks are stored in a single table, workers can be configured to execute only specific task types.
    • Reliability: Supports task retries with custom backoff modes.
  2. Overview of Fang

    master
    Fang is a background task processing library for Rust. It utilizes PostgreSQL, SQLite, or MySQL as an asynchronous task queue. It supports both async (Tokio-based) and threaded workers, scheduled tasks, periodic (CRON) tasks, unique task enforcement, single-purpose workers, and customizable retry backoff modes.
  3. When to use Async Fang vs Threaded Fang

    master

    Fang supports two modes of operation. Choosing the right one depends on your workload:

    Use Async Fang when:

    • You have a large number of IO-bound tasks (e.g., database queries, network requests).
    • You want to minimize CPU and memory overhead.
    • Tasks are lightweight.

    Use Threaded Fang when:

    • You have heavy, CPU-bound tasks.
    • You want to avoid overloading a single tokio runtime.
    • You require features like graceful shutdown (currently implemented for threaded processing).

    Tip: If you must use async for heavy tasks, consider starting a separate tokio runtime specifically for Fang workers to isolate them from your main application runtime.

  4. Deduplicate tasks using uniq() and uniq_hash

    master

    Fang supports task deduplication based on the task's metadata.

    1. Implementation: In your Runnable or AsyncRunnable implementation, return true for the uniq() method to enable deduplication for that task type.
    2. Mechanism: Fang calculates a uniq_hash from the task's JSON metadata. If a task with the same uniq_hash is enqueued, it can be deduplicated based on this field in the fang_tasks table.

    This is useful for preventing the same logical task from being queued multiple times simultaneously.

    fn uniq(&self) -> bool {
        true
    }
  5. Enqueue tasks in blocking and async modes

    master

    Blocking Mode

    Use Queue::enqueue_task (or insert_task as shown in examples) after building a queue with a connection pool.

    let queue = Queue::builder().connection_pool(pool).build();
    let task_inserted = queue.insert_task(&MyTask::new(1)).unwrap();

    Async Mode

    Use AsyncQueue::builder() to configure the connection (e.g., via a Postgres URI) and pool size. You must call .connect().await before performing operations.

    To enqueue, use AsyncQueueable::insert_task passing the task as &dyn AsyncRunnable.

    // Blocking
    let queue = Queue::builder().connection_pool(pool).build();
    let task_inserted = queue.insert_task(&MyTask::new(1)).unwrap();
    
    // Async
    let mut queue = AsyncQueue::builder()
        .uri("postgres://postgres:postgres@localhost/fang")
        .max_pool_size(2)
        .build();
    queue.connect().await.unwrap();
    
    let task = AsyncTask { 8 };
    let task_returned = queue
        .insert_task(&task as &dyn AsyncRunnable)
        .await
        .unwrap();
  6. Run Fang migrations

    master

    Before using Fang, you must create the fang_tasks table in your database. You can do this by running the provided migration scripts manually or by running them in your code. To run migrations in code, include the migrations-{database} feature (e.g., migrations-postgres).

    # Example for PostgreSQL
    [dependencies]
    fang = { version = "0.11.0" , features = ["asynk-postgres", "migrations-postgres" ], default-features = false }
    use fang::run_migrations_postgres;
    
    // Assuming `connection` is your database connection
    run_migrations_postgres(&mut connection).unwrap();
  7. Implement async background processing with Fang

    master

    Fang provides an async background processing framework for Rust using tokio and postgres. It is designed for lightweight, IO-bound tasks, allowing you to run many tasks with low CPU and memory overhead by using tokio tasks instead of OS threads.

    Core Workflow

    1. Define a task: Create a struct and derive Serialize and Deserialize using fang::serde.
    2. Implement AsyncRunnable: Use the #[async_trait] and #[typetag::serde] macros to define the task's execution logic.
    3. Initialize the queue: Create an AsyncQueue using a connection URI (e.g., Postgres).
    4. Start workers: Use an AsyncWorkerPool to manage and execute tasks.
    5. Enqueue tasks: Insert tasks into the queue using insert_task.
    use fang::serde::{Deserialize, Serialize};
    use fang::async_trait;
    use fang::typetag;
    use fang::AsyncRunnable;
    
    #[derive(Serialize, Deserialize)]
    #[serde(crate = "fang::serde")]
    pub struct MyTask {
        pub number: u16,
    }
    
    #[async_trait]
    #[typetag::serde]
    impl AsyncRunnable for MyTask {
        async fn run(&self, queue: &mut dyn AsyncQueueable) -> Result<(), Error> {
            // Task logic here
            Ok(())
        }
    }
  8. Install Fang with Blocking feature

    master

    To use Fang with blocking (threaded) workers, add the following to your Cargo.toml. Note that default-features is set to false to avoid pulling in unnecessary async dependencies.

    [dependencies]
    fang = { version = "0.11.0" , features = ["blocking"], default-features = false }
  9. Schedule periodic or one-time tasks using CRON

    master

    In Fang 0.9+, you can schedule tasks by implementing the cron method within your Runnable (for blocking workers) or AsyncRunnable (for asynk workers) trait implementation. You must use the fang::Scheduled enum to define the schedule.

    CRON Patterns

    To execute a task periodically, return Scheduled::CronPattern with a valid cron expression. For example, to run a task every 20 seconds, use the expression "0/20 * * * * * *".

    One-time Scheduling

    To schedule a task to run once at a specific time in the future, return Scheduled::ScheduleOnce containing a DateTime<Utc> value.

    Note: You do not need to start a separate scheduler process; WorkerPool or AsyncWorkerPool will automatically handle re-scheduling periodic tasks.

    // Example: Periodic CRON task
    impl AsyncRunnable for MyCronTask {
      async fn run(&self, _queue: &mut dyn AsyncQueueable) -> Result<(), Error> {
        log::info!("CRON!!!!!!!!!!!!!!!",);
        Ok(())
      }
    
      fn cron(&self) -> Option<Scheduled> {
        // cron expression to execute a task every 20 seconds.
        let expression = "0/20 * * * * * *";
        Some(Scheduled::CronPattern(expression.to_string()))
      }
    
      fn uniq(&self) -> bool {
        true
      }
    }
    
    // Example: One-time scheduled task
    impl AsyncRunnable for MyCronTask {
      async fn run(&self, _queue: &mut dyn AsyncQueueable) -> Result<(), Error> {
        log::info!("CRON!!!!!!!!!!!!!!!",);
        Ok(())
      }
    
      fn cron(&self) -> Option<Scheduled> {
        // Schedules the task for 7 seconds in the future
        Some(Scheduled::ScheduleOnce(Utc::now() + Duration::seconds(7i64)))
      }
    
      fn uniq(&self) -> bool {
        true
      }
    }
  10. Install Fang with Async features and derive macro

    master

    To use the derive-error macro alongside an async backend, use the following pattern in your Cargo.toml, replacing {database} with postgres, sqlite, or mysql.

    [dependencies]
    fang = { version = "0.11.0" , features = ["asynk-{database}", "derive-error" ], default-features = false }
  11. Install Fang with Async features

    master

    Fang supports asynchronous workers using tokio. You must select a database backend feature. Use default-features = false to ensure only the selected backend is included.

    # PostgreSQL as a queue
    [dependencies]
    fang = { version = "0.11.0" , features = ["asynk-postgres"], default-features = false }
    
    # SQLite as a queue
    [dependencies]
    fang = { version = "0.11.0" , features = ["asynk-sqlite"], default-features = false }
    
    # MySQL as a queue
    [dependencies]
    fang = { version = "0.11.0" , features = ["asynk-mysql"], default-features = false }
  12. Start workers in blocking and async modes

    master

    Blocking Workers

    Workers run in separate threads and are automatically restarted on panic. Use WorkerPool::<Queue>::builder() to configure the pool.

    Async Workers

    Workers run as tokio tasks and are automatically restarted on panic. Use AsyncWorkerPool::<AsyncQueue>::builder() to configure the pool.

    Both builders allow filtering by task_type to only process specific kinds of tasks.

    // Blocking
    let mut worker_pool = WorkerPool::<Queue>::builder()
        .queue(queue)
        .number_of_workers(3_u32)
        .task_type("my_task_type")
        .build();
    worker_pool.start();
    
    // Async
    let mut pool: AsyncWorkerPool<AsyncQueue> = AsyncWorkerPool::builder()
            .number_of_workers(max_pool_size)
            .queue(queue.clone())
            .task_type("my_task_type")
            .build();
    pool.start().await;