failsafe 1.3.0

repository·master·Indexed 42 days ago

https://github.com/dmexe/failsafe-rs

A Rust circuit breaker implementation designed to detect failures and prevent them from recurring during system difficulties. It provides configurable failure detection policies, such as consecutive failures and success rate over time windows, along with various backoff strategies including constant, exponential, equal jittered, and full jittered. Supports both synchronous and asynchronous operations via the futures-support feature.

Tokens
7.5K
Snippets
28
Records
34
Agent score
49%

What's inside failsafe

  1. Failsafe features and requirements

    master

    Features

    • Async Support: Works with both Fn() -> Result and Future (enabled via the futures-support feature).
    • Backoff Strategies:
      • constant
      • exponential
      • equal_jittered
      • full_jittered
    • Failure Detection Policies:
      • consecutive_failures
      • success_rate_over_time_window

    Requirements

    • Minimum Rust version: 1.63
  2. Implement a custom FailurePredicate

    master

    A FailurePredicate<ERROR> is used to determine if a specific error should be recorded as a failure, which in turn increases the failure rate used by the failsafe mechanism.

    You can implement this in two ways:

    1. Using a closure or function: Any function with the signature Fn(&ERROR) -> bool automatically implements FailurePredicate.
    2. Implementing the trait: Define a struct and implement the is_err(&self, err: &ERROR) -> bool method.

    If you want every error to be treated as a failure regardless of its content, use the built-in Any struct.

    // Option 1: Using a closure
    let my_predicate = |err: &MyError| err.is_critical();
    
    // Option 2: Using the Any predicate (always returns true)
    let always_fail = Any;
    
    // Option 3: Implementing the trait manually
    struct MyPredicate;
    impl FailurePredicate<MyError> for MyPredicate {
        fn is_err(&self, err: &MyError) -> bool {
            err.code == 500
        }
    }
  3. How the CircuitBreaker future works

    master

    When you call .call() or .call_with(), you receive a ResponseFuture. This future manages the lifecycle of the wrapped operation:

    1. Permission Check: Upon the first poll, the ResponseFuture checks is_call_permitted(). If the circuit is open, it immediately returns Poll::Ready(Err(Error::Rejected)).
    2. Execution: If permitted, it polls the underlying TryFuture.
    3. Outcome Recording:
      • Success: If the inner future returns Ok, the circuit breaker records a success via on_success().
      • Failure: If the inner future returns Err and the provided FailurePredicate identifies it as a failure, the circuit breaker records a failure via on_error(). If the predicate identifies the error as non-critical, it records a success instead.

    This ensures that the circuit breaker's state (open/closed/half-open) is updated automatically based on the real-time results of the asynchronous tasks.

  4. How backoff strategies work as Iterators

    master

    All backoff strategies in failsafe implement the Iterator<Item = Duration> trait. This means you can consume them using standard iterator methods like .next(), .take(n), or .collect().

    Note that while the streams are conceptually infinite, the internal attempt counter increments up to a MAX_RETRIES limit (30), after which the delay calculation stabilizes (e.g., stays at max for exponential strategies).

    // Example of consuming a backoff strategy
    let backoff = exponential(Duration::from_secs(1), Duration::from_secs(60));
    for delay in backoff.take(5) {
        println!("Waiting for {:?}", delay);
    }
  5. Control time in tests using freeze() and MockClock

    master

    To test time-dependent logic deterministically, use the freeze function. freeze provides a MockClock instance to a closure and sets it as the active clock for the current thread. While inside the freeze scope, calls to failsafe::now() will return the time managed by the MockClock instead of the system time.

    Once the closure finishes (or panics), the clock is automatically reset to the system clock via a RAII guard.

    Note: You cannot nest freeze calls in the same thread; attempting to do so will trigger an assertion failure.

    // Example of how to use freeze to control time
    freeze(|clock| {
        let start = failsafe::now();
        
        // Advance the mock clock manually
        clock.advance(std::time::Duration::from_secs(10));
        
        let later = failsafe::now();
        assert_eq!(later.duration_since(start), std::time::Duration::from_secs(10));
    });
  6. Use the CircuitBreaker to prevent recurring failures

    master

    A CircuitBreaker detects failures and encapsulates the logic to prevent them from constantly recurring during maintenance or external system outages. You can use it to wrap functions that might fail. When the failure threshold is met, the circuit breaker enters an 'open' state and rejects subsequent calls with Error::Rejected until the backoff period expires.

    By default, Config::new().build() provides a circuit breaker with a reasonable default backoff and failure accrual policy.

    use rand::{thread_rng, Rng};
    use failsafe::{Config, CircuitBreaker, Error};
    
    // A function that sometimes fails.
    fn dangerous_call() -> Result<(), ()> {
      if thread_rng().gen_range(0..2) == 0 {
        return Err(())
      }
      Ok(())
    }
    
    // Create a circuit breaker with default backoff and failure accrual policy.
    let circuit_breaker = Config::new().build();
    
    // Call the function in a loop. After some iterations, the circuit breaker 
    // will be in an open state and reject next calls.
    for n in 0..100 {
      match circuit_breaker.call(|| dangerous_call()) {
        Err(Error::Inner(_)) => {
          eprintln!("{}: fail", n);
        },
        Err(Error::Rejected) => {
           eprintln!("{}: rejected", n);
           break;
        },
        _ => {}
      }
    }
  7. Configure custom backoff and failure policies

    master

    You can customize the circuit breaker's behavior by providing a specific failure_policy to the Config builder. A policy typically combines a failure detection strategy with a backoff strategy.

    • Backoff strategies: constant, exponential, equal_jittered, full_jittered.
    • Failure detection policies: consecutive_failures, success_rate_over_time_window.

    Example of configuring an exponential backoff and a consecutive failure policy:

    use std::time::Duration;
    use failsafe::{backoff, failure_policy, CircuitBreaker};
    
    // Create an exponential growth backoff which starts from 10s and ends with 60s.
    let backoff = backoff::exponential(Duration::from_secs(10), Duration::from_secs(60));
    
    // Create a policy which failed when three consecutive failures were made.
    let policy = failure_policy::consecutive_failures(3, backoff);
    
    // Creates a circuit breaker with given policy.
    let circuit_breaker = Config::new()
      .failure_policy(policy)
      .build();
  8. Use the default circuit breaker configuration

    master

    You can create a circuit breaker with reasonable default backoff and failure accrual policies using Config::new().build(). Use the .call() method to wrap functions that might fail. If the circuit breaker is in an 'open' state (preventing calls to protect the system), it will return Err(Error::Rejected). If the wrapped function fails, it returns Err(Error::Inner(e)).

    use failsafe::{Config, CircuitBreaker, Error};
    
    // A function that sometimes failed.
    fn dangerous_call() -> Result<(), ()> {
      if thread_rng().gen_range(0, 2) == 0 {
        return Err(())
      }
      Ok(())
    }
    
    // Create a circuit breaker which configured by reasonable default backoff and
    // failure accrual policy.
    let circuit_breaker = Config::new().build();
    
    // Call the function in a loop, after some iterations the circuit breaker will
    // be in a open state and reject next calls.
    for n in 0..100 {
      match circuit_breaker.call(|| dangerous_call()) {
        Err(Error::Inner(_)) => {
          eprintln!("{}: fail", n);
        },
        Err(Error::Rejected) => {
           eprintln!("{}: rejected", n);
           break;
        },
        _ => {}
      }
    }
  9. Combine multiple failure policies with or_else

    master

    The or_else method allows you to combine two FailurePolicy instances into a single OrElse policy. The combined policy triggers if either of the underlying policies determines the backend is dead.

    When mark_dead_on_failure() is called on an OrElse policy:

    1. Both underlying policies are updated.
    2. If both policies suggest a dead period, the one with the maximum Duration is returned.
    3. If only one policy suggests a dead period, that duration is returned.

    This is useful for creating complex detection logic, such as "fail if there are 5 consecutive errors OR if the success rate drops below 80% over 30 seconds".

  10. Configure a custom CircuitBreaker with backoff and failure policy

    master

    You can customize how a CircuitBreaker behaves by providing a custom FailurePolicy. A policy typically combines a failure detection strategy (like consecutive failures) with a backoff strategy (how long to wait before attempting to close the circuit again).

    Common components include:

    • backoff::exponential: Creates an exponential growth backoff starting from a minimum duration up to a maximum duration.
    • failure_policy::consecutive_failures: Creates a policy that triggers when a specific number of consecutive failures occur.
    use std::time::Duration;
    use failsafe::{backoff, failure_policy, Config, CircuitBreaker};
    
    fn circuit_breaker() -> impl CircuitBreaker {
      // Create an exponential growth backoff which starts from 10s and ends with 60s.
      let backoff = backoff::exponential(Duration::from_secs(10), Duration::from_secs(60));
    
      // Create a policy which failed when three consecutive failures were made.
      let policy = failure_policy::consecutive_failures(3, backoff);
    
      // Creates a circuit breaker with given policy.
      Config::new()
        .failure_policy(policy)
        .build()
    }
  11. Create an exponential backoff strategy

    master

    Use exponential(start: Duration, max: Duration) to create an infinite stream where the delay grows exponentially from start until it reaches max.

    Constraints:

    • start must be > 1s.
    • max must be > 1s.
    • max must be greater than or equal to start.
    use std::time::Duration;
    use failsafe::backoff::exponential;
    
    let backoff = exponential(Duration::from_secs(10), Duration::from_secs(100));
    // Returns 10s, 20s, 40s, 80s, 100s, 100s, ...