DOCSEARCHbyAlgolia
Submit Docs

Safely Clojure Circuit-Breaker Library

repository·master·Indexed 42 days ago

https://github.com/brunobonacci/safely

A Clojure library for declarative handling of retries and failures when interacting with external systems. It provides a circuit-breaker mechanism to prevent overwhelming failing systems, supports multiple retry delay strategies including exponential backoff, and includes built-in logging and tracking. Version 1.0.0.

Tokens
6.6K
Snippets
23
Records
29
Agent score
53%

What's inside Safely

  1. How tracing levels (inner vs outer) work in safely

    master

    When using :track-as, safely produces a hierarchy of events using the μ/trace library:

    • Inner Events: One event is emitted for every attempt made to execute the inner block. These events include :safely/attempt <N> (e.g., :safely/attempt 0) and :safely/call-level :inner. If an attempt fails, the event includes :mulog/outcome :error and the :exception object.
    • Outer Event: One event is emitted for the entire safely block execution. It includes :safely/call-level :outer and represents the final :mulog/outcome (e.g., :ok if any attempt succeeded).

    This structure allows for distributed tracing where the outer call acts as a parent span to the individual attempt spans.

    (safely
  (some/function-call)
  :track-as ::some/function-call)

  2. Use `safely` with distributed platforms like Hadoop or Spark

    master
    Because the backoff logic in safely does not require coordination between different nodes, it is highly suitable for distributed platforms like Hadoop and Spark. When running ETL jobs that read or save data outside of HDFS or S3, wrapping those external calls in safely ensures job resilience against transient network or service errors across the distributed cluster.

  3. Basic usage of the `safely` function

    master

    To use the library, require the safely.core namespace and wrap your critical calls (e.g., API calls, database queries) in a safely block. You can define how to handle errors, how many times to retry, and what default value to return upon failure.

    (ns foo.bar
  (:require [safely.core :refer [safely]]))

(safely
  (api-call "other-system")

  :on-error
  :max-retries 5
  :default {:some :value})

  5. Configure automatic retries

    master

    To handle transitory failures (like network glitches), use the :on-error :max-retries <n> policy.

    • :max-retries 3 means a maximum of 4 total attempts will be made.
    • If the operation succeeds, the result is returned.
    • If all retries fail, safely will return the :default value if provided.
    • If no :default is provided, a clojure.lang.ExceptionInfo is thrown containing the number of attempts and the original cause.
    ;; Automatic retry with default value
(safely
  (http/get "http://user.service.local/users?active=true")
  :on-error
  :max-retries 3
  :default {:accounts [] :status "BUSY"})

  6. Enable tracing instrumentation with :track-as

    master

    You can enable event-based logging and tracing for a safely block by adding the :track-as option. This instruments the block at two levels:

    1. Inner block: The code expression being executed (which may run multiple times due to retries).
    2. Outer block: The overall execution of the safely block (runs once per evaluation).

    Each attempt generates an event with :safely/call-level :inner, and the overall result generates an event with :safely/call-level :outer. All events include :mulog/timestamp, :mulog/duration (in nanoseconds), and logical relationship IDs for tracing.

    (safely
  (users/load-user :id "d548a66b")
  :on-error
  :max-retries 3
  :default     nil
  :track-as    ::load-user)

  8. Run tests in sleepless-mode

    master

    When writing automated tests, you can skip the actual sleep/wait times of the retry logic by binding safely.core/*sleepless-mode* to true. This allows the code to execute the same retry logic and paths without the actual delay.

    (binding [safely.core/*sleepless-mode* true]
  (safely
    (slurp "/not/existing/file")
    :on-error
    :max-retries 5
    :default ""))

  9. Handle circuit breaker exceptions with HTTP 429

    master

    When a safely circuit breaker is triggered (e.g., the circuit is open or the queue is full), it raises an exception. To provide a clean API response, you should catch these exceptions and return an appropriate HTTP status code.

    According to RFC6585, HTTP 429 Too Many Requests is the recommended status code to inform callers that they should slow down and retry later. You can identify if an exception was triggered by safely by checking if the :origin in the exception data matches :safely.core/circuit-breaker.

    (defn wrap-circuit-breaker [handler]
  (fn [r]
    (try
      (handler r)
      (catch Exception x
        (if (= :safely.core/circuit-breaker
               (:origin (ex-data (.getCause x))))
          {:status 429 :body "Please try later"}
          (throw x))))))

  10. Use `safely` for resilient ETL-loading jobs

    master

    In ETL (Extract, Transform, Load) contexts where a database might be temporarily unavailable or might throttle requests due to quota limits, safely provides a mechanism to wrap operations with automatic retries. It uses a randomized exponential backoff strategy by default, which allows the application to adapt to changing system capacity (like DynamoDB scaling events) without manual coordination. This is particularly useful for handling ProvisionedThroughputExceededException or transient HTTP 500 errors.

    (defn save-record
  """takes a single record as a Clojure map and stores it into a dynamo-db
  table. In case of failures it retries until successful completion."""
  [{:keys [table-name region] :as cfg} rec]
  (safely

   (dyn/put-item
    {:endpoint region}
    :table-name table-name
    :return-consumed-capacity "TOTAL"
    :return-item-collection-metrics "SIZE"
    :item rec})

   :on-error
   :max-retries :forever))

  11. Configure Zipkin for distributed tracing

    master

    Since safely uses μ/trace, it is compatible with OpenZipkin. You can send safely events as traces to a Zipkin instance by starting the Zipkin publisher via μ/start-publisher!. Each attempt will appear as a span, and the outer call will appear as the top-level span.

    (μ/start-publisher!
  {:type :zipkin
   :url  "http://localhost:9411/"})

  12. Use the Circuit Breaker

    master

    To activate the circuit breaker, add the :circuit-breaker <name> option. This creates a dedicated thread pool and state machine for the named operation.

    States:

    • :closed: Normal operation. Requests pass through to a thread pool.
    • :open: The circuit is tripped due to failures. All requests are rejected immediately with a :circuit-open error. It stays open for a :grace-period.
    • :half-open: A testing state where a limited number of requests are allowed through to check if the system has recovered. The strategy for this is controlled by :half-open-strategy (e.g., :linear-ramp-up).

    Important: Do not use high-cardinality values for the circuit breaker name, as each unique name allocates a new thread pool and state machine.

    ;; activating circuit breaker
(safely
  (http/get "http://user.service.local/users?active=true")
  :on-error
  :circuit-breaker :fetch-active-users
  :timeout 30000)