DOCSEARCHbyAlgolia
Submit Docs

throttled-py

repository·main·Indexed 20 days ago

https://github.com/zhuozhuocrayon/throttled-py

A high-performance Python rate limiting library (v3.4.1) supporting multiple algorithms including Fixed Window, Sliding Window, Token Bucket, Leaky Bucket, and GCRA. It provides both synchronous and asynchronous APIs, a decorator for automatic enforcement, and supports In-Memory and Redis storage backends. Features include configurable quotas via readable strings, wait-and-retry behavior with timeouts, and optional integrations for FastAPI and OpenTelemetry.

Tokens
19.8K
Snippets
59
Records
93
Agent score
67%

What's inside throttled-py

  1. Introduction to throttled-py

    main

    Overview

    throttled-py is a high-performance Python rate limiting library designed for both synchronous and asynchronous (async/await) applications. It provides developers with multiple algorithms and storage backends to control request rates and prevent service overload.

    Key Features

    Supported Algorithms

    • Fixed Window: Simple window-based limiting.
    • Sliding Window: Smoother window transitions.
    • Token Bucket: Allows for bursts of traffic up to a defined bucket size.
    • Leaky Bucket: Smooths out traffic by processing requests at a constant rate.
    • Generic Cell Rate Algorithm (GCRA): A sophisticated algorithm often used in network traffic shaping.

    Storage Backends

    • In-Memory: High-performance local storage with support for key expiration and eviction.
    • Redis: Distributed storage suitable for multi-node/distributed systems.

    Usage Modes

    • Function Call: Explicitly calling the rate limiter within your logic.
    • Decorator: Using @ syntax to wrap functions with rate limiting logic.
    • Context Manager: Using with blocks to limit code execution within a specific scope.

    Response Strategies

    • Immediate Response: Returns a failure/status immediately if the limit is exceeded.
    • Wait-Retry: Pauses execution until the rate limit allows for the next request.

  2. Wait & Retry in Decorator and Context Manager modes

    main

    When using Throttled as a Decorator or a Context Manager with a timeout enabled, the library will attempt to retry. However, if the rate limit cannot be satisfied within the specified timeout period, a throttled.exceptions.LimitedError will be raised.

    from throttled import Throttled, per_sec, LimitedError

# Decorator mode
@Throttled(per_sec(2, burst=2), timeout=1.5)
def my_function():
    pass

try:
    my_function()
except LimitedError:
    # Raised if request is not allowed after the timeout
    pass

  4. How the Leaky Bucket (As a meter) algorithm works

    main

    Unlike the 'As a queue' implementation which requires background workers to process requests at a fixed rate, throttled-py implements the Leaky Bucket as a meter. This is a lightweight approach that does not require extra service processes or threads.

    Mechanism:

    1. Maintain a bucket with a maximum capacity and a token discard rate.
    2. Tokens are conceptually discarded at a preset rate.
    3. When a request arrives:
      • Calculate how many tokens should have been discarded since the last request based on the time elapsed.
      • Update the remaining token count.
      • Attempt to add a token for the current request.
      • If adding the token causes the bucket to exceed maximum capacity, the request is rejected.
      • Otherwise, the request is allowed.

    This approach provides the same rate-limiting benefits as the Token Bucket algorithm but is optimized for lightweight Python environments.

  5. Configure Quota and Rate limits

    main

    Rate limiting is configured using Quota and Rate objects.

    Rate

    Defines the base limit over a specific time window.

    • period (datetime.timedelta): The time duration for the limit.
    • limit (int): The maximum number of requests allowed within that period.

    Quota

    Defines the overall capacity, including burst support.

    • rate (Rate): The base Rate configuration.
    • burst (int, optional): Allows exceeding the rate limit momentarily. This is supported by Token Bucket, Leaky Bucket, and GCRA algorithms.

  6. Understand rate limiting algorithms in throttled-py

    main

    The throttled-py project implements various rate limiting (traffic control) algorithms to protect systems from excessive traffic. Understanding these algorithms helps you choose the right mechanism for your specific use case (e.g., smoothing traffic vs. handling bursts).

    Supported algorithm concepts include:

    • Fixed Window Counter: Divides time into fixed periods. Simple and memory-efficient, but can allow double the quota at window boundaries.
    • Sliding Window: Tracks individual request timestamps to ensure limits are respected within any moving window. More accurate but higher memory overhead.
    • Token Bucket: Tokens are added at a fixed rate to a bucket with a maximum capacity. Allows for bursts of traffic while maintaining a steady long-term rate.
    • Leaky Bucket (As a meter): A mirror of the Token Bucket. Instead of adding tokens, it 'discards' them at a fixed rate. Requests add tokens to the bucket; if the bucket overflows, the request is rejected. This implementation is lightweight and doesn't require background worker threads.
    • GCRA (Generic Cell Rate Algorithm): A highly efficient variant of the Leaky Bucket. It uses a single tat (theoretical arrival time) value to determine if a request can be allowed, making it more performant than standard Token Bucket implementations while remaining memory-friendly.

  7. Understand RateLimitResult and RateLimitState

    main

    When performing a limit operation on a specific key, the library returns a RateLimitResult object. This object tells you if the request was allowed and provides the current state of the rate limiter for that key.

    RateLimitResult

    • limited (bool): Indicates whether the current request was allowed to pass.
    • state (RateLimitState): The current state of the rate limiter for the given key.

    RateLimitState

    • limit (int): The maximum number of requests allowed in the initial state.
    • remaining (int): The maximum number of requests still allowed for the given key in the current state.
    • reset_after (float): Time in seconds required for the rate limiter to return to its initial state. In the initial state, limit equals remaining.
    • retry_after (float): The wait time in seconds before a rejected request can be retried. This is 0 when a request is allowed.

  9. Use In-Memory storage for single-process rate limiting

    main

    The MemoryStore (available as throttled.store.MemoryStore for sync or throttled.asyncio.store.MemoryStore for async) is a thread-safe, LRU-based cache with expiration. It is ideal for rate limiting within a single process.

    Key behaviors:

    • Default usage: The Throttled class automatically initializes a global MemoryStore with a maximum capacity of 1024. You usually do not need to create it manually.
    • Isolation: Synchronous and asynchronous usage use different global instances (implemented via threading.RLock and asyncio.Lock respectively).
    • Shared state: To limit the same key across different parts of your program, you must pass the same MemoryStore instance and the same Quota configuration to all Throttled instances.
    # Example of using MemoryStore for both 'ping' and 'pong' keys
from throttled import Throttled, MemoryStore, Quota

store = MemoryStore()
quota = Quota(limit=100, period=60)

throttled_ping = Throttled(store=store, quota=quota)
throttled_pong = Throttled(store=store, quota=quota)

  10. Configure In-Memory storage

    main

    By default, throttled-py uses a global MemoryStore with a capacity of 1024. You usually don't need to create one manually.

    Important: If you want to throttle the same key across different parts of your program, you must ensure they share the same MemoryStore instance. Different instances create different storage spaces.

    from throttled import Throttled, store

# Create a shared store
mem_store = store.MemoryStore()

@Throttled(key="ping-pong", quota="1/m", store=mem_store)
def ping() -> str: return "ping"

@Throttled(key="ping-pong", quota="1/m", store=mem_store)
def pong() -> str: return "pong"

ping()  # Success
pong()  # Raises LimitedError (because they share the same store and key)

  11. Monitor rate limiting with the Hook system

    main

    throttled-py provides observability through a flexible Hook system. Hooks allow you to monitor rate limiting events (such as allowed or denied requests) and integrate with external monitoring systems for metrics, alerting, and analytics.

    To implement observability, you can use built-in hooks or create custom ones to track the current state of your rate limiters.

  12. Wait & Retry in Function Call mode

    main

    When using Throttled in Function Call mode with a timeout enabled, the call will block (or await) and eventually return the last RateLimitResult obtained after the retries are completed.

    # Sync Function Call Example
from throttled import Throttled, per_sec

# Assuming throttled is configured with a timeout
result = throttled.call(my_function, arg1, arg2)
# Returns the last RateLimitResult after retries