pottery

repository·master·Indexed 22 days ago

https://github.com/brainix/pottery

A Pythonic interface for Redis that allows developers to use Redis data structures as standard Python collections. It provides Redis-backed containers including RedisDict, RedisSet, RedisList, RedisCounter, and RedisDeque, as well as resilience patterns such as Redlock for distributed locking, AIORedlock for asyncio, NextID for distributed ID generation, and probabilistic data structures like BloomFilter and HyperLogLog. It also includes a redis_cache decorator for memoization and CachedOrderedDict for ordered cached lookups.

Tokens
3.7K
Snippets
12
Records
17
Agent score
29%

What's inside pottery

  1. Use Redlock for distributed locking

    master

    Redlock provides a safe and reliable lock to coordinate access to resources shared across threads, processes, and machines using Redis. It implements the Python threading.Lock API.

    Best Practices:

    • Instantiate per use: It is safest to instantiate a new Redlock object every time you need to protect a resource rather than sharing instances. Use the key to identify the resource.
    • Avoid single points of failure: In production, use 5 Redis masters so the lock remains available even if 2 masters are lost.
    • Manage timeouts: Locks are automatically released to prevent deadlocks. Ensure your critical section completes well within the auto_release_time (default is 10 seconds).
  2. Use HyperLogLog for cardinality estimation

    master

    A HyperLogLog is a probabilistic data structure used to estimate the number of distinct elements (cardinality) in a large dataset using minimal storage (approx. 1.5 KB).

    Key Characteristics:

    • Probabilistic Accuracy: It is accurate within a margin of error of up to 2%.
    • Use Case: Best for answering "How many unique items have I seen?" rather than "What are the items?".
    • Membership Testing: While not designed for it, you can perform membership testing (element in hll). This is probabilistic: it may return false positives, but it will never return false negatives (if it returns False, the element definitely was not seen).
    • Limitations: Elements must be JSON serializable.

    Common Operations:

    • add(element): Inserts a single element.
    • update(elements): Inserts multiple elements.
    • len(hll): Returns the estimated cardinality.
    • contains_many(*elements): Performs efficient membership testing for multiple elements, returning a tuple of booleans.
    • clear(): Removes all elements.
  3. Initialize a Redis client for Pottery

    master

    Before using Pottery containers, you must set up a standard Redis client (e.g., using the redis-py library).

    from redis import Redis
    redis = Redis.from_url('redis://localhost:6379/1')
  4. Use RedisList for Redis-backed lists

    master

    RedisList is a Redis-backed container compatible with Python's list.

    Initialization:

    • redis: Your Redis client instance.
    • key: The Redis key name for the list.

    Limitations:

    • Elements must be JSON serializable.
    • Performance Note: Under the hood, Redis implements lists as doubly linked lists. While inserting at the head or tail is fast (O(1)), accessing elements by index is slow (O(n)). For better performance with index-based access, consider using RedisDeque instead.
    from pottery import RedisList
    squares = RedisList([1, 4, 9, 16, 25], redis=redis, key='squares')
    
    # Standard list operations
    print(squares[0])       # 1
    print(squares[-1])      # 25
    squares.append(36)
    from pottery import RedisList
    squares = RedisList([1, 4, 9, 16, 25], redis=redis, key='squares')
  5. Use RedisCounter for Redis-backed counters

    master

    RedisCounter is a Redis-backed container compatible with Python's collections.Counter.

    Initialization:

    • redis: Your Redis client instance.
    • key: The Redis key name for the counter.
    • Optional keyword arguments (e.g., a=4, b=2) to initialize counts.

    Limitations:

    • Keys must be JSON serializable.
    from pottery import RedisCounter
    from collections import Counter
    
    c = RedisCounter(redis=redis, key='my-counter', a=4, b=2, c=0, d=-2)
    
    # Standard Counter operations
    print(c.most_common(3))
    
    # Subtracting another Counter
    d = Counter(a=1, b=2, c=3, d=4)
    c.subtract(d)
    # c is now RedisCounter{'a': 3, 'b': 0, 'c': -3, 'd': -6}
    from pottery import RedisCounter
    c = RedisCounter(redis=redis, key='my-counter', a=4, b=2, c=0, d=-2)
  6. Measure elapsed time with ContextTimer

    master

    ContextTimer is a utility to measure wall (real-world) time. The .elapsed() method returns the time in milliseconds.

    You can use it in two ways:

    1. Stand-alone: Manually call .start() and .stop().
    2. Context Manager: Use the with statement to automatically manage the timer lifecycle.
    import time
    from pottery import ContextTimer
    
    # Stand-alone usage
    timer = ContextTimer()
    timer.start()
    time.sleep(0.1)
    print(timer.elapsed()) # Returns time in ms
    timer.stop()
    
    # Context manager usage
    with ContextTimer() as timer:
        time.sleep(0.1)
        print(timer.elapsed())
  7. Use RedisDeque for Redis-backed deques

    master

    RedisDeque is a Redis-backed container compatible with Python's collections.deque. It is optimized for fast head/tail operations.

    Initialization:

    • redis: Your Redis client instance.
    • key: The Redis key name for the deque.

    Limitations:

    • Elements must be JSON serializable.
    from pottery import RedisDeque
    d = RedisDeque('ghi', redis=redis, key='letters')
    
    # Standard deque operations
    d.append('j')
    d.appendleft('f')
    d.pop()
    d.rotate(1)
    d.extendleft('abc')
    from pottery import RedisDeque
    d = RedisDeque('ghi', redis=redis, key='letters')
  8. Use RedisSet for Redis-backed sets

    master

    RedisSet is a Redis-backed container compatible with Python's set.

    Initialization:

    • redis: Your Redis client instance.
    • key: The Redis key name for the set.

    Limitations:

    • Elements must be JSON serializable.

    Efficient Membership Testing: You can use .contains_many(*elements) to perform efficient membership testing for multiple elements at once, returning a tuple of booleans.

    from pottery import RedisSet
    basket = RedisSet({'apple', 'orange', 'pear'}, redis=redis, key='basket')
    
    # Standard set operations
    print('orange' in basket)
    print(sorted(basket))
    
    # Bulk membership check
    nirvana = RedisSet({'kurt', 'krist', 'dave'}, redis=redis, key='nirvana')
    print(nirvana.contains_many('kurt', 'krist', 'chat', 'dave'))
    # Output: (True, True, False, True)
  9. Use CachedOrderedDict for ordered, cached lookups

    master

    CachedOrderedDict extends Python's collections.OrderedDict by adding a Redis-backed cache. It is ideal for scenarios where you have an ordered list of IDs (e.g., from a search engine) that need to be hydrated into full objects and cached for future use.

    Usage Pattern

    1. Instantiate with redis_client, a redis_key, and dict_keys (an ordered iterable of keys to track).
    2. Check for misses() to see which keys haven't been hydrated yet.
    3. Populate the cache by assigning values to the missing keys.
    from pottery import CachedOrderedDict
    
    # Initialize with keys to be tracked
    search_results = CachedOrderedDict(
        redis_client=redis,
        redis_key='search-results',
        dict_keys=(1, 2, 3, 4, 5),
    )
    
    # Check for keys that need hydration
    print(search_results.misses())
    
    # Populate the cache
    search_results[1] = 'one'
    search_results[2] = 'two'
    
    # All keys are now cached
    print(search_results.misses())

    Key Methods

    • misses(): Returns an iterable of keys that are currently not in the cache.
    • items(): Iterates over the key-value pairs, preserving the order of dict_keys.

    Limitations

    • Keys and values must be JSON serializable.
    search_results_1 = CachedOrderedDict(
        redis_client=redis,
        redis_key='search-results',
        dict_keys=(1, 2, 3, 4, 5),
    )
  10. Use RedisDict for Redis-backed dictionaries

    master

    RedisDict is a Redis-backed container compatible with Python's dict. It allows you to interact with Redis keys as if they were local dictionaries.

    Initialization:

    • redis: Your Redis client instance.
    • key: The Redis key name for the dictionary.

    Limitations:

    • Keys and values must be JSON serializable.
    from pottery import RedisDict
    # Initialize with existing data
    tel = RedisDict({'jack': 4098, 'sape': 4139}, redis=redis, key='tel')
    
    # Standard dict operations
    tel['guido'] = 4127
    print(tel['jack'])  # 4098
    del tel['sape']
    print('guido' in tel)  # True
    from pottery import RedisDict
    tel = RedisDict({'jack': 4098, 'sape': 4139}, redis=redis, key='tel')
    tel['guido'] = 4127
  11. Implement probabilistic membership testing with BloomFilter

    master

    A BloomFilter is a probabilistic data structure used to test if an element is a member of a set. It is highly space-efficient but allows for false positives (reporting an element is present when it is not). It never produces false negatives.

    Usage

    Initialize with the expected number of elements and your desired false positive probability.

    from pottery import BloomFilter
    
    dilberts = BloomFilter(
        num_elements=100,
        false_positives=0.01,
        redis=redis,
        key='dilberts',
    )
    
    # Add elements
    dilberts.add('rajiv')
    dilberts.update({'raj', 'dan'})
    
    # Test membership
    print('rajiv' in dilberts)  # True (or potentially True)
    print('raj' in dilberts)    # False (guaranteed)
    
    # Batch membership testing
    print(dilberts.contains_many('rajiv', 'raj', 'dan', 'luis'))
    
    # Get approximate count
    print(len(dilberts))
    
    # Clear the filter
    dilberts.clear()

    Key Methods

    • add(element): Inserts an element.
    • update(iterable): Inserts multiple elements.
    • contains_many(*elements): Performs efficient batch membership testing.
    • clear(): Removes all elements.
    • __len__(): Returns an approximation of the number of elements inserted.

    Limitations

    • Elements must be JSON serializable.
    • len(bf) and membership tests are probabilistic. Accuracy is controlled by num_elements and false_positives during initialization.
    dilberts = BloomFilter(
        num_elements=100,
        false_positives=0.01,
        redis=redis,
        key='dilberts',
    )