DOCSEARCHbyAlgolia
Submit Docs

python-redis-lock

repository·master·Indexed 20 days ago

https://github.com/ionelmc/python-redis-lock

A distributed lock implementation for Python using Redis SETNX/BLPOP. It provides a thread-lock-compatible interface and context manager to manage concurrency across multiple processes or machines. Features include support for blocking and non-blocking acquisition, lock timeouts, auto-renewal to prevent stale locks, and a specialized RedisCache backend for Django to mitigate the dogpile effect.

Tokens
6.2K
Snippets
26
Records
30
Agent score
69%

What's inside python-redis-lock

  1. Understand benchmark implementation and metrics

    master

    The benchmark script evaluates lock performance based on the following parameters:

    • Implementation: The locking mechanism used (redis_lock vs native).
    • Lock duration: The artificial amount of time the script sleeps while holding the lock before releasing it.
    • Concurrency: The number of processes simultaneously attempting to acquire the same lock.
    • Acquires: The total number of successful lock acquisitions recorded.
    • Avg/Min/Max: The average, minimum, and maximum number of acquisitions per process.

    Key Performance Observations

    • Single Client (No Contention): The redis_lock implementation is slightly slower than the native implementation due to overhead in the lock releasing script.
    • Two Clients (Contention): The native implementation loses throughput because its acquiring routine uses a while True: sleep(0.1) loop. It also tends to favor the first client, as the waiting client's sleep interval is relatively large.
    • High Concurrency or High Duration: Under high contention or long lock durations, the native implementation becomes unpredictable. Some clients may fail to acquire the lock entirely (indicated by a Min value of 0), while others may acquire it disproportionately often (indicated by a high Max value).

  2. How the lock implementation works

    master

    The library implements locks using two Redis keys for every lock name <name>:

    1. lock:<name>: A string value representing the actual lock.
    2. lock-signal:<name>: A list value used for signaling waiters when the lock is released.

    This mechanism avoids spinloops during the acquire phase.

  4. Use locks as context managers

    master

    The recommended way to use locks is via a context manager (with statement). This ensures the lock is automatically released when the block is exited.

    By default, the context manager is blocking. If you pass blocking=False to the Lock constructor, it will raise a NotAcquired exception if the lock cannot be acquired immediately.

    import redis_lock
from redis import StrictRedis
import time

# Standard blocking context manager
conn = StrictRedis()
with redis_lock.Lock(conn, "name-of-the-lock"):
    print("Got the lock. Doing some work ...")
    time.sleep(5)

# Non-blocking context manager (raises NotAcquired if lock is held)
with redis_lock.Lock(conn, "name-of-the-lock", blocking=False):
    print("Got the lock. Doing some work ...")
    time.sleep(5)

  5. Handle stale locks with auto_renewal and expire

    master

    If a process crashes or a server blackouts, a lock might remain in Redis indefinitely. To prevent this, use the expire parameter to set a timeout and auto_renewal=True to allow the lock to stay active as long as your Python process is running and executing the code within the context manager.

    # Get a lock with a 60-second lifetime but keep renewing it automatically
# to ensure the lock is held for as long as the Python process is running.
with redis_lock.Lock(conn, name='my-lock', expire=60, auto_renewal=True):
    # Do work....
    pass

  6. Integrate with Django to avoid the dogpile effect

    master

    To prevent cache stampedes (the dogpile effect) in Django, you can use the redis_lock.django_cache.RedisCache backend. This backend adds a .lock(name, expire=None) method to the Django cache interface.

    1. Install the Django extra: pip install "python-redis-lock[django]".
    2. Configure your CACHES setting to use the RedisCache backend.
    3. Use cache.lock(key) in your application logic.
    pip install "python-redis-lock[django]"
    # settings.py
CACHES = {
    'default': {
        'BACKEND': 'redis_lock.django_cache.RedisCache',
        'LOCATION': 'redis://127.0.0.1:6379/1',
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient'
        }
    }
}
    # usage.py
from django.core.cache import cache

def function():
    val = cache.get(key)
    if not val:
        with cache.lock(key):
            val = cache.get(key)
            if not val:
                # DO EXPENSIVE WORK
                val = ...
                cache.set(key, value)
    return val

  7. Basic usage of redis_lock.Lock

    master

    To use python-redis-lock, create a Lock instance by providing a Redis connection and a unique name for the lock. The interface is designed to be compatible with threading.Lock. You can attempt to acquire the lock without blocking by setting blocking=False.

    from redis import Redis
import redis_lock

conn = Redis()
lock = redis_lock.Lock(conn, "name-of-the-lock")

if lock.acquire(blocking=False):
    print("Got the lock.")
    lock.release()
else:
    print("Someone else has the lock.")

  8. Release a lock using a specific ID

    master

    If you need to ensure that a release operation matches a specific lock instance, you can pass the id of the original lock to a new Lock instance. This is useful when the lock state is not strictly managed within a single scope.

    import redis_lock
from redis import StrictRedis

conn = StrictRedis()
lock1 = redis_lock.Lock(conn, "foo")
lock1.acquire()

# Create a second lock object with the same name and ID to release the first one
lock2 = redis_lock.Lock(conn, "foo", id=lock1.id)
lock2.release()

  9. Use redis_lock.Lock as a context manager

    master

    The Lock class supports the context manager protocol, allowing you to use the with statement. This ensures the lock is automatically released when the block is exited.

    import time
from redis import StrictRedis
import redis_lock

conn = StrictRedis()
with redis_lock.Lock(conn, "name-of-the-lock"):
    print("Got the lock. Doing some work ...")
    time.sleep(5)

  10. Acquire a non-blocking lock

    master

    To attempt to acquire a lock without waiting, call .acquire(blocking=False). This will immediately return True if the lock was acquired, or False if it is currently held by another process.

    import redis_lock
from redis import StrictRedis
import time

conn = StrictRedis()
lock = redis_lock.Lock(conn, "name-of-the-lock")
if lock.acquire(blocking=False):
    print("Got the lock. Doing some work ...")
    time.sleep(5)
else:
    print("Someone else has the lock.")

  11. Run performance benchmarks locally

    master

    You can run the built-in benchmarks to compare the redis_lock implementation against the native implementation.

    Prerequisites:

    • A running Redis server on the default port.

    Warning: Running these benchmarks will cause the Redis database to lose all its current data.

    Execution: Use tox to run the benchmark script. The argument passed to examples/bench.py determines the duration of the benchmark in seconds.

    tox -e py38-dj3-cover -- python examples/bench.py 10

  12. Acquire a blocking lock

    master

    To acquire a lock that blocks until it is available, instantiate redis_lock.Lock with a Redis connection and a lock name, then call .acquire(). By default, this will wait until the lock is acquired.

    import redis_lock
from redis import StrictRedis
import time

conn = StrictRedis()
lock = redis_lock.Lock(conn, "name-of-the-lock")
if lock.acquire():
    print("Got the lock. Doing some work ...")
    time.sleep(5)