redis-py

repository·master·Indexed 11 days ago

https://github.com/redis/redis-py

The official Python interface for the Redis key-value store. It provides support for standard commands, connection pooling, the RESP3 protocol, and various deployment modes including Redis Cluster. Features include pipeline batching, PubSub support, OpenTelemetry instrumentation, and specialized clients for Active-Active setups and bulk hash ingestion via HIMPORT.

Tokens
79.5K
Snippets
276
Records
340
Agent score
96%

What's inside redis-py

  1. Understand the Response Mode Matrix

    master

    The Python response shape depends on both the protocol and the legacy_responses setting. Note that decode_responses is independent and still controls bulk-string decoding.

    Client optionsWire protocolPython response shape
    Redis()Default RESP3 wireLegacy RESP2-compatible shape
    Redis(protocol=2)RESP2Legacy RESP2 shape
    Redis(protocol=3)RESP3Native RESP3 shape
    Redis(legacy_responses=False)Default RESP3 wireUnified shape
    Redis(protocol=2, legacy_responses=False)RESP2Unified shape
    Redis(protocol=3, legacy_responses=False)RESP3Unified shape
  2. Strategies for reading PubSub messages

    master

    There are three main ways to consume messages from a PubSub instance:

    1. get_message() (Polling): Uses the system's select module to poll the socket. It returns the message if data is available, or None immediately if no data is present. This is ideal for integration into an existing event loop.

      while True:
          message = p.get_message()
          if message:
              # do something
          time.sleep(0.001)
    2. listen() (Blocking Generator): A generator that blocks until a message is available. Use this if your application's primary purpose is to act on Redis messages.

      for message in p.listen():
          # do something
    3. run_in_thread() (Background Thread): Starts a separate thread running a non-blocking event loop. This is a wrapper around get_message().

      • Note: You must have registered message handlers for all subscribed channels/patterns, otherwise run_in_thread() will prevent execution.
      • Stopping: Use the returned thread object's .stop() method to shut it down.
      • Exception Handling: You can pass an exception_handler function that accepts (exception, pubsub, thread) arguments.
      thread = p.run_in_thread(sleep_time=0.001)
      # ... later ...
      thread.stop()
    # Example: Running in a background thread with an exception handler
    def exception_handler(ex, pubsub, thread):
        print(ex)
        thread.stop()
    
    p.subscribe(**{'my-channel': my_handler})
    thread = p.run_in_thread(exception_handler=exception_handler)
  3. Understand Redis Keyspace and Keyevent Notifications

    master

    Redis provides two primary ways to listen for changes to keys using PubSub channels:

    • Keyspace Notifications: Use the prefix __keyspace@<db>__:<keyname> to listen for operations performed on a specific key. The notification payload contains the operation performed (e.g., set, del).
    • Keyevent Notifications: Use the prefix __keyevent@<db>:<event> to listen for specific operation events (e.g., __keyevent@0__:del). The notification payload contains the name of the impacted key.

    Note: <db> refers to the database index (e.g., 0).

    # Example Channel Names
    # Keyspace: __keyspace@0__:mykey
    # Keyevent: __keyevent@0__:del
  4. Handle binary data and byte responses

    master

    By default, the SDK may return responses as strings. If you need to work with raw bytes or pass byte arguments to commands, configure the client with decode_responses=False. When this option is set, command responses are returned as bytes, and you can pass bytes instead of str for arguments like keys.

    import redis
    # Responses will be returned as bytes
    client = redis.Redis(decode_responses=False)
    
    # You can pass bytes as keys or arguments
    client.set(b'my_key', b'my_value')
    value = client.get(b'my_key')
    assert isinstance(value, bytes)
  5. Perform atomic and non-atomic multi-key operations in a cluster

    master

    In Redis Cluster, multi-key commands (like mset or mget) are only atomic if all keys map to the same hash slot. You can force keys into the same slot using hash tags (e.g., {foo}1 and {foo}2).

    • Atomic operations: Use standard methods like mset() or mget(). If keys belong to different slots, a RedisClusterException will be raised.
    • Non-atomic operations: Use mset_nonatomic() or mget_nonatomic(). The client will batch keys by their hash value and send separate commands to the respective slot owners. This allows operations across different slots but sacrifices atomicity.
    # Atomic: requires hash tags to ensure same slot
    rc.mset({'{foo}1': 'bar1', '{foo}2': 'bar2'})
    rc.mget('{foo}1', '{foo}2')
    
    # Non-atomic: works across different slots
    rc.mset_nonatomic({'foo': 'value1', 'bar': 'value2', 'zzz': 'value3'})
    rc.mget_nonatomic('foo', 'bar', 'zzz')
  6. Use Subkey-aware Notifications for Hashes

    master

    To perform fine-grained cache invalidation or react to partial object changes (like specific fields in a Hash), Redis supports subkey-aware notifications:

    • Subkeyspace: Prefix __subkeyspace@<db>__:<keyname>. Payload format: <event>|<subkey_len>:<subkey>[,<subkey_len>:<subkey>...].
    • Subkeyevent: Prefix __subkeyevent@<db>:<event>. Payload format: <key_len>:<key>|<subkey_len>:<subkey>[,<subkey_len>:<subkey>...].
    • Subkeyspaceitem: Prefix __subkeyspaceitem@<db>__:<keyname>\n<subkey>. Payload format: <event>.
    • Subkeyspaceevent: Prefix __subkeyspaceevent@<db>__:<event>|<keyname>. Payload format: A length-prefixed subkey list.
  7. Implement client-side geographic failover with MultiDBClient

    master

    The MultiDBClient allows your application to connect to multiple Redis databases (typically replicas) to support Active-Active setups. It monitors database health and automatically fails over to the next highest-weighted healthy database when a failure is detected. It can also automatically switch back to a higher-weighted database once it becomes healthy again.

    Key Concepts

    • Weight: Each database has a priority. The client prefers the highest-weight healthy database.
    • Circuit Breaker: Protects databases using states: CLOSED (healthy), OPEN (unhealthy), and HALF_OPEN (probing).
    • Health Checks (Proactive): Background checks (defaulting to PING) that run at intervals to detect issues before they affect traffic.
    • Failure Detection (Reactive): Monitors organic command failures over a moving window to trigger failover based on real-time error rates.
    • Auto Fallback: If configured, the client periodically checks if a higher-weighted database has recovered to switch back to it.
    • Pub/Sub: The client automatically re-subscribes to channels when a failover occurs.

    MultiDBClient is designed to be a drop-in replacement for Redis or RedisCluster clients, sharing the same API.

    from redis.multidb.client import MultiDBClient
    from redis.multidb.config import MultiDbConfig, DatabaseConfig
    
    cfg = MultiDbConfig(
        databases_config=[
            DatabaseConfig(from_url="redis://db-primary:6379/0", weight=1.0),
            DatabaseConfig(from_url="redis://db-secondary:6379/0", weight=0.5),
        ]
    )
    
    client = MultiDBClient(cfg)
    client.set("key", "value")
    print(client.get("key"))
  8. PubSub in Redis Cluster

    master

    When creating a ClusterPubSub instance without specifying a node, the client transparently selects a node for the connection by hashing the requested channel name to find its keyslot.

    Limitations: pattern subscribe and publish are currently not recommended because pattern matching (e.g., fo*) involves an unknown number of potential channel names that cannot be mapped to specific keyslots in advance.

    # Connection is automatically set to the node holding the 'foo' keyslot
    p1 = rc.pubsub()
    p1.subscribe('foo')
    
    # Manually specifying a node for PubSub
    p2 = rc.pubsub(rc.get_node('localhost', 6379))
  9. Understand the Command API structure

    master

    Commands are exposed through different client types (Standalone, Cluster, and Sentinel) and are organized by their functional area (Core, Module, Cluster, or Sentinel).

    All command implementations eventually call the generic execute_command(*args, **kwargs) method. The SDK provides both synchronous and asynchronous versions of the API through method overloading.

  10. Handle Failover Exceptions and Automatic Fallback

    master

    Weight-based failover selects the highest-weighted database with a CLOSED circuit.

    • TemporaryUnavailableException: Thrown when no database is currently healthy. The client will retry for a configurable period (defaulting to 120 seconds based on failover_attempts * failover_delay). Your application should handle this to potentially switch to a different data source (like a cache).
    • NoValidDatabaseException: Thrown if no databases become available after the retry period.

    To automatically return to a higher-priority database once it becomes healthy again, set the auto_fallback_interval in MultiDbConfig.

    from redis.multidb.config import MultiDbConfig, DatabaseConfig
    
    cfg = MultiDbConfig(
        databases_config=[
            DatabaseConfig(from_url="redis://db-primary:6379/0", weight=1.0),
            DatabaseConfig(from_url="redis://db-secondary:6379/0", weight=0.5),
        ],
        # Try to fallback to higher-weight healthy database every 30 seconds
        auto_fallback_interval=30.0,
    )
    client = MultiDBClient(cfg)
  11. Understand Redis Response Modes (Wire Protocol vs Python Shape)

    master

    The behavior of redis-py is determined by the combination of the wire protocol (protocol) and the response mode (legacy_responses).

    Client optionsWire protocolPython response shape
    Redis()Default RESP3Legacy RESP2-compatible shape
    Redis(protocol=2)RESP2Legacy RESP2 shape
    Redis(protocol=3)RESP3Native RESP3 shape
    Redis(legacy_responses=False)Default RESP3Unified shape
    Redis(protocol=2, legacy_responses=False)RESP2Unified shape
    Redis(protocol=3, legacy_responses=False)RESP3Unified shape
  12. Configure Redis Search query dialects

    master

    Since release 6.0.0, redis-py defaults to DIALECT 2 for search and query commands (like FT.SEARCH and FT.AGGREGATE), automatically appending *DIALECT 2* to the command. This may impact result sets.

    You can explicitly set a different dialect using the .dialect() method on a Query object.

    from redis.commands.search.field import TextField
    from redis.commands.search.query import Query
    from redis.commands.search.index_definition import IndexDefinition
    import redis
    
    r = redis.Redis(host='localhost', port=6379, db=0)
    
    # Query with default DIALECT 2
    query = "@name: James Brown"
    q = Query(query)
    res = r.ft().search(q)
    
    # Query with explicit DIALECT 1
    query = "@name: James Brown"
    q = Query(query).dialect(1)
    res = r.ft().search(q)