txredisapi

repository·master·Indexed 42 days ago

https://github.com/ilyaskriblovsky/txredisapi

A Python-based Redis API client and wrapper featuring lazy connections, connection pooling, and sharding via consistent hashing. It supports asynchronous operations with Twisted, Redis Sentinel for failover, and provides specialized handlers for TCP and Unix sockets.

Tokens
2.4K
Snippets
6
Records
8
Agent score
4%

What's inside txredisapi

  1. Perform Redis transactions with MULTI/EXEC

    master

    Transactions are supported on normal, lazy, and connection pool connections (but not on sharded connections).

    Because exec is a reserved keyword in Python, you must use the .commit() method to finalize a transaction. To use optimistic locking, use .watch(key).

    • multi(): Starts the transaction.
    • commit(): Commits the transaction and returns a list of all command responses.
    • discard(): Aborts the transaction and returns OK.
    @defer.inlineCallbacks
    def main():
        rc = yield redis.ConnectionPool()
        yield rc.delete(["a1", "a2", "a3"])
    
        # Start transaction
        t = yield rc.multi()
    
        # Commands return "QUEUED"
        yield t.set("a1", "1")
        yield t.set("a2", "2")
    
        # Commit and get all responses
        r = yield t.commit()
        print "commit=", repr(r)
    
        yield rc.disconnect()
  2. How connection handlers and reconnection work

    master

    A connection handler is the primary interface for interacting with Redis.

    • Normal (non-lazy) connections: Return a Deferred that fires with the connection handler once the connection is established. For pools, it fires once all connections are ready.
    • Lazy connections: Return the connection handler immediately, even before the connection is made. This is ideal for web servers that shouldn't block startup if Redis is down.
    • Error Handling: If a connection is lost, commands will raise a ConnectionError. If reconnect=True was set, the driver will attempt to reconnect in the background.
    • Pools: Handlers automatically select available connections from a pool and handle reconnection.
  3. Use Pipelining to batch commands

    master

    While txredisapi automatically pipelines commands to reduce round-trip delays, you can explicitly use the pipeline() method to group large batches of commands. This is more efficient for bulk loading data as it reduces TCP packet overhead.

    Best Practice: Group commands in batches (e.g., 10,000 commands) rather than sending everything at once to avoid excessive memory usage on the client or server.

    @defer.inlineCallbacks
    def main():
        rc = yield redis.ConnectionPool()
    
        # Start grouping commands
        pipeline = yield rc.pipeline()
    
        pipeline.set("foo", 123)
        pipeline.set("bar", 987)
        pipeline.get("foo")
        pipeline.get("bar")
    
        # Execute the batch and wait for all replies
        results = yield pipeline.execute_pipeline()
    
        print "foo:", results[2] # 123
        print "bar:", results[3] # 987
    
        yield rc.disconnect()
  4. Use Sharded Connections for automatic distribution

    master

    Sharded connections distribute keys across multiple Redis servers using a built-in consistent hashing algorithm.

    Note: Not all commands are supported on sharded connections. If you attempt an unsupported command, the handler will raise a NotImplementedError.

    @defer.inlineCallbacks
    def main():
        # Provide a list of host:port pairs
        rc = yield redis.ShardedConnection(["localhost:6379", "localhost:6380"])
        
        keys = []
        for x in xrange(100):
            key = "foo%02d" % x
            yield rc.set(key, "bar%02d" % x)
            keys.append(key)
    
        # mget is supported on sharded connections
        response = yield rc.mget(keys)
        for val in response:
            print val
    
        yield rc.disconnect()
  5. Install txredisapi

    master

    Since txredisapi.py is a pure-python, single-file module, you can simply copy the file directly into your project directory to use it without a formal installation.

    Alternatively, you can install it via pip:

    pip install txredisapi

    If you are using the cyclone web framework, txredisapi is likely already available as cyclone.redis.

    pip install txredisapi
  6. Use txredisapi for basic Redis commands

    master

    To use txredisapi, you must use an asynchronous approach (typically with Twisted's defer.inlineCallbacks). The driver returns a connection handler that supports standard Redis commands like set, get, and delete.

    Note: It is strongly recommended to pass arguments as keyword arguments to avoid issues with positional parameter shifts if the API evolves.

    #!/usr/bin/env python
    # coding: utf-8
    
    import txredisapi as redis
    from twisted.internet import defer
    from twisted.internet import reactor
    
    @defer.inlineCallbacks
    def main():
        # Establish a connection
        rc = yield redis.Connection()
        print rc
    
        # Execute commands
        yield rc.set("foo", "bar")
        v = yield rc.get("foo")
        print "foo:", repr(v)
    
        yield rc.disconnect()
    
    if __name__ == "__main__":
        main().addCallback(lambda ign: reactor.stop())
        reactor.run()
  7. Connect using Redis Sentinel

    master

    You can use txredisapi.Sentinel to discover Redis master and slave addresses and handle automatic failover.

    • Use sentinel.master_for("service_name") to connect to the master.
    • Use sentinel.slave_for("service_name") to connect to a slave.
    • Use min_other_sentinels=N in the Sentinel constructor to mitigate split-brain risks during network partitioning.
    @defer.inlineCallbacks
    def main(reactor):
        # List of (sentinel_host, sentinel_port) tuples
        sentinel = txredisapi.Sentinel([("sentinel-a", 26379), ("sentinel-b", 26379), ("sentinel-c", 26379)])
        
        # Get master connection
        redis = sentinel.master_for("service_name")
        
        yield redis.set("foo", "bar")
        print (yield redis.get("foo"))
        
        yield redis.disconnect()
        yield sentinel.disconnect()
  8. Reference all connection methods and arguments

    master

    The driver provides several connection types depending on your needs (single, pool, sharded, or Unix socket). All methods support lazy versions which return the handler immediately before the connection is established.

    Connection Types

    • Connection(...) / lazyConnection(...) (Standard TCP)
    • ConnectionPool(...) / lazyConnectionPool(...) (TCP with pooling)
    • ShardedConnection(...) / lazyShardedConnection(...) (Automatic sharding via consistent hashing)
    • ShardedConnectionPool(...) / lazyShardedConnectionPool(...) (Sharded pooling)
    • UnixConnection(...) / lazyUnixConnection(...) (Unix Socket)
    • UnixConnectionPool(...) / lazyUnixConnectionPool(...) (Unix Socket pooling)
    • ShardedUnixConnection(...) / lazyShardedUnixConnection(...) (Sharded Unix Socket)
    • ShardedUnixConnectionPool(...) / lazyShardedUnixConnectionPool(...) (Sharded Unix Socket pooling)

    Arguments

    ArgumentDescription
    hostIP address or hostname [default: localhost]
    portPort number [default: 6379]
    pathPath of redis server's socket [default: /tmp/redis.sock]
    dbidDatabase ID [default: 0]
    poolsizeNumber of connections in a pool [default: 10]
    reconnectAuto-reconnect if connection is lost [default: True]
    charsetString encoding (use None to skip decoding/encoding) [default: utf-8]
    hostsList of host:port pairs (for sharded)
    pathsList of pathnames (for sharded Unix sockets)
    passwordRedis server password [default: None]
    usernameRedis 6+ ACL username [default: None]
    ssl_context_factoryBoolean for SSL/TLS or a specific ClientContextFactory [default: False]
    connectTimeoutTimeout in seconds for establishing connection [default: None]
    replyTimeoutTimeout in seconds for waiting for a reply [default: None]
    convertNumbersAutomatically convert numeric replies to Python int/float [default: True]