DOCSEARCHbyAlgolia
Submit Docs

Twemproxy (Nutcracker)

repository·master·Indexed 11 days ago

https://github.com/twitter/twemproxy

A fast, lightweight proxy for Memcached and Redis protocols designed to reduce backend connection counts and enable horizontal scaling through sharding and pipelining. It supports Memcached ASCII commands, YAML-based server pool configuration, and provides observability via a dedicated TCP stats port.

Tokens
13.4K
Snippets
32
Records
46
Agent score
90%

What's inside Twemproxy

  1. Important constraints and behaviors for Memcached commands

    master

    When using Twemproxy as a Memcached proxy, observe these behaviors:

    • Key Limits: The maximum length of a key is 250 characters.
    • Expiration (<expiry>):
      • 0 means the item never expires (though it may be evicted).
      • Non-zero values can be a Unix timestamp (seconds since 01/01/1970) or an offset in seconds from the current time (up to 30 days).
      • Expiry is calculated relative to the server time, not the client time.
    • Arithmetic:
      • decr of 0 results in 0.
      • incr of UINT64_MAX results in 0.
    • Storage Logic:
      • set always creates a mapping.
      • add only succeeds if the key is absent.
      • replace only succeeds if the key is present.
      • append and prepend ignore <flags> and <expiry> values.
    • Data Length: <datalen> can be zero, in which case the <data> block is empty.
    • Noreply: The noreply flag instructs the server to skip sending a response even if an error occurs.

  2. Understand the impact of server_connections > 1 on request ordering

    master

    Twemproxy multiplexes multiple client connections over a limited number of server connections. If server_connections is configured to be greater than 1, the "read my last write" constraint is not guaranteed.

    In a pipelined scenario where a client sends a write (e.g., set foo 0 0 3\r\nbar\r\n) followed immediately by a read (e.g., get foo\r\n), these two requests might be dispatched over different server connections. This creates a race condition where the read might complete before the write is fully processed by the backend server.

    To ensure strict ordering for "read my last write" semantics, you must either:

    1. Set server_connections: 1 in your configuration.
    2. Use clients that only make synchronous requests to twemproxy.
    # Example of a race condition with server_connections > 1
# Client pipeline:
set foo 0 0 3\r\nbar\r\n
get foo\r\n

# If server_connections: 2, these may execute on different connections,
# causing the GET to potentially return the old value of 'foo'.

  3. Optimize mbuf size for high concurrency

    master

    Twemproxy uses mbuf chunks for zero-copy request/response forwarding. The default size is 16KB. While larger mbufs reduce syscalls, they increase memory usage per connection. If you need to handle a very large number of concurrent client connections, reduce the chunk size using the -m or --mbuf-size=N flag (e.g., to 512 bytes).

    # Example: setting a small mbuf size for high concurrency
nutcracker -m 512

  4. Understand Redis command support and limitations

    master

    Twemproxy supports Redis commands with the following characteristics:

    • Case Insensitivity: Redis commands are not case sensitive.
    • Vectored Command Fragmentation: Certain 'vectored' commands (commands that take multiple keys or key-value pairs at once) must be fragmented by the client. This is because Twemproxy distributes individual keys across different shards.

    The following commands require fragmentation:

    • MGET key [key ...]
    • MSET key value [key value ...]
    • DEL key [key ...]
    • UNLINK key [key ...]
    • EXISTS key [key ...]

  5. Use RPOPLPUSH with Twemproxy

    master
    The RPOPLPUSH command is supported, but it requires that the source and destination keys hash to the same server. Twemproxy does not verify this requirement; it simply forwards the command to the server that the source key hashes to. To ensure the command works, use the same hashtag for both the source and destination keys.

  6. Use EVAL and EVALSHA with multiple keys

    master

    Twemproxy's support for EVAL and EVALSHA is limited to scripts that include at least one key. If your script uses multiple keys, all keys must hash to the same backend server.

    To ensure this, use hash tags so that all keys in the script share the same hash slot. If you provide multiple keys without ensuring they hash to the same server, twemproxy will not perform a validation check; instead, it will simply forward the entire command to the backend server that the first key in the script hashes to, which may lead to errors if the other keys reside elsewhere.

  7. How pipelining works in twemproxy

    master

    Twemproxy improves throughput by enabling the proxying of multiple client connections onto a single or small number of server connections. This allows twemproxy to batch requests from different clients into a single message sent to the backend server, reducing round-trip time (RTT).

    Example Scenario: If three clients send the following requests:

    1. get key\r\n
    2. set key 0 0 3\r\nval\r\n
    3. delete key\r\n

    Twemproxy can batch these into a single message sent to the server connection: get key\r\nset key 0 0 3\r\nval\r\ndelete key\r\n

  8. Build twemproxy from distribution tarballs

    master

    To build twemproxy 0.5.0+ from a distribution tarball, use the standard configure and make workflow. For a debug build, use the --enable-debug=full flag with specific CFLAGS.

    # Standard build
$ ./configure
$ make
$ sudo make install

# Debug mode build
$ CFLAGS="-ggdb3 -O0" ./configure --enable-debug=full
$ make
$ sudo make install

  9. Deploy twemproxy in production

    master

    When deploying twemproxy in a production environment, it is recommended to review the tuning parameters in the official recommendation document to ensure efficient operation.

    Refer to notes/recommendation.md in the repository for specific guidance on tuning parameters.

  10. Enable debug logging in nutcracker

    master

    By default, debug logging is disabled. To enable it, you must compile nutcracker with the --enable-debug=log configure option.

    Once compiled, you can run nutcracker with verbosity level LOG_INFO using the -v 6 or --verbose=6 flag. This level logs the lifecycle of client/server connections and important events like server ejection from the hash ring, with minimal runtime overhead.

    # Example of running with verbosity 6
nutcracker --verbose=6 [other options]

  11. Monitor twemproxy using stats and logs

    master

    Twemproxy provides observability through two primary channels: stats and logs.

    Stats Monitoring

    Twemproxy exposes statistics via a dedicated TCP monitoring port. The stats are provided as JSON-formatted key-value pairs representing counters.

    • Default Port: 22222
    • Default Aggregation Interval: 30 seconds
    • Configuration:
      • Use -c or --conf-file to specify a configuration file.
      • Use -i or --stats-interval to change the aggregation interval.
    • Discovery: Run with -D or --describe-stats to see a description of all exported metrics.

    Logging

    Logging is available if twemproxy is built with logging enabled.

    • Default Output: stderr
    • File Output: Use -o or --output <file> to redirect logs to a specific file.
    • Runtime Control: Send signals to a running process to manage log levels:
      • SIGTTIN: Increase log level.
      • SIGTTOU: Decrease log level.
      • SIGHUP: Reopen log files.
    # Describe all exported stats
nutcracker --describe-stats

# Run with custom config and stats interval
nutcracker -c /path/to/config.yaml -i 60

# Run with log output to a file
nutcracker -o /var/log/twemproxy.log

  12. Build twemproxy from source

    master

    To build from the git repository with debug logs and assertions enabled, you must first run autoreconf -fvi to prepare the build system. Ensure automake and libtool are installed on your system.

    $ git clone git@github.com:twitter/twemproxy.git
$ cd twemproxy
$ autoreconf -fvi
$ ./configure --enable-debug=full
$ make
$ src/nutcracker -h