DOCSEARCHbyAlgolia
Submit Docs

Libtpa (Transport Protocol Acceleration)

repository·main·Indexed 41 days ago

https://github.com/bytedance/libtpa

A high-performance, DPDK-based userspace TCP stack designed to accelerate specific TCP connections while coexisting with the host TCP/IP stack. Optimized for Mellanox NICs, it utilizes a run-to-completion and shared-nothing model to reduce latency and increase throughput for applications like Redis. Features include flow bifurcation via rte_flow, zero-copy read/write APIs, and a dedicated cleanup daemon called libtpad.

Tokens
10.5K
Snippets
45
Records
56
Agent score
41%

What's inside Libtpa

  1. Libtpa supported features and hardware

    main

    TCP Features

    • New Reno
    • Fast retransmission
    • Timed out retransmission
    • Spurious fast retransmission detection
    • Congestion window validation
    • Selective ACK (SACK)
    • Delayed ACK
    • Keepalive
    • Zero window probe
    • Protection against wrapped sequence numbers (PAWS)
    • Timestamp option
    • Window scale option
    • Maximum segment size (MSS) option

    Other Features

    • IPv6
    • TSO (TCP Segmentation Offload)
    • Checksum offload
    • Jumbo frame support
    • Multiple thread support
    • Zero copy read/write
    • epoll-like interface

    Supported Hardware

    • Platforms: AMD64, ARM (not well tested)
    • NICs: Mellanox NICs (ConnectX-4 through ConnectX-7)

  2. How libtpa's embedded TCP stack works

    main

    Libtpa is an embedded TCP stack implementation designed for high performance. Unlike standalone stacks, it operates in run-to-completion mode and does not create its own datapath threads. Instead, the stack is embedded directly within the application thread.

    Additionally, libtpa is not a replacement for the host TCP/IP stack. It works alongside the host stack by taking control of only specific TCP connections that require acceleration. All other connections (including non-TCP traffic like UDP) continue to be handled by the host stack. This design ensures that if libtpa crashes, only the accelerated application is affected, while the rest of the system remains stable.

  3. Run multiple libtpa instances using TPA_ID

    main

    Libtpa uses TPA_ID as a unique identifier for each instance. While libtpa can automatically generate IDs (e.g., swing, swing1, swing2), it is highly recommended to set TPA_ID manually. This provides better control when running multiple instances of the same application, as most libtpa tools require a unique ID.

    Example: TPA_ID=client tpa run swing ...

    # Running two distinct instances of tperf
TPA_ID=server taskset -c 1 tperf -s -n 1
TPA_ID=client taskset -c 2 tperf -c 127.0.0.1 -t rr -m 1

  4. Understand the Libtpa architecture and execution model

    main

    Libtpa (Transport Protocol Acceleration) is a DPDK-based userspace TCP stack implementation designed for high performance. It operates using two key architectural principles:

    1. Embedded TCP Stack (Run-to-Completion): Libtpa does not create its own datapath threads. Instead, it is embedded directly into the application thread, operating in a run-to-completion mode.
    2. Selective Connection Acceleration: Libtpa is not a standalone stack. It coexists with the host TCP/IP stack. It only takes control of specific TCP connections targeted for acceleration (e.g., all TCP connections belonging to a Redis instance). All other traffic (non-accelerated TCP or UDP) continues to be handled by the host stack. This isolation ensures that if Libtpa crashes, only the accelerated application is affected, while the rest of the system remains stable.

  6. Understand the Libtpa shared-nothing model

    main

    Libtpa uses a shared-nothing model to achieve high performance by avoiding locks in the datapath.

    Key Constraint: A single TCP connection is bound to exactly one worker thread. All operations for a specific connection—including connect, write, read, and close—must be invoked within that same worker thread.

    Warning: Attempting to create a connection in one worker thread (e.g., Worker A) and performing a write operation in a different thread (e.g., Worker B) will result in undefined behavior, including application crashes.

  7. Manage memory for Zero Copy Write

    main

    When using zero-copy write APIs (tpa_write or tpa_zwritev), the application must not free the write buffer immediately after the API returns. Because libtpa uses the buffer for potential retransmissions, the buffer remains referenced by the library until the data is acknowledged (ACKed) by the remote end.

    To safely free the buffer, you must provide a callback function in the tpa_iovec structure using the iov_write_done field. Libtpa will invoke this callback once the data is no longer needed.

    /* Example concept: the callback is triggered when data is ACKed */
struct tpa_iovec iov;
iov.iov_write_done = my_write_done_callback;
iov.iov_param = my_context;
tpa_zwritev(..., &iov, ...);

  8. How packet offloading works via rte_flow

    main
    Libtpa leverages flow bifurcation to steer specific packets of interest to the userspace stack. This is implemented using the DPDK rte_flow interface. Specifically, it uses the QUEUE action to steer packets to a specific worker, enabling a shared-nothing model for high-performance processing.

  9. Understand Libtpa Loopback Mode

    main

    Libtpa supports a loopback mode that differs from the standard Linux lo interface. This mode requires physical loopback support from the Network Interface Card (NIC); packets travel into the NIC and are returned to the same host.

    Important Limitation: Libtpa cannot connect to loopback TCP connections if the other end is the standard Linux kernel TCP/IP stack. Loopback testing only works when both the client and the server are running with libtpa.

  10. Understand the Libtpa run-to-completion model

    main

    Libtpa follows the classical DPDK run-to-completion model. To drive the TCP stack and process network events, your application must continuously call the tpa_worker_run function.

    This means your application architecture must be designed to operate within a loop that drives the stack, rather than relying on an asynchronous or interrupt-driven model.

  11. Analyze socket read/write latencies

    main

    When using tpa sk -v, you can inspect the internal latency measurements for both read and write operations. This helps identify bottlenecks in the stack or application.

    Write Latency Stages

    1. write_lat.submit: Latency from invoking the TPA write API to submitting the request to the socket TX queue.
    2. write_lat.xmit: Latency from invoking the API to fetching the data from the TX queue, encapsulating it with headers, and sending it to the NIC.
    3. write_lat.complete: Latency from invoking the API to receiving the ACK from the remote host.

    Read Latency Stages

    1. read_lat.submit: Latency from receiving the packet from the NIC to delivering it to the socket RX queue.
    2. read_lat.drain: Latency from receiving the packet to the application reading the data via the libtpa read API.
    3. read_lat.complete: Latency from receiving the packet to the application finishing processing via the iov.iov_read_done callback.
    # Example: Filter for socket ID and latency metrics
tpa sk -v | grep -e sid -e _lat

  12. Hardware requirements for Libtpa

    main

    Libtpa requires flow bifurcation support from the Network Interface Card (NIC) to selectively route specific TCP connections to the userspace stack while leaving others to the host stack.

    • Mellanox NICs: Fully supported. Mellanox provides native flow bifurcation support that works with upstream DPDK without requiring SR-IOV or internal patches/firmware modifications.
    • Other NICs: While many NICs support flow bifurcation via SR-IOV, they may require specific internal DPDK/Linux patches or firmware updates to meet Libtpa's requirements. Currently, Libtpa is optimized for and primarily supports Mellanox NICs.