Ztunnel Documentation

Ztunnel performance is measured by throughput and latency. The proxy handles two primary types of request flows:

TCP to TCP

This is a simple bi-directional byte copy between sockets. Ztunnel uses dynamically sized buffers that grow from 1kb → 16kb → 256kb based on traffic volume. This allows high throughput for heavy workloads while maintaining low memory overhead for low-bandwidth services.

TCP to HBONE

This flow is more complex as it involves HTTP/2 and TLS encapsulation:

1. Data Ingest: Data is read from the TCP socket into a dynamic buffer (up to 256kb).
2. H2 Layer: Data is buffered as an HTTP/2 DATA frame (max size configured to 256k).
3. TLS Layer: The connection driver calls rustls.write_vectored. TLS records are at most 16k. In practice, this often results in up to 4 chunks being written via writev calls.

HBONE to TCP

This flow is driven by the HTTP/2 receive side:

1. Decoding: Uses a LengthDelimitedCodec. The internal buffer starts at 8kb and grows to meet frame sizes (up to a max of 1mb via config.frame_size).
2. TLS Read: Calls rustls.read(buf), which typically performs 4kb reads from the underlying TCP connection.
3. H2 Buffering: Once the frame is read from the wire, it is buffered by h2 and read via recv_stream.poll_data, eventually writing the DATA frame to the upstream TCP connection.

In-pod mode allows you to run ztunnel in a custom network namespace to simulate a pod environment.

1. Create the network namespace and veth pair

ip netns add pod1
ip -n pod1 link set lo up

# Create veth device
ip link add pod1 type veth peer name pod1-eth0
# Move one end to the pod
ip link set pod1-eth0 netns pod1
# Configure the veth devices
ip link set pod1 up
ip -n pod1 link set pod1-eth0 up
ip addr add dev pod1 10.0.0.1/24
ip -n pod1 addr add dev pod1-eth0 10.0.0.2/24

2. Run a fake server in the pod

INPOD_UDS=/tmp/ztunnel cargo run --example inpodserver -- pod1

3. Run ztunnel

Run ztunnel as root (using the CARGO_TARGET... override to sudo the binary):

RUST_LOG=debug PROXY_MODE=shared INPOD_UDS=/tmp/ztunnel FAKE_CA="true" XDS_ADDRESS="" LOCAL_XDS_PATH=./examples/localhost.yaml cargo run --features testing

4. Verify and redirect traffic

Check the sockets inside the namespace:

ip netns exec pod1 ss -ntlp | grep ztunnel

Redirect traffic to ztunnel:

ip netns exec pod1 ./scripts/ztunnel-redirect-inpod.sh

# Example: Running ztunnel with sudo via cargo
export CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUNNER="sudo -E"
# ... then run cargo command

Ztunnel supports several environment variable overrides to facilitate local development by mocking components or using static configurations:

• FAKE_CA="true": Uses self-signed fake certificates (removes CA dependency).
• XDS_ADDRESS="": Disables the XDS client completely.
• LOCAL_XDS_PATH=./examples/localhost.yaml: Reads XDS configuration from a local file.
• CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUNNER="sudo -E": Instructs cargo to run the resulting binary with sudo -E (preserves environment).
• PROXY_MODE=dedicated: Enables single-tenant proxy mode. This is strongly recommended for local development as it avoids the need for manual Linux network namespace construction.
• PROXY_WORKLOAD_INFO=default/local/default: Sets the workload identity.

To run ztunnel entirely locally without Kubernetes or Istiod dependencies, use:

FAKE_CA="true" \
XDS_ADDRESS="" \
LOCAL_XDS_PATH=./examples/localhost.yaml \
CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUNNER="sudo -E" \
PROXY_MODE=dedicated \
PROXY_WORKLOAD_INFO=default/local/default \
cargo run --features testing

This setup (Linux only) allows you to run ztunnel as a local process on your host while it connects to an Istiod instance running in a KinD cluster. This replaces the ztunnel running on a specific worker node.

1. Prepare KinD Cluster

Create a cluster with an extraMount to allow the local ztunnel to connect to the node's CNI socket:

kind create cluster --config=- <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: ambient
nodes:
- role: control-plane
- role: worker
  extraMounts:
  - hostPath: /tmp/worker1-ztunnel/
    containerPath: /var/run/ztunnel/
- role: worker
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:5000"]
    endpoint = ["http://\\${KIND_REGISTRY_NAME}:5000"]
EOF

2. Remove ztunnel from the target node

Label the node to prevent ztunnel from scheduling there and patch the DaemonSet:

kubectl label node ambient-worker ztunnel=no
kubectl patch daemonset -n istio-system ztunnel --type=merge -p='{"spec":{"template":{"spec":{"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"ztunnel","operator":"NotIn","values":["no"]}]}]}}}}}}}'

3. Retrieve Credentials

Create a temporary pod to extract the ztunnel service account token and the Istio CA root cert:

kubectl get cm -n istio-system istio-ca-root-cert -o jsonpath='{.data.root-cert\.pem}' > /tmp/istio-root.pem

kubectl create -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: fake-tunnel-worker1
  namespace: istio-system
spec:
  nodeName: ambient-worker
  terminationGracePeriodSeconds: 1
  serviceAccountName: ztunnel
  containers:
  - name: cat-token
    image: ubuntu:22.04
    command:
    - bash
    - -c
    args:
    - "sleep 10000"
    ports:
    - containerPort: 80
    volumeMounts:
    - mountPath: /var/run/secrets/tokens
      name: istio-token
  volumes:
  - name: istio-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: istio-ca
          expirationSeconds: 43200
          path: istio-token
EOF

# Extract the token
kubectl exec -n istio-system fake-tunnel-worker1 -- cat /var/run/secrets/tokens/istio-token > ./var/run/secrets/tokens/istio-token

4. Configure Istiod and Run Ztunnel

Set ISTIOD_CUSTOM_HOST to localhost and port-forward the service:

kubectl set env -n istio-system deploy/istiod ISTIOD_CUSTOM_HOST=localhost
kubectl port-forward -n istio-system svc/istiod 15012:15012 &

# Run ztunnel
xargs env <<EOF
INPOD_UDS=/tmp/worker1-ztunnel/ztunnel.sock
CLUSTER_ID=Kubernetes
RUST_LOG=debug
PROXY_MODE="shared"
ISTIO_META_DNS_CAPTURE="true"
ISTIO_META_DNS_PROXY_ADDR="127.0.0.1:15053"
SERVICE_ACCOUNT=ztunnel
POD_NAMESPACE=istio-system
POD_NAME=ztunnel-worker1
CA_ROOT_CA=/tmp/istio-root.pem
XDS_ROOT_CA=/tmp/istio-root.pem
CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUNNER="sudo -E"
cargo run proxy ztunnel
EOF

Profiling memory usage requires ztunnel to be built with the jemalloc feature enabled (which is disabled by default).

To capture a heap profile:

1. Port-forward the admin port (15000).
2. Capture the profile using the /debug/pprof/heap endpoint.
3. For accurate symbol resolution in tools like pprof, you must copy the ztunnel binary and the system standard libraries from the container to your local machine.
4. Use the pprof tool with the PPROF_BINARY_PATH environment variable pointing to the directory containing your copied binaries.

# 1. Port-forward the admin port (15000)
kubectl port-forward -n istio-system ztunnel-qkvdj 15000:15000

# 2. Capture the memory profile
curl localhost:15000/debug/pprof/heap > mem.pb.gz

# 3. Copy binaries for symbol resolution
# ztunnel main binary
kubectl cp istio-system/ztunnel-qkvdj:/usr/local/bin/ztunnel ../../ztunnel-libs-pprof/ztunnel
# stdlibs (optional)
kubectl cp istio-system/ztunnel-qkvdj:/usr/lib/$BINARY_COMPILED_ARCH/ ../../ztunnel-libs-pprof/

# 4. Analyze with pprof
PPROF_BINARY_PATH=../../ztunnel-libs-pprof pprof -http=:8080 mem.pb.gz

To achieve FIPS compliance using the boring option, you must use vendored OS/arch specific FIPS-compliant binaries.

1. Identify your platform: Supported platforms are linux/x86_64 and linux/arm64.
2. Configure .cargo/config.toml: Manually edit the [env] section to point to the correct vendored paths.

For linux/x86_64:

BORING_BSSL_FIPS_PATH = { value = "vendor/boringssl-fips/linux_x86_64", force = true, relative = true }
BORING_BSSL_FIPS_INCLUDE_PATH = { value = "vendor/boringssl-fips/include/", force = true, relative = true }

For linux/arm64:

BORING_BSSL_FIPS_PATH = { value = "vendor/boringssl-fips/linux_arm64", force = true, relative = true }
BORING_BSSL_FIPS_INCLUDE_PATH = { value = "vendor/boringssl-fips/include/", force = true, relative = true }

3. Build: Run cargo build or use the release script:

TLS_MODE=boring ./scripts/release.sh