Rainbond Documentation

etcd storage consumes physical memory through two primary components:

1. In-memory index: A B-tree structure that holds all keys and pointers to on-disk values to speed up lookups. This is the primary driver of predictable memory usage.
2. Page cache: Managed by the operating system, this stores recently-accessed data from disk for quick reuse. Value size has a minor impact on memory usage here as it increases the amount of data held in the OS page cache.

To estimate the theoretical memory consumption of the in-memory index, use the following formula:

N * (c1 + avg_key_size) + N * (avg_versions_of_key) * (c2 + size_of_pointer)

Where:

• N: Number of keys.
• avg_key_size: Average size of the keys.
• avg_versions_of_key: Average number of versions per key.
• size_of_pointer: Size of the pointer to the on-disk data.
• c1: Key metadata overhead.
• c2: Version metadata overhead.

The behavior of the etcd client balancer (how it connects to and switches between endpoints) depends on the gRPC version used by the client:

clientv3-grpc1.0

• Mechanism: Maintains multiple TCP connections to all configured endpoints. It pins one address for all requests.
• Failover: If an error occurs, it randomly picks another address and retries.
• Limitation: High resource usage due to multiple connections; does not understand node health or cluster membership, which can lead to getting stuck on a failed/partitioned node.

clientv3-grpc1.7

• Mechanism: Maintains only one TCP connection. It attempts to connect to all endpoints and pins the first successful one, closing others.
• Failover: Uses an error handler to decide whether to retry on the same endpoint or switch addresses based on error codes/messages. For stream RPCs (Watch/KeepAlive), it uses HTTP/2 pings.
• Limitation: Uses a hard-coded 5-second dial timeout for the "unhealthy" list, which can cause false positives. It cannot detect network partitions effectively.

clientv3-grpc1.14

• Mechanism: Creates multiple sub-connections (one per endpoint) and uses a round-robin policy by default. It delegates complex balancing to the upstream gRPC resolver group.
• Failover: Simplifies logic by round-robinning to the next endpoint whenever a disconnection occurs, rather than maintaining a complex (and potentially stale) unhealthy endpoint list.
• Retry Logic: Implements retries via a gRPC interceptor chain, supporting advanced policies like backoff.
• Limitation: Higher resource consumption (one TCP connection per endpoint) and still lacks advanced cluster-membership-aware health checking.

In the Raft consensus implementation, the leader tracks the progress of each follower using two primary attributes and three distinct states. This mechanism controls how log entries are replicated and how snapshots are sent.

Progress Attributes

• match: The index of the highest known matched entry on the follower. If the leader has no knowledge of the follower's status, this is set to 0.
• next: The index of the first entry that will be replicated to the follower in the next replication message (a msgApp containing log entries).

Progress States

• probe: The leader sends at most one replication message per heartbeat interval to cautiously test the follower's progress. This state is used for newly elected leaders or when a follower has fallen behind.
• replicate: An optimized state for fast log replication. The leader sends replication messages and optimistically increases next to the latest entry sent to maximize throughput.
• snapshot: The leader stops sending replication messages and instead sends a snapshot to catch the follower up when it has fallen too far behind.

etcd provides a consistent and durable key-value store with mini-transaction support. The KV API provides the following core guarantees:

• Atomicity: All API requests are atomic; an operation either completes entirely or not at all. Watch requests never observe partial events for a single operation.
• Consistency: etcd ensures sequential consistency. Clients read the same events in the same order regardless of which member server they contact. For range operations, etcd provides linearized access by default.
• Isolation: etcd ensures serializable isolation, meaning read operations will never observe intermediate data.
• Durability: Any completed operation is durable. A read will never return data that has not been made durable.
• Linearizability: etcd ensures linearizability for most operations by default, meaning operations appear to execute in a sequential order consistent with their timestamps. However, watch operations are not linearizable; users must verify the revision of watch responses to ensure correct ordering.

The etcd client is designed to provide a single logical cluster view of multiple physical machines by implementing automatic failover between replicas. To ensure high availability and correctness, the client must adhere to several key requirements:

• Correctness: Requests must not violate consistency guarantees (e.g., global ordering, no corrupted data, at-most once semantics for mutable operations).
• Liveness: Clients must make progress even if servers fail or disconnect briefly. They should detect unavailable servers using HTTP/2 ping and failover to other nodes without deadlocking.
• Effectiveness: Clients should minimize resource usage, such as gracefully closing previous TCP connections after an endpoint switch and predicting the next replica to connect without wasteful retries.
• Portability: The implementation should align with gRPC design goals (like pluggable retry policies) to ensure consistent error handling across different language bindings.

The etcd v3 authentication system is designed for gRPC-based transport, moving away from the RESTful v2 model. Key characteristics include:

• Connection-based Authentication: Authentication is performed once per connection rather than per request, improving performance.
• State Machine Enforcement: Unlike v2, permission checking occurs in the state machine layer during the Raft apply phase. This ensures that permission checks are consistent with the latest auth metadata and prevents stale permission attacks during linearized requests.
• Permission Model: Uses interval matching (key ranges) rather than a directory structure, as etcd v3 uses a flat key space.
• Token Types:
  • Simple Tokens: Not cryptographically signed; requires stateful tracking by servers. Use only for development/testing.
  • JWT Tokens: Cryptographically signed and stateless. Recommended for production deployments.

Security Note: Because etcd v3 is a Key-Value Store (KVS) and not a file system, permissions can be granted to non-existent keys or key ranges. Developers must ensure they do not unintentionally grant broad permissions via key intervals.