DOCSEARCHbyAlgolia
Submit Docs

etcd

repository·main·Indexed 11 days ago

https://github.com/etcd-io/etcd

A distributed, reliable key-value store designed for managing critical configuration data in distributed systems, utilizing the Raft consensus algorithm. Includes documentation for the Go client v3, distributed locking with fencing tokens, systemd deployment for multi-node clusters, and monitoring integration via Prometheus and Grafana.

Tokens
83.4K
Snippets
334
Records
487
Agent score
99%

What's inside etcd

  1. Overview of the etcd client cache library

    main

    The etcd cache is an experimental client library designed to provide caching capabilities for etcd clients.

    Important Limitation: This library is not compatible with a gRPC proxy. The cache mechanism depends on RequestProgress RPCs, which are not forwarded by gRPC proxies. To use this cache, your client must communicate directly with the etcd cluster.

  2. Use the etcdctl CLI client

    main

    etcdctl is the command-line interface for interacting with etcd. By default, the main branch uses the v3 API.

    To switch between API versions, use the ETCDCTL_API environment variable:

    • For the v2 API: Set ETCDCTL_API=2.
    • For the v3 API (on versions earlier than v3.4): Set ETCDCTL_API=3.
    export ETCDCTL_API=3

  3. What is etcdutl and when to use it

    main

    etcdutl is a command-line administration utility for etcd designed to operate directly on etcd data files on disk.

    Important Distinction:

    • Use etcdutl for offline operations on data files (e.g., defragmenting a directory while etcd is not running or inspecting snapshots).
    • Use etcdctl for operations performed over a network against a live etcd instance.

  4. Use fragmented watch events to handle large watch responses

    main

    Starting in v3.3.14, you can request the etcd server to split watch events into smaller chunks if the total size exceeds the server's --max-request-bytes limit. This is done by setting the fragment field to true in etcdserverpb.WatchCreateRequest.

    Important Considerations:

    • The default server-side limit is embed.DefaultMaxRequestBytes (1.5 MiB) plus 512 bytes gRPC overhead.
    • If fragment is true and the response exceeds the limit, the server sends multiple fragmented events.
    • If the client's clientv3.Config.MaxCallRecvMsgSize is smaller than the server's limit, the client will error with "code = ResourceExhausted desc = grpc: received message larger than max (...)".
    • Clients must implement logic to merge these fragmented events. Note that clientv3 handles this automatically starting in etcd v3.4.

  5. Understand etcd security boundaries and trust assumptions

    main

    etcd is designed as a secure backend storage system that relies on several distinct trust boundaries. To ensure a secure deployment, developers and operators must respect these boundaries:

    • Network Boundary: etcd must be deployed in a strictly isolated, private network. It should never be exposed to the public internet.
    • Client-to-Server Boundary (Port 2379): Communication requires mTLS encryption. Clients must prove identity via certificates. Once authenticated, traffic is considered 'trusted input'.
    • Peer-to-Peer Boundary (Port 2380): Cluster members communicate via Raft consensus using dedicated, private peer certificates (mTLS). This boundary is limited to authorized cluster members.
    • Authentication & Authorization Boundary: etcd provides optional Role-Based Access Control (RBAC) layered behind mTLS. It is disabled by default. Note that watch-stream permission revocation is eventually consistent; existing streams may continue to receive events for a bounded period after revocation.
    • Data Storage Boundary: etcd writes data to local storage exactly as received. Data-at-rest protection (e.g., envelope encryption or filesystem encryption) is the responsibility of the client or the operator.
    • Host Execution Boundary: The etcd server is a statically linked Go binary (CGO_ENABLED=0) and does not rely on host dynamic libraries.

  6. Understand etcd feature stages (Alpha, Beta, GA)

    main

    etcd categorizes its features into three lifecycle stages. Understanding these stages helps you determine the stability and support level of a feature you are using:

    • Alpha: New features that are disabled by default. They may be buggy, support may be dropped without notice, and they can be removed in minor or major releases without following the standard deprecation policy. Use these with caution in production.
    • Beta: Features that are enabled by default and supported as part of supported etcd releases. Discontinuation of support must follow the official feature deprecation policy.
    • GA (General Availability): Features that are always enabled and cannot be disabled. The feature gate is removed once a feature reaches this stage. These are fully supported as part of supported releases and must follow the deprecation policy to be removed.

  7. Monitor etcd cluster peer connectivity with Prometheus

    main

    In etcd v3.4.44+, you can monitor the health and connectivity of cluster peers using the following Prometheus metrics. These metrics allow you to distinguish between active connections and disconnected peers.

    • etcd_network_active_peers: Tracks active remote peers. A value of 1 for a specific Local and Remote pair indicates an active connection.
    • etcd_network_disconnected_peers_total: Tracks the total number of disconnected peers. If a peer is down, this metric will increment for that specific Remote ID while etcd_network_active_peers for that peer will drop to 0.

    Example: If a 3-node cluster has node 7339c4e5e833c029 and node b548c2511513015 goes down, the local node's metrics will show etcd_network_disconnected_peers_total{Local="7339c4e5e833c029",Remote="b548c2511513015"} 1.

    etcd_network_active_peers{Local="7339c4e5e833c029",Remote="729934363faa4a24"} 1
etcd_network_disconnected_peers_total{Local="7339c4e5e833c029",Remote="b548c2511513015"} 1

  8. Identify issues for new contributors using help wanted and good first issue labels

    main

    The etcd project uses specific labels to signal that an issue is suitable for contributors with varying levels of experience.

    Help wanted

    Issues with the help wanted label should meet these criteria:

    • Low Barrier to Entry: Easy for new contributors to start.
    • Clear: The task is agreed upon and requires no further community discussion.
    • Goldilocks priority: The priority is high enough to be useful for core contributors to review, but not so high that a core contributor should be the one doing the work.

    Good first issue

    Issues with the good first issue label are intended for first-time contributors. These issues are a subset of help wanted and imply that maintainers will provide extra assistance (e.g., shepherding the PR through the process, helping with reviews, or explaining build failures).

    To qualify for good first issue, an issue should ideally have:

    • No Barrier to Entry: No advanced setup or domain knowledge required.
    • Solution Explained: A recommended solution is clearly described.
    • Examples: Links to similar implementations for reference.
    • Identified Code: Links to the relevant code and tests to be changed.
    • Ready to Test: Existing tests are available to be modified or copied. If no tests exist, a test fixture should be added before labeling.

  9. Identify the correct etcd Go module for your use case

    main

    The etcd project is organized into several distinct Go modules. Depending on whether you are building a client, using the CLI, or implementing a consensus protocol, you should import the corresponding module:

    • go.etcd.io/etcd/client/v3: The recommended client library for contacting etcd over the network via gRPC. Use this for almost all new etcd integrations.
    • go.etcd.io/etcd/api/v3: Contains API definitions, including protobufs and generated libraries that define the communication protocol between clients and servers.
    • go.etcd.io/etcd/etcdctl/v3: The command-line tool used to access and manage etcd.
    • go.etcd.io/raft/v3: An implementation of the distributed consensus protocol (hosted in a separate repository).
    • go.etcd.io/bbolt: A persistent b-tree implementation (hosted in a separate repository).
    • go.etcd.io/etcd/pkg/v3: A collection of utility packages. These are intended to be lightweight and general-purpose.
    • go.etcd.io/etcd/server/v3: The core etcd implementation. Note: This module is internal to etcd and should not be consumed by external projects as its API is subject to change within minor versions.

  10. Use watch event fragmentation to handle large event volumes

    main

    When watch events exceed the server's --max-request-bytes limit (defaulting to 1.5 MiB plus 512 bytes gRPC overhead), the server may fail to send them. To prevent this, you can use the WithFragment OpOption in clientv3. This tells the server to split the watch events into multiple smaller chunks (fragments), each staying below the request size limit. This is particularly useful for clients with limited bandwidth.

    Note: The client must be able to merge these fragmented events. clientv3 handles this automatically in etcd v3.4+.

    // Example usage of WithFragment in clientv3
// (Note: Exact syntax depends on the specific Watch call implementation)
clientv3.WithFragment(true)

  11. Handle non-blocking etcd client creation in clientv3

    main

    Starting with v3.7.0, the clientv3 package no longer honors the deprecated grpc.WithBlock dial option. Client creation is now non-blocking by default.

    If your application requires the previous blocking behavior (where the client creation waits until the connection is established), you must follow the official grpc-go guidance for implementing blocking dials to avoid using deprecated patterns.

    // Note: grpc.WithBlock is no longer honored in v3.7.0+
// Use grpc-go anti-pattern guidance to implement manual blocking if required.