Kuma Service Mesh Documentation

repository·master·Indexed 26 days ago

https://github.com/kumahq/kuma

Kuma is a CNCF Sandbox project and Envoy-based service mesh providing L4-L7 connectivity, security, and observability across Kubernetes, VMs, and Bare Metal. It supports multi-zone and multi-mesh architectures. Documentation covers installation via Helm charts, Control Plane and Data Plane configuration, CNI installation, Mesh Ingress/Egress for cross-cluster communication, Prometheus monitoring setup for Grafana dashboards, and transparent proxy configuration.

Tokens
171.7K
Snippets
387
Records
840
Agent score
87%

What's inside Kuma

  1. Overview of Kuma Features

    master

    Kuma provides a comprehensive set of service mesh capabilities including:

    • Connectivity & Discovery: Automatic discovery, ingress, and connectivity across single and multi-zones. Supports internal and external services.
    • Security: Automatic mTLS with identity and encryption, TLS rotation, and MeshTrafficPermission for zero-trust firewalling.
    • Traffic Control: Dynamic load-balancing for blue/green, canary, and rollback deployments using MeshHTTPRoute and MeshTCPRoute. Includes MeshRetry for reliability.
    • Observability: Native Prometheus/Grafana support via MeshMetric, full traffic tracing via MeshTrace, and activity logging via MeshAccessLog.
    • Resilience: System hardening via MeshFaultInjection.
    • Deployment Models: Supports Multi-Mesh (isolated meshes on one control plane), Single/Multi-Zone (cross-cloud/cross-cluster), and Global/Remote Control Planes for scalability.
    • Management: Platform-agnostic (K8s, VM, Bare Metal), transparent proxying, and a browser-based GUI for exploration.
  2. Understand the new Envoy resource and stat naming formats

    master

    Kuma is migrating to a consistent naming convention for non-system Envoy resources and stats. This change affects how resources are identified and how their metrics are reported in observability tools like Prometheus and Grafana.

    Applicability

    These new naming formats apply only to environments using the new service discovery model. If your environment uses the legacy kuma.io/service tag, these changes do not apply to you. The new formats are used for resources generated from:

    • MeshService
    • MeshExternalService
    • MeshMultiZoneService

    Impacted Components

    • Observability: Grafana dashboards installed via kumactl install observability are being updated to support both old and new formats.
    • Kuma GUI: The Dataplane view (which shows inbound/outbound endpoints) is being updated to handle the transition between legacy and new naming formats during the migration period.
  3. Understand the Kuma Release Pipeline (Option 1)

    master

    Kuma uses a release pipeline designed to minimize release times by skipping redundant tests on tags. Instead of running the full end-to-end (e2e) and unit test suites when a tag is created, the pipeline relies on the fact that these tests were already passed and the artifacts were already published during the push run on the branch.

    Key Pipeline Characteristics:

    • Reduced Latency: Cuts release time from ~100–220m down to ~20–30m.
    • Verification Strategy: On a tag, the pipeline only verifies the ldflags delta using container-structure tests on the freshly-built image and a binary-version assert.
    • Security & Reliability: Security controls (SBOM, provenance, signing, and scanning) remain unchanged in the build_publish step. A "hard green-SHA gate" ensures that a tag can only be released if the specific commit SHA has a successful, published push run.
    • No Rehearsal Needed: Every push already builds and publishes a preview via _build_publish.yaml, so the publish/sign path is continuously exercised.
  4. Understand the Policy Matching Algorithm structure

    master

    Kuma's policy matching algorithm maps user-provided policies into a structured list of Rules. Each rule consists of a Subset (defining the criteria for matching), a Conf (the configuration applied when the criteria are met), and an Origin.

    This structure is utilized by both "from" and "to" policy plugins. Policy plugins iterate over the Rules list to compute the appropriate Conf for a known destination by checking if the destination belongs to the Rules[].Subset.

    Rules:
      - Subset:
          - Key: kuma.io/service
            Not: false
            Value: foo
        Conf:
          connectionTimeout: 2s
          http:
            requestTimeout: 15s
          idleTimeout: 20s
        Origin: [ ... ]
      - Subset:
          - Key: kuma.io/service
            Not: false
            Value: bar
        Conf:
          connectionTimeout: 2s
          http:
            requestTimeout: 20s
          idleTimeout: 20s
        Origin: [ ... ]
  5. Compare SNI format options (KRI-derived vs. Hash-based)

    master

    Kuma has transitioned to a KRI-derived SNI format to solve issues with the previous hash-based format.

    FeatureKRI-derived SNI (Current)Hash-based SNI (Legacy)
    ReversibilityBidirectional: SNI can be converted back to KRI without a lookup.Lossy: Impossible to reverse SNI to KRI without a lookup.
    ReadabilityHuman-readable: Segments are explicit and predictable.Opaque: Contains a non-human-readable hash prefix.
    PredictabilityHigh: Users/integrations can construct SNIs from known attributes.Low: Users cannot predict the hash.
    DNS ComplianceRequires validation to stay under 253 chars.Naturally stays within length limits.
    ComplexityHigher (requires dual-format support during migration).Lower (minimal changes to existing logic).
  6. Understand the transition from Source/Destination selectors to Inbound/Outbound selectors

    master

    Kuma is moving away from the traditional source and destination selector model to a more transparent inbound and outbound selector model. This change aims to resolve ambiguity in how policies map to Envoy proxy connections.

    In the previous model, Kuma hid whether a policy was selecting 'inbounds' or 'outbounds' under the hood, which led to complexity regarding:

    • Limitations on destination selectors (often limited to kuma.io/service tags).
    • Unexpected behavior in source selectors (e.g., TrafficPermission policies shadowing each other).
    • Ambiguity in determining policy specificity.
    • Implementation difficulties with source selectors for TCP traffic.

    Users should prepare for a model where policy mapping to inbounds or outbounds is explicitly defined via inbound and outbound selectors.

  7. Understand Zone Egress Identity and mTLS behavior

    master

    Kuma uses ZoneEgress as a centralized exit point for outgoing requests to destinations outside the local zone network. This allows users to restrict egress traffic to specific external addresses.

    Communication between application sidecars and the ZoneEgress workload is secured using mutual TLS (mTLS). In the current implementation, ZoneEgress uses SNI-based matching for specific targets, where each filter chain is assigned a separate identity unique to each mesh. This ensures that outbound traffic from different meshes can be handled by a single ZoneEgress instance while maintaining security boundaries.

  8. Understand Kuma Policy Matching with TargetRef

    master

    Kuma uses a targetRef selector to attach policies to specific resources. This allows for a hierarchical and role-oriented approach to policy management (e.g., Mesh operators vs. Service owners).

    Supported TargetRef Kinds

    • Mesh: Affects all Data Plane Proxies (DPPs) in the mesh.
    • MeshSubset: Affects a subset of DPPs in the mesh (selected via tags).
    • MeshService: Affects all DPPs of a specific service (selected via name).
    • MeshServiceSubset: Affects a subset of a specific service (selected via name and tags).
    • MeshGatewayRoute: Affects a specific MeshGatewayRoute (selected via name).
    • MeshHTTPRoute: Affects a specific MeshHTTPRoute (selected via name).

    TargetRef Structure

    Each policy contains a single top-level targetRef that identifies the set of DPPs affected by the policy. Traffic direction is then expressed using to (for outbound) or from (for inbound) arrays.

    message TargetRef {
      // Kind of the referenced resource
      // +kubebuilder:validation:Enum=Mesh;MeshSubset;MeshService;MeshServiceSubset;MeshGatewayRoute;MeshHTTPRoute
      string kind = 1;
    
      // Name of the referenced resource
      string name = 2;
    
      // Tags are used with MeshSubset and MeshServiceSubset to define a subset of proxies
      map<string, string> tags = 3;
      
      // Mesh is used with MeshService and MeshServiceSubset to identify the service from another mesh.
      string mesh = 4;
    }
  9. Understand Envoy resource naming conventions in Kuma

    master

    Kuma classifies Envoy resources (listeners, clusters, routes, virtual hosts, and secrets) into two categories to improve observability and management:

    1. System resources: Internal to Kuma. These are used for Kuma's own operations (e.g., readiness checks, metrics, tracing) and are generally only relevant during debugging. They follow a naming pattern starting with system_*.
    2. User resources: Derived from Kuma resources (like TrafficRoute or HTTPProxy). These are the resources users typically monitor and manage.

    Example Naming:

    • System resource: kuma:envoy:admin (legacy) or system_... (new standard).
    • User resource: kri_msvc_mesh-1_us-east-2_kuma-demo_backend_httpport.
  10. Understand MeshTrafficPermission matching semantics

    master

    Unlike standard inbound policies that typically follow a 'first-match-wins' evaluation order, MeshTrafficPermission requires a custom matching algorithm. This is because MeshTrafficPermission must evaluate all matching rules for a single request to ensure that if any matched rule contains a Deny, it takes precedence over any number of Allow rules.

    Standard inbound policies (like those following Gateway API patterns) allow for short-circuiting once a match is found, whereas MeshTrafficPermission cannot short-circuit because a subsequent rule might contain a Deny that overrides previous matches.

  11. Shared telemetry backend resource for observability policies

    master

    Kuma provides a shared telemetry backend resource to centralize OpenTelemetry (OTel) collector configurations. This resource allows operators to define a single collector endpoint and connection settings (such as TLS and authentication) that can be referenced by multiple observability policies: MeshMetric, MeshTrace, and MeshAccessLog.

    This approach prevents endpoint duplication and ensures that updates to the collector address or connection settings (e.g., moving to a different namespace or changing ports) only need to be performed in one place.

  12. Understand Envoy resource and stat naming formats

    master

    Kuma uses three distinct naming schemes for Envoy resources and statistics to ensure consistency, ease of correlation, and low cardinality in monitoring systems:

    1. KRI (Key Resource Identifier): Used for resources that map directly to a Kuma resource (e.g., MeshService, MeshExternalService, MeshHTTPRoute, ZoneIngress, ZoneEgress).
    2. Contextual (self_...): Used for resources local to a single proxy (e.g., inbounds, transparent-proxy passthrough). These include a scope token (dp for sidecar, zi for ZoneIngress, ze for ZoneEgress) to distinguish proxy types.
    3. System (system_...): Used for internal Kuma components. This includes resources derived from Kuma resources (system_<kri>) or purely internal components (system_<namespace>...).