DOCSEARCHbyAlgolia
Submit Docs

Fleet Documentation

repository·main·Indexed 23 days ago

https://github.com/rancher/fleet

A GitOps and HelmOps engine designed to manage Kubernetes deployments at scale across multiple clusters. Fleet treats all deployment sources—including YAML, Helm, and Kustomize—as Helm charts to ensure consistent and auditable deployments. The documentation covers installation via Helm charts (fleet, fleet-crd, and fleet-agent), end-to-end testing with k3d, and the use of the Fleet Benchmark Suite to measure GitOps, targeting, and deployment performance.

Tokens
40.2K
Snippets
60
Records
212
Agent score
83%

What's inside Fleet

  1. Use the Fleet Benchmark Suite for performance analysis

    main

    The Fleet Benchmark Suite is designed to evaluate the performance and scalability of Fleet. Use this suite for the following purposes:

    • Performance Regression Detection: Identify if new Fleet versions introduce performance degradation.
    • Controller Optimization: Measure the direct impact of changes made to the controller code.
    • Scalability Validation: Verify Fleet's ability to handle increasing numbers of clusters and deployments.
    • Capacity Planning: Determine the resource requirements needed for specific deployment scales.
    • Historical Tracking: Build a database of performance metrics to monitor trends over time.

  2. What is Fleet?

    main

    Fleet is a GitOps and HelmOps engine designed for managing deployments across multiple clusters at scale. It provides high visibility and control over cluster state through constant monitoring.

    Key capabilities include:

    • Deployment Management: Fleet can manage raw Kubernetes YAML, Helm charts, Kustomize, or combinations thereof.
    • Unified Engine: Regardless of the source format, Fleet dynamically converts all resources into Helm charts, using Helm as the underlying engine to ensure consistency, control, and auditability.

  3. Understand the Fleet Benchmark Suite core experiments

    main

    The suite measures performance across three core stages of the Fleet bundle lifecycle:

    1. GitOps Performance (GitRepo to Bundle Creation)

    Measures the speed of creating bundles from Git repositories.

    • create-1-gitrepo-50-bundle: 50 bundles from 1 GitRepo.
    • create-50-gitrepo-50-bundle: 50 bundles from 50 GitRepos.

    2. Targeting Performance (Bundle to BundleDeployment)

    Measures how long it takes to target bundles to clusters.

    • create-50-bundle: Creates 50 bundles and measures targeting time.
    • create-150-bundle: Creates 150 bundles targeting all clusters (skipped if >1000 clusters).

    3. Deployment Performance (BundleDeployment to Ready Resources)

    Measures the time from BundleDeployment creation to resources reaching a ready state.

    • create-1-bundledeployment-10-resources: 1 BundleDeployment resulting in 10 resources per cluster.
    • create-50-bundledeployment-500-resources: 50 BundleDeployments creating 500 resources per cluster.

  4. Use structured logging with controller-runtime

    main

    Fleet uses controller-runtime for structured logging. Instead of using fmt.Sprintf to create unstructured strings, you should use constant log messages paired with variable key-value pairs. This allows for better filtering and programmatic analysis of logs.

    When implementing logging, use logger.V(verbosity).Info(msg, key1, value1, key2, value2) to attach context to your messages.

    // Structured
logger.V(1).Info(
    "Reconciling bundle, checking targets, calculating changes, building objects",
    "generation",
    bundle.Generation,
    "observedGeneration",
    bundle.Status.ObservedGeneration,
)

  5. Best practices for writing reliable Fleet tests

    main

    To maintain high quality and prevent flaky tests, follow these architectural recommendations:

    Test Hierarchy

    1. Unit Tests: Use these as the primary method for testing logic combinations. They are lightweight and easy to set up/tear down.
    2. Integration Tests: Use these when unit tests cannot cover a specific case. They allow for flexible mocking while remaining easier to debug than E2E tests.
    3. End-to-End (E2E) Tests: Use these as a last resort, as they require spinning up full clusters and additional infrastructure.

    E2E Test Guidelines

    • Minimize Resources: Deploy the smallest possible amount of resources (e.g., use test charts containing only ConfigMaps) to speed up execution and reduce load.
    • Randomize Names: Randomize namespaces, release names, and deployed resource names to prevent conflicts between concurrent test runs.
    • Resource Cleanup: Always clean up created resources after execution. Use Gomega's BeforeEach and AfterEach hooks to ensure a clean state or perform pre-execution cleanup.
    • Avoid time.Sleep: Never use time.Sleep(<duration>) to wait for conditions. Instead, use Gomega's Eventually for asynchronous assertions.
    • Prefer Function-based Assertions: When using Eventually, pass a function that accepts a Gomega object as a parameter. This allows assertions to be made inside the function, providing much more detailed error output than a simple boolean check.
    • Avoid Ephemeral Dependencies: Do not rely on logs or events (or the absence of them). If necessary, scale deployments down and back up before the test run to ensure a predictable state.

  6. Relationship between Fleet and Rancher objects

    main

    As of Rancher v2.6+, Fleet is a required component. Fleet clusters are mapped to specific native Rancher object types. Understanding these mappings is essential for managing cluster lifecycle and workspaces via the Rancher API.

    Object Mappings

    • clusters.fleet.cattle.io/v1alpha1 maps to clusters.provisioning.cattle.io/v1 and clusters.management.cattle.io/v3.
    • fleetworkspaces.management.cattle.io/v3 maps to standard Kubernetes namespaces/v1.
    ┌───────────────────────────────────┐  ==  ┌────────────────────────────────────┐  ==  ┌──────────────────────────────────┐
│ clusters.fleet.cattle.io/v1alpha1 ├──────┤ clusters.provisioning.cattle.io/v1 ├──────┤ clusters.management.cattle.io/v3 │
└────────────────┬────────────────┘      └───────────────────┬────────────────┘      └──────────────────────────────────┘
                 │                                             │
                 └──────────────────────┬──────────────────────┘
                                        │
                          ┌─────────────▼────────────────────────┐
                          │                                      │
       ┌──────────────────▼──────────────────────┐  ==  ┌────────▼──────┐
       │ fleetworkspaces.management.cattle.io/v3 ├──────┤ namespaces/v1 │
       └─────────────────────────────────────────┘      └───────────────┘

  7. Detect performance regressions using the Database Comparison feature

    main

    The benchmark suite can compare current results against a historical database of previous runs to detect performance regressions and trends.

    How to build a database

    To enable comparison, move your generated benchmark JSON files into the directory specified by the --db flag (default is db/).

    # Run benchmarks (generates a file like b-2024-01-15_10:30:45.json)
fleet-benchmark run

# Move the report to your database folder
mv b-2024-01-15_10:30:45.json db/

    Running comparisons

    When you run the report command with a populated --db directory, the tool calculates statistical metrics including Mean Duration, Standard Deviation, and Z-Score.

    • Negative Z-Score: The current run is faster than the historical average (improvement).
    • Positive Z-Score: The current run is slower than the historical average (potential regression).
    # Run benchmarks with a custom database location
fleet-benchmark run --db=/path/to/my-benchmarks/

# Generate report comparing against the database
fleet-benchmark report --db=/path/to/my-benchmarks/ -i b-latest.json

# View detailed statistics including StdDev and Z-scores
fleet-benchmark report --db=db/ -i b-latest.json --stats
    fleet-benchmark report --db=db/ -i b-latest.json --stats

  8. Identify performance bottlenecks in Fleet

    main

    Fleet's design philosophy places most business logic in the local cluster via the fleet-controller. When troubleshooting performance, distinguish between three primary bottleneck types:

    1. Compute-based: High CPU usage in pods.
    2. Memory-based: High memory consumption in pods.
    3. Network-based: High network traffic. Note that high compute usage can sometimes be a symptom of high network traffic (e.g., a pod processing excessive incoming data).

    Monitoring Strategy:

    • Local Cluster: Monitor the fleet-controller via pod logs, network traffic, and resource usage.
    • Downstream Clusters: Monitor the fleet-agent deployments. Because agents perform Kubernetes API requests back to the local cluster, you must monitor inbound traffic to the local cluster from agents, in addition to the standard outbound traffic from the fleet-controller.

  9. Important notes for running Fleet Benchmarks

    main

    When running the Fleet Benchmark Suite, keep the following behaviors in mind:

    • Reproducibility: The benchmarks use a fixed random seed to ensure results can be reproduced.
    • Error Handling: Tests are designed to fail fast on the first error encountered to provide quick feedback.
    • Memory Tracking: By default, memory metrics track the benchmark test process. To track the Fleet controller's memory, you must enable metrics collection via Prometheus endpoints.

  10. How Fleet behaves when managing Fleet (Nested Fleet)

    main

    In scenarios where Fleet is managing another Fleet instance (e.g., Rancher managing Rancher, or Hosted Rancher), every managed Fleet cluster will run two fleet-agent deployments. These agents communicate with two distinct fleet-controller deployments across different namespaces.

    Deployment Structure

    • Local Fleet Cluster: Contains the primary fleet-controller (in cattle-fleet-system) and a fleet-agent (local) (in cattle-fleet-local-system).
    • Managed Fleet Cluster: Contains a fleet-agent (downstream) (in cattle-fleet-system) to receive instructions from the local controller, and its own fleet-controller (in cattle-fleet-system) and fleet-agent (local) (in cattle-fleet-local-system) to manage its own downstream clusters.
    • Downstream Cluster: Contains a fleet-agent (downstream) (in cattle-fleet-system) managed by the Managed Fleet Cluster's controller.

  11. Understand Fleet Benchmark measurement metrics

    main

    The benchmark suite measures performance across three primary stages and collects several categories of metrics:

    Performance Stages

    1. GitOps Performance: Measures the time from GitRepo creation to Bundle creation.
    2. Targeting Performance: Measures the time from Bundle creation to BundleDeployment creation.
    3. Deployment Performance: Measures the time from BundleDeployment creation to resources reaching a Ready state.

    Collected Metrics

    • Duration Metrics: Time-based measurements for the stages above.
    • Resource Metrics: Counts and scales of resources created.
    • Memory Metrics: Tracks the memory usage of the test process (Note: this tracks the benchmark process itself, not the Fleet controller, unless specific metrics collection is enabled).
    • Controller Metrics: If enabled, these use Prometheus metrics endpoints to track controller-specific performance.
    • Environment Context: Metadata regarding the environment in which the benchmark was run.

  12. Configure public_hostname for k3d

    main

    The public_hostname variable allows scripts to set up k3d with port forwardings (80 and 443) that redirect host ports to services inside the cluster.

    • Default: 172.18.0.1.sslip.io (points to the Docker network gateway).
    • Usage: Set this to a DNS record pointing to the public interface IP of your host.
    • Warning: Some routers use DNS rebind protection which may block .sslip.io domains. You may need to use a custom A record or a different wildcard DNS resolver.
    • Customization: You can provide additional port forwarding arguments via the k3d_args variable.