Flux v2

Flux's OpenTelemetry implementation is designed to handle the asynchronous and potentially unreliable nature of distributed controllers. It ensures trace continuity through two main mechanisms:

• Asynchronous Event Processing: The system does not assume events arrive in sequential order. Each event independently calculates its Trace ID to either locate an existing parent span or create a new root span.
• Fault Tolerance & Recovery: If the notification-controller experiences downtime or latency, it uses a recovery mechanism during event processing:
  • It attempts to locate an existing root span using the calculated ID.
  • If the root span is found, the new span is attached as a child.
  • If the root span is not found (due to out-of-order arrival or previous failures), it automatically creates a new root span to ensure the event is still traced.

This approach maintains a coherent span hierarchy even when controllers are stateless or events are delayed.

To support multi-tenant workload identity at the object level, Flux associates Flux resources (like OCIRepository, GitRepository, Bucket, Kustomization, etc.) with Kubernetes ServiceAccounts.

By specifying a serviceAccountName in the resource's spec, the Flux controller can exchange a token for that ServiceAccount for a short-lived cloud provider access token. This allows different tenants to have isolated permissions for cloud resources (e.g., different AWS IAM roles or GCP Service Accounts) even when using the same Flux controllers.

Key Requirements:

• The Flux controller's ServiceAccount must have RBAC permissions to create tokens for other ServiceAccounts in the cluster.
• The target ServiceAccount must be annotated with the appropriate cloud provider identity metadata (e.g., AWS IAM role ARN, Azure Client ID, or GCP Service Account email).

# Example: Associating an OCIRepository with a specific ServiceAccount
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: tenant-a-repo
  namespace: tenant-a
spec:
  provider: aws
  serviceAccountName: tenant-a-ecr-sa
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tenant-a-ecr-sa
  namespace: tenant-a
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789123:role/tenant-a-ecr

Workload identity relies on OpenID Connect (OIDC) discovery. The Kubernetes API acts as an OIDC issuer, providing a discovery document at https://kubernetes.default.svc.cluster.local/.well-known/openid-configuration and a JSON Web Key Set (JWKS) at the URI specified in that document.

Cloud providers (AWS, GCP, Azure) use these endpoints to verify the sub (subject) claim in a Kubernetes ServiceAccount token. This allows the cloud provider's Security Token Service (STS) to exchange the Kubernetes token for a short-lived cloud access token.

Key differences in implementation:

• GCP: Can host the JWKS document directly in IAM, and GKE automatically manages the trust relationship.
• AWS: Requires the issuer URL to be reachable by AWS STS; EKS users must manually create an OIDC provider for each cluster (unless using EKS Pod Identity, which is not supported for Flux multi-tenant identity).
• Azure: Typically uses token volume projection where the kubelet mounts a rotating token file into the pod.

Flux uses a deterministic approach to generate Trace IDs to ensure reliable correlation across multiple controllers in a stateless, distributed environment. This allows events from different controllers (like GitRepository and Kustomization) to be grouped into a single coherent trace view in your tracing backend.

Trace ID Generation

The Trace ID is created by concatenating two specific values and passing them through a checksum algorithm (SHA-256 by default):

1. Alert Object UID: A unique identifier guaranteed by Kubernetes.
2. Source's revision ID: Extracted from event payloads (e.g., a Git commit SHA).

Trace Lifecycle Example

1. A GitRepository reconciliation event occurs with a specific revision. The notification-controller captures this, calculates the Trace ID, and starts a new trace/span.
2. A Kustomization then acts on that same revision. It calculates the same Trace ID, allowing it to attach its span as a child of the original trace.
3. The tracing backend displays all these spans as a single, continuous trace.

# Input values
Alert UID: "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
Source Revision: "sha256:2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae"

# Concatenated value format:
# "<Alert-UID>:<source-revision>"
# "a1b2c3d4-e5f6-7890-abcd-ef1234567890:sha256:2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae"

# Resulting Trace ID (after SHA-256):
# "f7846f55cf23e14eebeab5b4e1550cad5b509e3348fbc4efa3a1413d393cb650"

The Request for Comments (RFC) process is used for substantial changes to Flux, such as API additions, breaking changes, security updates, or impactful UX changes. This process ensures a consistent and controlled path for evolution through public discussion and maintainer review.

Substantial changes include:

• API additions (new resource kinds or relationships).
• API breaking changes (field removals or new required fields).
• Security-related changes (permissions, tenant isolation, impersonation).
• Impactful UX changes (e.g., new required inputs to the bootstrap process).
• Dropping capabilities (e.g., sunsetting external service integrations).

The workflow follows these steps:

1. Discussion: Propose the idea via GitHub Discussions and seek feedback at weekly dev meetings.
2. Sponsorship: You must find a maintainer willing to sponsor your RFC.
3. Submission: Open a pull request using the RFC-0000 template.
4. Review: The sponsor labels the PR with area/RFC and initiates a review. At least two maintainers must approve the proposal.
5. Finalization: An RFC number is assigned, and the branch must be rebased with main before merging.
6. Implementation: Once merged, the proposal is implemented. Progress is tracked using the RFC number as a prefix for issues and PRs.