NVIDIA Cloud Functions

The worker-init component is an initialization process that prepares the NVIDIA Cloud Functions (NVCF) worker environment before a task container begins execution. Its primary responsibilities include:

• Downloading model and resource artifacts from NGC.
• Bootstrapping the ESS (secret) agent configuration.
• Emitting initialization metrics.

The NVCF GRPC Proxy facilitates communication between clients and NVIDIA Cloud Functions (NVCF) workers by forwarding GRPC requests over QUIC/TCP channels established via CONNECT endpoints. It exposes the following endpoints:

• HTTP/3 CONNECT endpoint: http://localhost:10084/v1/proxy
• HTTP/1 CONNECT endpoint: http://localhost:10086/v1/proxy (configurable)
• HTTP/1-2 forwarder: http://localhost:10081

The NVCF NATS Auth Callout Service is a pluggable authentication service designed to integrate with NATS Server's authorization callout feature. It uses a multi-plugin architecture to authenticate NATS clients through various mechanisms.

Supported Authentication Plugins

• NKey Plugin: Performs cryptographic NKey signature verification.
• OAuth Plugin: Handles OAuth2 JWT Bearer token authentication with JWKS validation.
• Webhook Plugin: Delegates authentication to external HTTP-based services.

Key Capabilities

• Observability: Supports OpenTelemetry/Lightstep for distributed tracing, Prometheus for metrics, and provides built-in health/readiness endpoints.
• Configuration: Supports environment variables, CLI flags, YAML config files, and JSON secrets files.
• Security: Integrates with Vault agent for secure credential management.

Stargate is a control plane and HTTP router designed for inference servers. It manages request routing by maintaining local state based on models and routing keys.

Core Workflow:

1. Pylons (sidecars) register local inference servers with Stargate.
2. Clients send OpenAI-compatible HTTP requests to Stargate.
3. Stargate forwards requests over an established QUIC tunnel to the appropriate Pylon.

Key Components (Crates):

• stargate: The main server binary.
• pylon / pylon-lib: Sidecar CLI and library used to register inference servers.
• stargate-k8s-router: An optional router for backend-facing gRPC, Raw QUIC, and WebTransport traffic.
• stargate-bench: A benchmark runner.

NV Spring Boot provides shared libraries and a Bill of Materials (BOM) designed for both internal/managed and self-hosted application deployments. It acts as a parent project that extends spring-boot-starter-parent 4.0.7 and provides a suite of nv-boot-starter-* libraries to standardize common functionality.

Baseline Versions:

• Spring Boot: 4.0.7
• Spring Cloud: 2025.1.2 (Oakwood)
• Java: 25

NVIDIA Cloud Functions (NVCF) is a platform designed for deploying, managing, and running GPU-accelerated workloads at scale. It provides a system to route inference, streaming, and other GPU-intensive tasks to worker clusters, allowing users to scale demanding workloads without managing the underlying infrastructure themselves.

Key capabilities include:

• Unified control plane: Manages and routes requests across multi-region GPU clusters.
• Load-balanced workload routing: Balances inference, streaming, and custom workloads based on worker availability.
• Multi-protocol support: Handles various protocols for different client needs.
• Multi-cluster autoscaling: Scales workloads from zero to maximum capacity across clusters.
• Mixed GPU support: Supports diverse GPU types across clusters.
• Health checks and telemetry: Monitors worker status and request latency.

NVCA is a Kubernetes agent designed to manage NVIDIA Cloud Functions (NVCF) Bring-Your-Own-Compute (BYOC) clusters. It handles workload scheduling, queue message processing (via SQS/NATS), storage management, and lifecycle operations for serverless GPU workloads.

Key capabilities include:

• Workload Management: Queue-based management for serverless tasks.
• Lifecycle Management: Uses a MiniService controller for Helm chart lifecycles.
• Resource Discovery: GPU resource discovery within the cluster.
• Observability: Comprehensive Prometheus metrics for monitoring queue operations, capacity, and health.
• NVCA Operator: An automated component that manages the deployment, upgrades, and health monitoring of the NVCA agent itself.

NVIDIA Cloud Functions (NVCF) is a multi-module service designed to manage the lifecycle of inferencing workloads on GPU-powered worker nodes. It provides both REST APIs and gRPC worker APIs.

The service is composed of two primary modules:

• nvcf-core: Contains the core business logic.
• nvcf-service: A Spring Boot executable that depends on nvcf-core.

CI/CD processes for this service are handled via GitHub workflows that build and test targets using Bazel.

NVCF Container Cache is a high-performance caching system designed to reduce latency and bandwidth for container images and large assets. It operates in two complementary modes within a single StatefulSet pod using an OpenResty/Nginx engine with dynamic TLS certificate generation:

1. Container Cache: Caches container image pulls using the OCI/Docker registry protocol. It is compatible with both containerd and CRI-O runtimes.
2. Proxy Cache: Caches S3 objects, NGC assets, HuggingFace models, and Nucleus files. This is achieved via DNS-based Man-in-the-Middle (MITM) redirection.

Key capabilities include transparent fallback to upstream registries if the cache is unavailable, automatic runtime configuration via a DaemonSet, and disk-pressure awareness using a min_free watermark.