Bare-Metal GPU Orchestration with NVKE: Factory Architecture and TPM Node Attestation

Deploying dedicated Kubernetes clusters across raw physical servers usually demands juggling separate Cluster API providers, bare-metal provisioning systems, and ad-hoc bootstrap scripts. When modern clusters host high-density graphics processing units, standard orchestrators encounter friction: device topology, non-volatile memory attachments, and remote direct memory access networks require specialized drivers and hardware attestation before nodes can safely join a tenant cluster. Managing this manual lifecycle creates fragility and drift across bare-metal fleets.

Problem

Operating multi-tenant infrastructure on physical hardware introduces several severe operational hurdles. In typical enterprise environments, operators either deploy monolithic, multi-tenant Kubernetes clusters with soft isolation namespaces, or they provision virtual machines on top of a hypervisor layer to achieve hard isolation. Both approaches present severe drawbacks when handling hardware accelerators.

Hypervisors add virtualization overhead, complicate PCIe passthrough for complex GPU topologies, and degrade network throughput across high-speed InfiniBand fabrics. Conversely, running multiple untrusted tenants on a single shared bare-metal control plane exposes workloads to noisy neighbor problems, kernel vulnerabilities, and shared kubelet failures. If a single customer workload triggers an out-of-memory kernel panic or corrupts driver state on a shared worker, adjacent tenant workloads drop offline.

Furthermore, manual bootstrapping routines fail to verify physical machine integrity. Without hardware root-of-trust verification, compromised server nodes or tampered operating system images can receive join tokens and join etcd clusters undetected, violating zero-trust isolation boundaries across fleet infrastructure.

Existing solutions

Platform teams traditionally attempt to solve bare-metal orchestration using standard Cluster API providers such as CAPM3 (Metal3) combined with Ironic, or proprietary orchestration tooling delivered by public cloud vendors. While functional for general compute instances, these traditional stacks reveal critical limitations in production bare-metal environments.

First, standard Cluster API setups require significant external scaffolding: DHCP servers, PXE boot configurations, TFTP endpoints, and complex state machine orchestrators. Debugging a stalled deployment during an operating system provisioning cycle frequently turns into an exercise in tracing network broadcast packets across isolated VLANs.

Second, credential distribution remains a persistent vulnerability in conventional bootstrap tooling. Standard node joins rely on static tokens or pre-shared keys generated during cluster initialization. If these tokens leak during machine provisioning, unauthorized systems can register rogue compute nodes into the control plane.

Platform teams need clean separation: a centralized management plane that provisions isolated workload clusters, while ensuring nodes cryptographically attest their hardware identity before receiving cluster credentials.

To address this operational gap, the open-source NVIDIA Kubernetes Engine (NVKE) project introduces a self-hosting architecture. Instead of treating bare metal as a generic virtualization target, NVKE acts as a dedicated factory for Kubernetes clusters. It uses an operator CLI called nvkectl to stand up a temporary seed cluster, bootstraps a central management plane, and provisions independent workload clusters on demand.

NVKE integrates hardware verification into the node registration lifecycle. Rather than passing static join tokens across cleartext metadata services, nodes must attest their identity using physical or virtual Trusted Platform Modules (TPM) via SPIFFE/SPIRE before receiving join credentials.

git clone https://github.com/dsx-ai-factory/nvke.git
cd nvke
make build

# Inspect the compiled management plane binaries
ls -la bin/
# Outputs: agent, controller, management-engine, nvkectl, operator-api, tenant-api

Compiling the repository yields nine distinct modular binaries, establishing a clear architectural boundary between the infrastructure controller plane, the tenant-facing APIs, and the local node agent running on physical hosts.

Future with AI

As machine learning workloads continue to scale in parameter count and model complexity, autonomous AI agents will increasingly interact directly with underlying hardware infrastructure. Today, an operator manually determines node pool sizing, schedules maintenance windows, and executes cluster lifecycle upgrades using configuration manifests. Tomorrow, infrastructure orchestration agents will interface directly with management APIs to dynamically allocate bare-metal partitions based on real-time computational load.

In this operational model, an autonomous cluster agent can monitor GPU health metrics, detect hardware degradation on specific PCIe lanes, and automatically cordon the affected node. The agent can request a replacement machine through the management engine API, initiate SPIFFE cryptographic attestation via hardware TPM, and migrate running distributed training jobs without human intervention.

Decoupling the management plane from individual workload clusters creates clean blast walls. When autonomous agents operate in dedicated tenant environments, security boundaries prevent compromised containers or rogue automation scripts from accessing adjacent hardware pools. Systems like NVKE demonstrate how infrastructure management moves away from brittle scripting and toward verifiable, hardware-attested bare-metal orchestration.