GPU Cloud Observability Is a Topology Problem

Oct 19, 2026 4 min read cybersecurity, Development

GPU cloud observability is broken. Not because the tools don’t exist — because the abstraction layers lie.

The Topology Problem

Traditional observability assumes a flat network: hosts talk to hosts, metrics flow to collectors, traces stitch together via request IDs. GPU clusters violate every assumption. A single training job spans dozens of nodes, each with 8 GPUs connected via NVLink, each GPU with its own memory fabric, each node connected via InfiniBand or RoCE to a spine-leaf fabric that has its own congestion domains, its own adaptive routing, its own failure modes.

Your monitoring stack sees “node CPU at 40%.” The reality: GPU 3 on node 7 is starved because the NVLink between GPU 3 and GPU 4 saturated during all-reduce, while the InfiniBand link to the parameter server is quietly dropping packets due to a misconfigured ECN threshold. The metrics are green. The job is 3x slower than baseline.

What You Actually Need to See

Per-GPU memory bandwidth utilization, not just “GPU memory used”
NVLink/NVSwitch counters: throughput, errors, replay counts
InfiniBand/RoCE port counters: symbol errors, link downtime, congestion signals
Collective operation latency breakdown: all-reduce, all-gather, broadcast per rank
Power and thermal state per GPU — throttling hides in plain sight

None of this lives in standard node exporters. You need DCGM, you need fabric managers, you need switch telemetry, and you need them correlated by job ID, not by hostname.

The Correlation Gap

Here’s what a typical incident looks like. Alert fires: “Training throughput dropped 40%.” On-call engineer checks Grafana. Node metrics look fine. Container metrics look fine. Kubernetes says pods are healthy. Three hours later, someone notices a single NIC on a leaf switch showing 15% packet loss. The job ran across 64 nodes. The loss affected 8 of them. The collective operation retried silently. No error surfaced.

The hard part isn’t collecting metrics. It’s knowing which metrics belong to the same logical operation across five different infrastructure layers.

A Practical Starting Point

Start with DCGM and NCCL debugging. Enable NCCL debug logging on a sample job:

export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL
export NCCL_TOPO_DUMP_FILE=/tmp/nccl_topo.xml
torchrun --nproc_per_node=8 train.py

Parse the topology dump. It tells you which GPUs share NVLink, which cross PCIe, which cross the network. Cross-reference with ibstat on each node to map GPU-to-NIC affinity. Then instrument your training loop to emit collective timestamps:

import torch.distributed as dist
import time

dist.all_reduce(tensor)
torch.cuda.synchronize()
print(f"RANK {dist.get_rank()} all_reduce latency: {time.time() - start:.3f}s")

Ship those logs to a time-series DB tagged by job ID, rank, and collective type. Now you can ask: “Show me all-reduce latency for rank 3 across the last 100 steps.” When rank 3 spikes, you know exactly which GPU, which node, which NIC, which switch port to check.

The Tooling Still Has Gaps

Prometheus scrapes node metrics every 15 seconds. NVLink counters update every microsecond. You either sample aggressively and miss microbursts, or you drown in cardinality. DCGM exports via dcgm-exporter but doesn’t correlate with Slurm job IDs or Kubernetes pod UIDs. NVIDIA’s nsight-systems captures beautiful traces — for one job, interactively. Not for production fleets.

The vendors are catching up. NVIDIA’s UFM Telemetry, Mellanox’s collective offload counters, AMD’s ROCm SMI — they exist. But stitching them together remains a custom engineering project at every shop running GPU clusters at scale.

What Changes When You Get It Right

You stop guessing. You see a collective slowdown, trace it to a specific NVLink error counter incrementing, correlate with a firmware version on that specific GPU, roll back the firmware, confirm the counter stops. Job throughput recovers. That’s the loop. Not “investigate for three hours.” Not “add more dashboards.”

The industry treats GPU observability as a metrics problem. It’s a topology problem. Until your monitoring understands the physical and logical topology of the job — not the cluster, the job — you’re flying blind with green dashboards.