GPU Cluster Networking
(AI-INFRA 100-03)
RoCE v2, PFC/ECN, and fabric fault diagnosis
A hands-on course in the fabric that carries GPU-to-GPU traffic. RDMA does not tolerate packet loss the way TCP does, so the fabric has to be made lossless at the switch and the NIC, and it has to be kept that way. You validate RDMA host readiness, read and set lossless configuration with mlnx_qos, record a healthy bandwidth baseline, then diagnose two live fabric incidents from symptom to root cause without being told the fault class in advance, and fix one of them.
Description
Why AI fabric is different
Synchronized RDMA traffic against loss-tolerant TCP, why RDMA requires a lossless fabric, and how PFC and ECN give Ethernet that property. NVIDIA NetQ as the telemetry surface for Ethernet and RoCE, with UFM named as its InfiniBand counterpart.
Lab: RDMA host readiness
Configuring a lossless fabric
RoCE v2 as RDMA semantics over UDP/IP, per-priority pause frames, and ECN with DCQCN as congestion control that works alongside PFC. Lossless configuration is a two-sided contract between switch and NIC, so both sides get read and set. BlueField-3 on the fabric, and the difference between NIC mode and DPU mode.
Lab: healthy fabric baseline
Fabric observability and fault detection
Fleet-level fabric telemetry as an investigation surface, an evidence hierarchy that separates what decides from what merely corroborates, and the counter signatures that distinguish one fault class from another.
Lab: incident A, blind diagnosis
Diagnosing and fixing misconfiguration
A four-layer recovery check run against a live fault: localize, diagnose, remediate, verify. Configuration drift as the failure mode that scales, and how the same workflow holds at several thousand ports.
Lab: incident B, diagnose and fix. Optional lab: MTU mismatch
FORMAT
2 hours, instructor-led
Short briefing, then terminal time
Two labs are blind diagnoses
Hosted lab environment, nothing to install
Who Should Attend
NETWORK AND INFRASTRUCTURE ENGINEER
You have run TCP/IP, VLANs, and Ethernet switching for years. RDMA and lossless fabric are new, and the instinct that retransmission will cover a dropped packet is the assumption this course spends the most time correcting. You are comfortable at a Linux terminal and with a REST API.
CLUSTER OPERATIONS AND SUPPORT
You are on the receiving end of "the training job got slow." You need the investigation workflow: which telemetry decides, which only corroborates, and what the counter signature of a degraded cable looks like next to a PFC mismatch.
Lab Requirements
TCP/IP, VLANs, L2 and L3 switching
ECMP across parallel paths
Linux CLI: ip link, ethtool, jq
REST basics: curl, JSON, HTTP status codes
