Operating AI Infrastructure at Scale

Every Operating AI Infrastructure at Scale lesson in the Linux for AI Engineering course, in the order you study them. Each page opens about the first half of its lesson to read, some the whole of it; the full lessons are in the plan.

3 lessons

  1. AI observability: DCGM metrics, inference SLIs, MFU, cardinality and Xid errors

    Why GPU utilization shows 100% while the GPU does little: DCGM counters, inference SLIs, model FLOPs utilization and request tracing.

  2. Performance engineering: the roofline model, Nsight profiling, NUMA placement and kernel optimisation

    Why a faster GPU barely speeds up a job: profiling with Nsight Systems, the roofline, fixing the timeline first, and NUMA placement.

  3. Fleet reliability: configuration drift, checkpoint cadence, canary upgrades, model registries and runbooks

    GPU fleet reliability: why large training jobs fail every few hours, how often to checkpoint, drifting nodes and safe upgrades.