AI Data and Training Infrastructure
Every AI Data and Training Infrastructure lesson in the Linux for AI Engineering course, in the order you study them. Each page opens about the first half of its lesson to read, some the whole of it; the full lessons are in the plan.
Data engineering for AI: the small-files problem, Parquet and WebDataset shards, and GPU input pipelines
Why GPUs sit idle waiting for data: the small-files problem, JSONL, Parquet and WebDataset shards, getting data in and checking it.
Single-node training: GPU memory accounting, LoRA, ZeRO and FSDP, checkpoint resume and evaluation
Why training runs out of GPU memory: the four piles of weights, gradients, optimizer state and activations, and how to shrink each.
Distributed training: parallelism strategies, RDMA and RoCE, NCCL collectives and Slurm
Why two-node training can be slower than one: all-reduce, NCCL over InfiniBand and RDMA, checking the fabric, and Slurm jobs.
