Production AI Infrastructure
Every Production AI Infrastructure lesson in the Linux for AI Engineering course, in the order you study them. Each page opens about the first half of its lesson to read, some the whole of it; the full lessons are in the plan.
LLM serving: prefill and decode, KV cache budgets, continuous batching, PagedAttention and gateways
How one GPU serves sixty-four conversations: prefill and decode, the KV cache as the budget, continuous batching and measuring a server.
Kubernetes for AI: the GPU Operator, GPU sharing, DRA and gang scheduling
Why Kubernetes shows no GPUs on a GPU node, extended resources and the device plugin, the GPU Operator, sharing GPUs and DRA.
Cloud GPU infrastructure: spot and on-demand capacity, time to a ready model, Terraform and paying only for work
Are cloud spot GPUs worth it? On-demand, reserved and spot pricing, what interruptions cost, boot time to a ready model, and Terraform.
Edge AI on ARM64: Jetson, NPUs, fitting models on small devices and remote updates
Why PyTorch on a Jetson installs but finds no GPU, how pip picks a wheel on ARM64, JetPack and tegrastats, NPUs and small models.
