Appendix F: the glossary of AI infrastructure terms

AI infrastructure is where operating systems, networking, high-performance computing, statistics and machine learning meet, and each brought its own words. So the same short word can mean several things: a LoRA rank is the size of the add-on matrices, while a distributed training rank is a process number.

In Daveth Pro (AI Engineering)

What this page answers

What does rank mean in AI?

In LoRA, the rank, such as r=16, is the size of the small add-on matrices trained instead of the whole model. In distributed training, a rank is the number of one process in the job, counting from 0. Read it in “Two ranks in one morning”

How do I tell which meaning of an AI infrastructure term is meant?

Ask which layer of the machine you are in, from hardware and the operating system up through libraries, frameworks, serving and clusters. Kernel appears in the operating system and in the libraries, and node means a cluster machine or a memory region on a server. Read it in “Where the words live”

Two ranks in one morning

Two log lines from the same morning. The first is from setting up a fine-tune, the second from a multi-GPU training job that stopped:

>>> model = get_peft_model(base, LoraConfig(r=16, target_modules="all-linear"))
>>> model.print_trainable_parameters()
trainable params: 41,943,040 || all params: 8,072,204,288 || trainable%: 0.5196

[rank3]:[E ProcessGroupNCCL.cpp] [Rank 3] Watchdog caught collective operation timeout

Real output formats from PEFT and PyTorch. The parameter counts are what LoRA with r=16 on every linear layer of Llama 3.1 8B gives.

In LoRA, r is called the rank. The second line talks about Rank 3. Is it the same idea?

  • Yes: both are about how a model is split up
  • No: one is the size of an add-on, the other is a process's number
  • One of them is a misuse of the word

Answer. In LoRA, the rank (r=16) is the size of the small add-on matrices trained instead of the whole model. It's a term from linear algebra, and it decides how many numbers you train: here 41.9 million, about half a percent of the model. In distributed training, a rank is simply the number of one process in the job: Rank 3 is the fourth process (counting from 0), usually the one driving the fourth GPU. Both uses are correct. They just come from different fields that happen to meet on AI machines.

That's why this glossary exists. Unknown words are easy, because you know to look them up. The dangerous ones are words you think you know, used in a chapter where they mean something else.

Where the words live

Every word in this glossary belongs to one layer of the machine, the same layers the course climbs through: hardware at the bottom, then the operating system, the system services, the runtimes and libraries, the training frameworks, the models and serving, and the clusters on top. When you meet a new word, the first useful question is “which layer is this?”. Here are a few from each:

Drawing: Clusters and operations, Node, Kubernetes, Slurm, Drain, Goodput, Models and serving, Token, KV cache, Quantization, TTFT, RAG, Training frameworks, Gradient, LoRA, FSDP, Rank, Checkpoint, Runtimes and libraries, CUDA, NCCL, Kernel (GPU), Wheel, Container, System services, systemd, Unit, Port, SSH, Cron, Operating system, Kernel (Linux), Process, Page cache, cgroup, glibc, Hardware, GPU,

Notice the pink words. “Kernel” appears twice, in the operating system and in the libraries. “Rank” has both of its meanings inside the framework layer: a process's number and the size of a LoRA add-on. “Node” means a machine in a cluster and a memory region on a server. Knowing the layer you're in is usually enough to pick the right meaning, which is why the glossary below shows each word's layer next to it.

The glossary

Every term from the course, alphabetically. Each entry links to the chapter that explains it with real examples.

Drawing: Definitions are deliberately short and plain; the linked lessons give the full mechanism. “See also” points at a related word or its twin.

The “Models and serving” layer matters most: it's the vocabulary of most AI news, and nearly every word there connects to memory. The KV cache, quantization, context window, prefill and decode are all, underneath, about how many bytes have to sit in GPU memory and how fast they can be read.