Appendix F: the glossary of AI infrastructure terms
AI infrastructure is where operating systems, networking, high-performance computing, statistics and machine learning meet, and each brought its own words. So the same short word can mean several things: a LoRA rank is the size of the add-on matrices, while a distributed training rank is a process number.
What this page answers
What does rank mean in AI?
In LoRA, the rank, such as r=16, is the size of the small add-on matrices trained instead of the whole model. In distributed training, a rank is the number of one process in the job, counting from 0. Read it in “Two ranks in one morning”
How do I tell which meaning of an AI infrastructure term is meant?
Ask which layer of the machine you are in, from hardware and the operating system up through libraries, frameworks, serving and clusters. Kernel appears in the operating system and in the libraries, and node means a cluster machine or a memory region on a server. Read it in “Where the words live”
Two ranks in one morning
Two log lines from the same morning. The first is from setting up a fine-tune, the second from a multi-GPU training job that stopped:
>>> model = get_peft_model(base, LoraConfig(r=16, target_modules="all-linear")) >>> model.print_trainable_parameters() trainable params: 41,943,040 || all params: 8,072,204,288 || trainable%: 0.5196 [rank3]:[E ProcessGroupNCCL.cpp] [Rank 3] Watchdog caught collective operation timeout
Real output formats from PEFT and PyTorch. The parameter counts are what LoRA with r=16 on every linear layer of Llama 3.1 8B gives.
In LoRA, r is called the rank. The second line talks about Rank 3. Is it the same idea?
- Yes: both are about how a model is split up
- No: one is the size of an add-on, the other is a process's number
- One of them is a misuse of the word
Answer. In LoRA, the rank (r=16) is the size of the small add-on matrices trained instead of the whole model. It's a term from linear algebra, and it decides how many numbers you train: here 41.9 million, about half a percent of the model. In distributed training, a rank is simply the number of one process in the job: Rank 3 is the fourth process (counting from 0), usually the one driving the fourth GPU. Both uses are correct. They just come from different fields that happen to meet on AI machines.
That's why this glossary exists. Unknown words are easy, because you know to look them up. The dangerous ones are words you think you know, used in a chapter where they mean something else.
Where the words live
Every word in this glossary belongs to one layer of the machine, the same layers the course climbs through: hardware at the bottom, then the operating system, the system services, the runtimes and libraries, the training frameworks, the models and serving, and the clusters on top. When you meet a new word, the first useful question is “which layer is this?”. Here are a few from each:

Notice the pink words. “Kernel” appears twice, in the operating system and in the libraries. “Rank” has both of its meanings inside the framework layer: a process's number and the size of a LoRA add-on. “Node” means a machine in a cluster and a memory region on a server. Knowing the layer you're in is usually enough to pick the right meaning, which is why the glossary below shows each word's layer next to it.
The glossary
Every term from the course, alphabetically. Each entry links to the chapter that explains it with real examples.

The “Models and serving” layer matters most: it's the vocabulary of most AI news, and nearly every word there connects to memory. The KV cache, quantization, context window, prefill and decode are all, underneath, about how many bytes have to sit in GPU memory and how fast they can be read.
Continue the lesson
The whole lesson, with its drawings and its questions. The rest of this section and 5 more after it.


