Appendix A: the command reference for AI infrastructure

A few commands do most of the work: in the lesson's example, ten commands made up 780 of the last 1,000 typed. Every command answers one of four questions for one layer of the machine: what is there, is it healthy, what is using it, and what happened.

In Daveth Pro (AI Engineering)

What this page answers

Which Linux commands should I learn first?

The everyday ten or twenty. In the example, ten commands made up 780 of the last 1,000, while 54 others shared the rest, many used once or twice on the day something broke. Read it in “Your own top ten”

How do I choose the right Linux command?

Pick the layer you suspect, from the machine and its disks up to the GPU and containers, and ask one of four questions: what is there, is it healthy, what is using it, and what happened. Read it in “Every command answers a question”

Your own top ten

Bash keeps a list of the commands you've typed, called your history. On Ubuntu it keeps the last 1,000. Here's an AI engineer's history after a few busy weeks, counted by command name. The first line takes the second word of each history line (the command), sorts them, counts the repeats, and shows the ten most common:

$ history | awk '{print $2}' | sort | uniq -c | sort -rn | head
    142 nvidia-smi
    131 cd
    118 ls
     97 git
     74 uv
     61 tail
     55 docker
     43 ssh
     38 grep
     21 du
$ history | awk '{print $2}' | sort -u | wc -l
64

The counts are a realistic example. The commands are real: run the same two lines on your own machine.

So this person used 64 different commands in their last 1,000. About how many of those 64 did most of the work?

  • About ten
  • About thirty
  • All 64, used roughly equally

Answer. Add up the ten counts: 780 of the last 1,000 commands were one of those ten. The other 54 commands shared the remaining 220, many of them used once or twice: a dkms status on the day the driver broke, an ss -tlnp to find out what grabbed port 8000.

That shape is the whole idea of this page. Learn the everyday ten or twenty until your fingers know them. For everything else, don't memorize: know what question you're asking, and look up the command that answers it. Most of this page is built for that lookup.

Every command answers a question

This course treats an AI machine as layers: the machine itself, its files and disks, the network, the running programs and services, your Python code, the GPU, and on top, containers and clusters. When something is wrong, you ask the same four plain questions of whichever layer you suspect. What's there? (Is the disk, the GPU, the package even present?) Is it healthy? What's using it? (Who is eating the memory, the space, the port?) And what happened? (What do the logs say?)

Almost every command you'll meet answers one of those four questions for one layer. The grid below gives one good answer for each pair. Read across a row to look at one layer from every side, or down a column to ask one question of every layer.

Drawing: What's there?, Is it healthy?, What's using it?, What happened?, Machine, lscpu, free -h, vmstat 1, sudo dmesg -T | tail, Files and disks, lsblk -f, df -h, du -sh * | sort -h, tail -f train.log, Network, ip -br a, ping -c 3 gpu-box, ss -tlnp, journalctl -u ssh, Processes and services, pstree -p 4120, systemctl status llm, ps aux --sort=-%mem, journalctl -u llm -f, Python and code, type -a

Notice how often the last column is a log. When you don't know where to start, “what happened?” is usually the fastest question, because Linux writes down nearly everything: the kernel in dmesg, services in journalctl, your own jobs in their log files. And notice the GPU row: two of its cells are nvidia-smi, once with -L to list the GPUs and once plain to check them. One tool, asked different questions.

Reading any command line

Before the lists, one skill that makes every entry readable: splitting a command line into its parts. Every line is built from the same few pieces. The command is the program to run. Options (also called flags) start with - or -- and change how it behaves. Arguments are what it works on, often files or folders. And the plumbing connects things: | (a pipe) hands one program's output to the next, and > saves output into a file. sudo at the front means “run this as the administrator”.

Here's a real line, coloured by part. It finds the five biggest things in /data:

Drawing: run as administrator, sudo, du, -sh, /data/*, |, sort, -h, tail, -n, 5, >, big.txt, Commands, the programs that run:, du, sort, tail, Options, change how each behaves:, -sh: total, in GB or MB, Arguments, what to work on:, /data/*, big.txt, Plumbing, | hands output to the next,, > saves it into a file, Left to right: size of each item in /data, sort by size, keep the last 5, save to big.t

Two things here trip up nearly everyone once. First, the * in /data/* is not seen by du at all. Your shell swaps it for the list of everything in /data before du starts (chapter 2 calls this expansion). Second, sudo applies to du only. The pipes and the > are handled by your own shell, as your own user. That second one has a famous consequence:

Predict. You want to lower swappiness for a training box, so you run sudo echo 'vm.swappiness=10' > /etc/sysctl.d/99-ai.conf. You used sudo. What happens?

Answer. bash: /etc/sysctl.d/99-ai.conf: Permission denied. Your shell sets up the > before it runs anything, and your shell is you, not root. sudo only made echo run as root, and echo doesn't need it.

The fix is to make the writing itself happen as root: echo 'vm.swappiness=10' | sudo tee /etc/sysctl.d/99-ai.conf. tee is a program that writes what it receives into a file, and now it's the one running under sudo. Add -a to add to a file instead of replacing it.

The twenty you'll use every day

If you learn only one table on this page, make it this one. These twenty cover the everyday work on a GPU machine: checking the GPU, finding space, watching jobs, reaching services, and reading logs. Each row says what the command tells you in plain words, and the chapter where it's explained properly. Try each one on your own machine once, today, so the output looks familiar when you need it in a hurry.

CommandWhat it tells you, or doesCh.
nvidia-smiDriver working? GPU memory used, and by which processesCh 14
df -hFree space on each diskCh 8
du -sh * | sort -hWhat's taking the space hereCh 8
free -hMemory left to hand out (the “available” column)Ch 11
htopLive CPU, memory and processesCh 11
tail -f train.logWatch a log as it growsCh 3
grep -i error train.logFind the lines that matterCh 3
ps aux --sort=-%mem | headThe biggest memory usersCh 5
kill -TERM 4120Stop a process, letting it save firstCh 5
tmux new -s trainKeep a job alive after you disconnectCh 5
systemctl status llmIs a service up, and its last log linesCh 10
journalctl -u llm -fFollow a service's logCh 10
sudo dmesg -T | tailLatest kernel messages: drivers, disks, GPU errorsCh 7
ss -tlnpWhat's listening on which portCh 9
ssh -L 8888:localhost:8888 gpu-boxOpen a remote service on your laptopCh 9
uv run python train.pyRun in the project's environmentCh 12
type -a python3Which program a name really runsCh 12
git statusWhat changed in your projectCh 13
docker ps -aWhich containers exist and their stateCh 17
docker logs -f llmFollow a container's outputCh 17

Look at the spread: GPU, disk, memory, processes, services, network, Python, git and containers all appear. That's the point. The everyday set isn't twenty GPU commands. It's one or two good questions for each layer, so that when something breaks you can check every layer quickly and find the one that answers “no”.

Predict. Your 64 GB machine is running a data-prep job. free -h shows free 1.2Gi and available 41Gi. Is the machine about to run out of memory?

Answer. No. Linux uses spare memory to keep recently read files ready (the page cache), which makes the second read of a dataset much faster. That memory counts as used, so “free” looks tiny. But the kernel gives it back the moment a program needs it. “available” is the number that matters: 41 GB can still be handed out.

Worry when available gets small and swap starts climbing. Down the Load goes through how to tell.

Which command answers this?

Here's the grid from the top of the page with every command from the course behind it. You'll get every command for that pair, what each one tells you, and the chapter it comes from. There's a fifth question as well, change it, for commands that do something rather than look at something.

Drawing: Each command is filed under the layer it looks at, which isn't always the chapter it's from. py-spy comes from the monitoring chapter, but it looks inside your Python code.

The first gives you commands that test whether things work (the driver answers, the GPU isn't slowing itself down). The second gives commands that only report versions and parts. When you're stuck, people often run the second kind and wonder why it didn't help. A version number tells you what's installed, not whether it's working.

Every chapter's commands

This is the full reference, in course order. Type part of a command or a plain word (“space”, “port”, “driver”, “log”) to search both the commands and their descriptions. Each chapter heading links to its lesson, where the command is explained with real output.

Drawing: Commands are written the way you'd type them, with example names (a process ID 4120, a service called llm, a node called gpu07). Swap in your own.

Search for “version” and see how many ways there are to ask it: the kernel, glibc, the driver loaded in memory, the driver on disk, the CUDA toolkit, PyTorch's CUDA build. Half the troubleshooting in this course is noticing that two of those disagree. Down the CUDA Stack explains which pairs have to match, and Appendix B walks through what happens when they don't.