Capstone E: AI infrastructure system-design interviews

Infrastructure design interviews reward the same five steps in order, because they mirror how systems are built: know the load before choosing hardware, and try to break a design before calling it good. The numbers usually decide it; the lesson's company assistant fits on one GPU, with a second for safety.

In Daveth Pro (AI Engineering)

What this page answers

How do I answer an AI infrastructure design interview question?

Start from the numbers. In the lesson's example, 10% of staff in the busiest hour is about 5.6 requests and 1,700 tokens per second, and one H100 running an 8B model writes about 5,000, so one GPU plus a spare fits. Read it in “The first sentence”

What is the shape of a good system design answer?

Five steps in a fixed order that mirror how real systems are built: you cannot choose hardware before you know the load, and you cannot claim a design is good until you have tried to break it. Read it in “The shape of a good answer”

The first sentence

The interviewer says: “Our company has 10,000 employees. Design a system that lets them all use an internal chatbot running on our own GPUs.” You have 45 minutes and a whiteboard.

The interview questions on this page are typical examples, not from any one company. Every number comes from an earlier lesson, linked where it's used.

What should your first sentence be?

  • “I'd run vLLM on Kubernetes with autoscaling behind a gateway.”
  • “Before I design anything: how many use it at once, how long are prompts and answers, and how fast must answers start?”
  • “We'd need a cluster of eight H100s to start with.”

Answer. The numbers decide everything. If 10% of staff use it in the busiest hour, sending 20 questions each with 300-token answers, that's about 5.6 requests and 1,700 written tokens per second. One H100 running an 8B model writes about 5,000 tokens per second when it's batching well (Down the Queue). So the whole company fits on one GPU, with a second one for safety. If instead they want a 70B model with 8,000-token documents in every prompt, the answer changes completely.

Interviewers aren't testing whether you know the tools' names. They're testing whether you know which numbers decide the design, and whether you'll ask for them before committing. The rest of this page is about doing exactly that, quickly and out loud.

The shape of a good answer

Almost every infrastructure design interview rewards the same five steps, in this order. It works because it mirrors how real systems are built: you can't choose hardware before you know the load, and you can't claim a design is good until you've tried to break it.

Drawing: 1 · Clarify, users, peak load,, lengths, latency,, budget, privacy, 2 · Estimate, tokens per second,, memory, GPUs,, storage, network, 3 · Design, boxes and arrows,, each one justified, by step 2, 4 · Break it, a GPU dies, traffic, triples, a deploy, goes wrong, 5 · Trade-offs, cost, what you'd, measure, what you'd, do with more time, Spend roughly 5, 10, 15, 10 and 5 minutes of a 45-minu

Notice where the course fits. Step 2 is Down the Bits, Down the Memory and Down the Queue: bytes per parameter, KV cache per token, tokens per second per GPU. Step 3 is the serving, Kubernetes, cluster and gateway lessons. Step 4 is the troubleshooting halves of every lesson: Xids, NCCL hangs, full disks, cold starts. You already know the material. The capstone is about using it in order, under time pressure, while talking.

Numbers to know by heart

Estimating quickly needs a handful of numbers you don't have to look up. These are the ones that come up again and again, each with the lesson where it's explained. Round them freely in an interview: being within a factor of two, and saying so, is exactly what's expected.

NumberValueLesson
Weights in memory2 bytes per parameter at 16-bit, 1 at FP8, about 0.6 at 4-bitDown the Bits
Full training stateAbout 16 bytes per parameter, before activationsDown the Memory
KV cache, Llama 3.1 8B128 KB per token (16-bit); 70B: 320 KB per tokenDown the Bits
Writing speed, one userAbout 70% of memory speed ÷ bytes read per tokenDown the Bits
Writing speed, batchedOne H100, 8B model: about 5,000 tokens/s across 64 chatsDown the Queue
GPU memory speedRTX 4090 about 1 TB/s; H100 about 3.35 TB/s; H200 4.8 TB/sDown the Box
LinksPCIe 4.0 x16 about 25 GB/s; NVLink hundreds of GB/s; 400 Gb/s InfiniBand about 50 GB/sDown the Bus, Down the Wire
Failure rateBig clusters lose a GPU or node every few hours, so checkpoint and plan restartsDown the Fleet
Cold startMinutes, mostly loading tens of GB of weights from storageDown the Image, Down the Cloud

Two habits make these numbers useful. First, always convert the business question into tokens per second and bytes: users and requests mean nothing to a GPU. Second, check memory before speed, as in Down the Box: a design that doesn't fit is wrong however fast it would be.

Sizing a serving platform

The single most common question is the hook's: how many GPUs to serve this many people? Here's the estimate you'd do on the whiteboard, step by step. Peak requests per second come from the number of people active in the busiest hour. Tokens per second follow from the answer length. How many requests are in flight at once follows from how long each one takes (that's Little's law: things in a system = arrival rate × time each spends inside). Then GPUs for the load, GPUs for memory, and spares so one failure doesn't take the service down:

Drawing: Planning estimates. GPU throughput: one H100 writes about 5,000 tokens/s with an 8B model when batching well (Down the Queue), and proportionally less for models that read more bytes per token. GPUs are sized to run about 70% busy, leaving room for bursts. A 70B FP8 model needs at least 2 GPUs for room to batch. Each user is assumed to read at 40 tokens/s. Real capacity needs a load test.

With the default answers, 10,000 people need about 1,700 tokens per second at peak, about one H100 with an 8B model, so two for safety. Switch to the 70B model and it's 6: four for the load, run at about 70% busy so bursts don't queue, and a spare pair. Now set questions to 60 per hour, typical of agents that call the model many times per task: the load triples and the design changes with it. That's the sentence interviewers love: “the answer depends on these two numbers, and here's how it changes.”

Five drills

These five questions cover most of what AI infrastructure interviews ask, in one form or another. Read only the question first, set a 30-minute timer, and answer out loud or on paper using the five steps. Then open the rest: what to ask, the numbers that matter, a reasonable design, the follow-ups you should expect, and which lessons to revisit.

Drawing: One reasonable answer each, not the only one. Interviewers care more that your choices follow from your numbers than about which tools you name.

Notice that every drill's design is mostly decided by one or two numbers from the estimate step: tokens per second for serving, bytes per parameter for training, documents and queries per second for RAG, utilization for cost, and “what changed?” for incidents. If you find yourself describing boxes before you've said those numbers, stop and go back a step.