Transformers in AI Engineering
Every Transformers lesson in the AI Engineering course, in the order you study them. Each page opens about the first half of its lesson to read, some the whole of it; the full lessons and their workbooks are in the plan.
Scaled dot-product self-attention: queries, keys and values, causal masking, multi-head attention and the KV cache
Self-attention from scratch: queries, keys and values, the square-root scaling, causal masking, multi-head attention, FlashAttention and the KV cache.
The decoder-only transformer block: MLP, residual stream, pre-norm LayerNorm and RMSNorm, RoPE
Build a transformer block: why attention needs an MLP, the residual stream, LayerNorm and RMSNorm placement, and position encodings such as RoPE.
