LLM Training in AI Engineering

Every LLM Training lesson in the AI Engineering course, in the order you study them. Each page opens about the first half of its lesson to read, some the whole of it; the full lessons and their workbooks are in the plan.

2 lessons

  1. Pre-training at scale: MinHash deduplication, Chinchilla scaling laws, bf16, ZeRO and pipeline parallelism

    LLM pre-training at scale: MinHash deduplication of web text, scaling laws, the Chinchilla compute trade-off, GPUs, mixed precision and parallelism.

  2. Post-training: SFT, LoRA and QLoRA, distillation, RLHF, DPO and KL-regularised RL

    How a base model becomes an assistant: supervised fine-tuning, LoRA and QLoRA, distillation, reinforcement learning, RLHF and DPO, and reward hacking.