Neural Networks in AI Engineering
Every Neural Networks lesson in the AI Engineering course, in the order you study them. Each page opens about the first half of its lesson to read, some the whole of it; the full lessons and their workbooks are in the plan.
Supervised learning: least squares, logistic regression and softmax classifiers
The four parts of every machine learning system, fitting a line by least squares and gradient descent, then logistic and softmax regression, with overfitting.
Backpropagation: a reverse-mode autograd engine, nonlinearities, SGD, momentum and Adam
Backpropagation from scratch: why networks need non-linear activations, how a reverse-mode autograd engine works, and how SGD, momentum and Adam step.
Subword tokenization and embeddings: BPE, WordPiece and word2vec skip-gram
How text becomes tokens and vectors: Unicode and UTF-8 bytes, byte-pair encoding merges, WordPiece, and word2vec embeddings learned from neighbours.
mini_lm v1: a Bengio-style fixed-window character MLP, tanh-gain initialization and checkpointing
Build mini_lm v1, a character-level neural language model: sliding-window training data, an MLP over embeddings, careful initialization and checkpoints.
Inductive biases: convolutions, residual connections, GRUs and backpropagation through time
Inductive biases explained: why CNNs use convolutions and residual connections for images, and how RNNs and GRUs keep a memory, with backprop through time.
