Build GPT From Scratch
Write a transformer line by line in PyTorch
5 9,120 students 4h 20m
Advanced
Free course
AnVisit channel
Taught by Andrej Karpathy
Lessons play via the official YouTube embed. All views credit the creator.
Create a free account to start learning
Every course is free — you just need an account so we can save your progress, bookmarks and certificates.
Let us build GPT: from scratch, in code
Video by Andrej Karpathy · Watch on YouTube
AI lesson summary
Implement a character-level transformer: dataset, embeddings, self-attention, blocks and the training loop.
Key takeaways
- Self-attention is weighted communication
- Residual connections stabilise deep stacks
- Layer norm keeps activations well behaved
Practical exercise
Train the model on your own text file and sample 500 characters.
About this course
Follow along and implement a working GPT: tokenization, self-attention, multi-head attention, residual blocks and training loop.
transformers
pytorch
from scratch
Knowledge check
Generate a free AI quiz to test what you just learned.