rasbt/LLMs-from-scratch
📚 Tutorialrasbt
Build a ChatGPT-like LLM from scratch with PyTorch, step by step.
This GitHub repository provides a step-by-step implementation of a ChatGPT-like LLM entirely from scratch. Starting with fundamental concepts, it guides users through building a tokenizer, implementing multi-head attention, constructing transformer blocks, and creating the full GPT architecture. The project includes pretraining on a small dataset and finetuning for specific tasks, with detailed code and explanatory text in Jupyter notebooks. It uses PyTorch for all computations and includes code for data loading, training loops, and evaluation. The repository is designed for educational purposes, making complex LLM concepts accessible to practitioners with basic deep learning knowledge.
💡Highlights
- ├─97k+ GitHub stars
- ├─Step-by-step Jupyter notebooks
- └─Covers pretraining and finetuning
🎯For
- ├─ML engineers
- ├─AI researchers
- └─Students learning NLP