CS336 Spring 2024 Executable Lectures

This repository contains the executable lectures for Spring 2024 offering of CS336: Language Models from Scratch.

An executable lecture is a Python program whose execution delivers the content of the lecture. The recommended way to consume an executable lecture is to step through a lecture (e.g., lecture 1) in VSCode. Each step corresponds to a bullet point (analogous to a slide build), and of course since everything is code, you can seamlessly execute code samples and inspect the computed variables.

Note: this code has been tested on Python 3.12 on CPUs, A100s and H100s.

Setup

Check out the repo:

git clone https://github.com/stanford-cs336/spring2024-lectures

Install the necessary packages:

pip install -r requirements.txt

Optional configuration:

export OPENAI_API_KEY=...
export TOGETHER_API_KEY=...
export WANDB_API_KEY=...
nvidia-smi

The code will run without a GPU, but many parts of the lecture do depend on the GPU.

Check that the code works:

python lecture_01.py
python lecture_02.py
...

Viewing lectures

Let's start with lecture 1 as an example.

1. Open up main.py in vscode (code lecture_01.py).
2. Set a breakpoint on the main function and press F5 to start stepping through it.
3. Press F11 to dive into a section.
4. Press F10 to step over a line.
5. Mouse over variables to see their values.
6. Open view.html to see an execution log of the lecture, which includes any images that can't be rendered in vscode (this part is a bit clunky).