Interactive model notebook / 01

See what a tiny transformer sees.

Break text into character tokens, expose the causal mask, and watch temperature, top-k, and nucleus sampling reshape the next-token decision.

Live controls / 02

Transformer explorer

Every chart updates from the same prompt and seed.

Experiment controls

Prompt 19/64 characters

Model shape Context window 16 Embedding width 96

Heads Layers

Sampling policy Temperature 0.90 Top-k 12 Top-p 0.92 Seed

Tokens19

Head width24

Parameters340K

Active candidates8

Vocabulary IDs, one character at a time

Head

After filtering

Generated sequence

to be, or not to be

Ready for a deterministic sample.

System map / 03

The trainable implementation follows this same data path.

01
Encode

Character and position embeddings turn a short text window into vectors.
[batch, time] → [batch, time, channels]
02
Attend

Pre-normalized heads mix earlier context while the causal mask blocks the future.
softmax(QKᵀ / √d) · V
03
Sample

Temperature reshapes logits; top-k and top-p remove unlikely candidates.
token ~ categorical(filtered logits)