A language model has one job: look at the text so far and guess the next
token (here, a word). It does this by counting patterns it saw in its training data,
nothing more. Play with the tiny model below to see where that works beautifully, and where it
falls apart.
Temperature 0 = always pick the top word (safe, but loops). Higher = more
random (creative, but wanders). This is the same knob you set on a real model.
The model's guesses for the next word
Try these: capability vs. limitation
Click a starter, then press Predict a few times and watch the confidence
bars.
What you're seeing
Capability (green bars, high confidence). On common phrases the model has seen
thousands of times (once upon a ___), one word dominates. This is what LLMs are
great at: well-worn paths such as summarizing, reformatting, and finishing familiar patterns.
Limitation: sparse data (orange bars, flat/low confidence). Feed it rare or novel
words it barely saw and the guesses spread out or collapse to a fallback. A real LLM does the
same on niche, post-cutoff, or obscure topics. It still answers, just less reliably.
Limitation: "sounds true" ≠ "is true". The model picks the word that fits the
pattern, not the word that is correct. It will happily complete a plausible-sounding
but wrong sentence. Prediction is about fluency, not facts.
Limitation: repetition. Set temperature to 0 and auto-write: the model often loops.
Real decoders add penalties to fight this (you'll meet repeat_penalty later).
An unplugged-style, in-browser demo for CS357: Foundations of Artificial Intelligence.
It trains a small trigram word model (with bigram/unigram back-off) entirely in your
browser on the short sample text below: no network, no real LLM. The point is the
mechanism, not the scale. View source to read the ~60 lines of model code.
Released under CC BY-NC-SA 4.0.