A language model has one job: look at the text so far and guess the next
token (here, a word). It does this by counting patterns it saw in its training data —
nothing more. Play with the tiny model below to see where that works beautifully, and where it
falls apart.
Temperature 0 = always pick the top word (safe, but loops). Higher = more
random (creative, but wanders). This is the same knob you set on a real model.
The model's guesses for the next word
Try these — capability vs. limitation
Click a starter, then press Predict a few times and watch the confidence
bars.
What you're seeing
Capability (green bars, high confidence). On common phrases the model has seen
thousands of times (once upon a ___), one word dominates. This is what LLMs are
great at: well-worn paths — summarizing, reformatting, finishing familiar patterns.
Limitation — sparse data (orange bars, flat/low confidence). Feed it rare or novel
words it barely saw and the guesses spread out or collapse to a fallback. A real LLM does the
same on niche, post-cutoff, or obscure topics — it still answers, just less reliably.
Limitation — "sounds true" ≠ "is true". The model picks the word that fits the
pattern, not the word that is correct. It will happily complete a plausible-sounding
but wrong sentence. Prediction is about fluency, not facts.
Limitation — repetition. Set temperature to 0 and auto-write: the model often loops.
Real decoders add penalties to fight this (you'll meet repeat_penalty later).
An unplugged-style, in-browser demo for CS357: Foundations of Artificial Intelligence.
It trains a small trigram word model (with bigram/unigram back-off) entirely in your
browser on the short sample text below — no network, no real LLM. The point is the
mechanism, not the scale. View source to read the ~60 lines of model code.
Released under CC BY-NC-SA 4.0.