The threshold no one sees
An essay on artificial intelligence, emergence and the question of whether there is a tipping point.
IWhat nobody programmed
A large language model is trained on a single task: predicting the next word of a text. Billions of times, on billions of sentences. An error is measured, the weights in the network are shifted slightly, and the whole thing starts again. That is the complete procedure.
Nobody taught these models to translate, interpret poems, write programs or find a flaw in an argument. These abilities emerged from that one simple task, just as the shape of a flock of birds emerges from three rules. They are, in the precise sense of the word, emergent.
This is not a claim about consciousness, about understanding or about a ghost in the machine. It is an observation about structure: out of simple rules and a very large number of repetitions arises something that is not written in the rules.
IIJumps on the inside
In recent years, research has begun not just to marvel at this process from the outside but to observe it inside the networks. Three findings stand out.
Grokking. In 2022 a group at OpenAI trained small networks on simple arithmetic tasks. At first the networks memorised the examples: perfect on known problems, helpless on new ones. If training continued, long after nothing seemed to be happening any more, there came a moment when the network suddenly solved the new problems as well. It had found the rule. The researchers called this grokking, after a word from a science-fiction novel for deep, intuitive understanding.
Induction heads. In the same year, researchers at Anthropic described how language models acquire the ability to learn from examples in the text. It does not appear gradually during training but within a short stretch, together with a particular circuit in the network that forms by itself. At this point the learning curve shows a distinct bend. The researchers spoke of a phase transition.
A board nobody showed it. In 2023 a group from Harvard trained a model exclusively on move sequences from the board game Othello, as bare lists of square names. The model never saw a board. Nevertheless, a representation of the board was found inside it, one that could be read out and even deliberately altered – whereupon the model suggested different moves. From strings of characters, a picture of the world had emerged that nobody had put there.
IIIThe dispute over the jumps
In 2022 a group led by Jason Wei published a widely noted paper on the emergent abilities of large language models: abilities that are practically absent in small models and suddenly appear beyond a certain size. A year later Rylan Schaeffer and colleagues objected. Many of these jumps, they argued, are produced by the way they are measured. If you only count whether an answer is entirely correct, you see a jump. If you measure how close it comes to the correct one, you often see a smooth curve.
The dispute is instructive because it reveals something about emergence itself. Whether a transition appears sudden also depends on how you look. But it does not dissolve everything. Grokking and induction heads show up as a genuine upheaval inside the network even in smooth measures. There are jumps that lie in the eye of the beholder. And there are jumps that lie in the network.
The band at the top of this page shows a picture of the second kind. Hundreds of small oscillators swing, each at its own pace, and each faintly senses the common rhythm of all the others. This coupling grows stronger slowly and evenly. For a long time little happens. Then, beyond a threshold, the oscillators find one another, and out of the confusion a shared order arises within a short time. It is not a model of a language model. It only shows how a steadily rising dial can produce a sudden transition.
The question is not whether something emerges from these systems that nobody put there. That happened long ago. The question is what emerges next.
IVIs AGI a tipping point?
The public debate revolves around an abbreviation: AGI, artificial general intelligence, a machine that can do everything a human can do intellectually. It is usually imagined as an event, a day on which it arrives. Some think of it as a tipping point in the strict sense. As early as 1965 the mathematician I. J. Good wrote that a machine cleverer than its designers could build even cleverer machines, and so on. The first such machine would be the last invention humanity would ever need to make. That is the logic of a self-reinforcing feedback loop, the logic of an avalanche.
The page on planetary cognition on emergenz.art contradicts this picture. It says: there is no deadline, but a gradual thickening of connections and feedback. No neuron ever noticed that it had become part of a mind.
What is remarkable is that both pictures describe emergence. They do not differ in whether something new arises, but in the dynamics: jump or thickening, avalanche or growth. And the observations so far fit both. Over wide ranges, the models' abilities grow steadily with their size. Inside, there are upheavals nonetheless. To be honest, nobody knows which picture will prove right in the end – or whether it is a mixture of both: a long thickening in which things tip here and there.
VThe way back
Anyone who talks about tipping points must also talk about hysteresis. For here, too, there are paths that cannot be retraced.
A model whose weights have been published cannot be recalled. It has been copied, modified, passed on. An ability that is out in the world does not disappear because someone regrets it. And institutions, professions and habits that have reorganised themselves around a technology do not return to their old state just because the technology is switched off again. This is not a warning of doom. It is the sober observation that the development of artificial intelligence, too, is a system with memory.
VIResponsibility
This is where the real challenge lies. The systems in question are built, but not designed. One determines the architecture, the data, the procedure. What comes out of it one learns only through observation. The research that examines the inside of the models therefore works like a natural science: with experiments, hypotheses, surprises.
Dealing with such systems calls for the lessons found in the other essays here. Where consequences cannot be undone, caution is cheap as long as one is still before the threshold. Hans Jonas called this the heuristics of fear; the essay on politics describes it. And whoever wants to steer an emergent system cannot fix its behaviour, only its rules: testing before release, transparency about what was tested and what was not, limits on where such systems may act without human oversight. In 2024 the European Union made a first attempt to set such rules with its AI Act.
The responsibility does not lie with the machines. It lies with those who set the conditions under which something emerges.
VIICoda: A text about itself
This essay was written with the help of a language model – a system of exactly the kind it is about. The model can report on the research described here. It cannot look inside itself and say what is happening in it. Its statements about itself would merely be another result of the process they are meant to explain.
Perhaps this is the most precise description of the situation. We have built systems that do not see through themselves, and we do not fully see through them either. That is no reason for panic and none for enthusiasm. It is a reason to look very closely – at what emerges, and at the rules under which it emerges.
Sources and further reading
- Good, I. J. (1965): Speculations Concerning the First Ultraintelligent Machine. Advances in Computers 6. — The idea of self-reinforcing intelligence.
- Power, A. et al. (2022): Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets. arXiv:2201.02177
- Olsson, C. et al. (2022): In-context Learning and Induction Heads. Transformer Circuits Thread. — The phase transition during training.
- Wei, J. et al. (2022): Emergent Abilities of Large Language Models. Transactions on Machine Learning Research.
- Li, K. et al. (2023): Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task. ICLR 2023. — The Othello board inside the model.
- Schaeffer, R., Miranda, B. & Koyejo, S. (2023): Are Emergent Abilities of Large Language Models a Mirage? NeurIPS 2023.
- European Union (2024): Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (AI Act).