Since late 2024, language models have been “reasoning” before they answer. I retrace here the path by which reasoning came to these statistical machines, from the discovery of 2022 to Anthropic’s interpretability research, so that this understanding may serve our uses and shed light on the collective choices ahead.
Next-word prediction already contained reasoning
In a previous article, Qu’est-ce qu’un LLM ? Explication du pourquoi et du comment !, I described how large language models are built. Over several months, on very powerful machines, the model learns to predict the next word of billions of human sentences, and each prediction error adjusts the billions of weighted parameters that connect words, then concepts, to one another through the successive layers of abstraction of the Transformer architecture.
It is worth pausing on this architecture, because it is what made generative AI possible. It was born in 2017, in a paper by Google researchers whose title has remained famous, “Attention is all you need”. The neural networks that processed language until then read texts word after word, in order, carrying along a compressed memory of what came before, a memory that faded as the sentence went on, so that a word located far upstream eventually ceased to weigh on what followed. These machines, unable to hold long-range dependencies, could only generate short texts, which quickly lost their coherence.
The Transformer replaces this sequential reading with a mechanism called “attention”, through which every word in a passage assesses, all at once and in parallel, its weighted relations with all the other words of the passage, the pronoun finding its antecedent, the verb its subject, an ambiguous word discovering from the surroundings of the sentence which of its meanings applies, whatever the distance between them. Predicting the next word is therefore not a short-sighted guess made from the last few words read; it draws on the entire network of relations that the text at hand contains. And since these attention computations can be run in parallel on graphics processors designed for massive calculation, training on gigantic corpora became materially possible. This twofold property, holding together all the relations of a text and being able to learn at very large scale, is what allowed today’s generative AI to come into being, and the acronym GPT, Generative Pretrained Transformer, carries this architecture in its very name.
Correctly predicting the next word, past a certain level of demand, nevertheless requires far more than memory. To rightly complete a sentence that begins with “therefore”, “yet” or “as a consequence”, to continue a mathematical proof or a computer program without derailing them, the model had to absorb into its parameters something of the regularities of human inference, since the texts it learned from are steeped in them. The statistics of language thus preserve the trace of the reasoning that shaped those texts, the way a footprint keeps the shape of the step, so that reasoning abilities were already deposited, in a latent state, in the large models of the early 2020s, before anyone had measured their extent.
“Let’s think step by step”, the discovery of 2022
In January 2022, a team of Google researchers, around Jason Wei, showed that when the model is given a few examples in which the answer is preceded by written-out intermediate steps, rather than the answer alone, its performance on calculation and logic problems leaps forward. A few months later, a team from the University of Tokyo and Google found that one could even do without the examples, and that it was enough to add to the question a sentence such as “let’s think step by step” for the model to write out the steps by itself and find the right answer far more often. This technique was named the “chain of thought”. Nothing was modified in the model, no retraining, one simple added sentence.
A language model generates its answer word by word, and each generated word immediately joins the text it reads in order to generate the next one. When the model writes out intermediate steps, it therefore gives itself, in the form of text, the footholds it needs for what follows, it poses a sub-question, writes down the result, and this written result becomes a piece of data available for the next step, exactly as we write down a carry when doing an addition on paper. Writing serves as its working memory. There is also a more material reason, which lies in the fact that generating each word mobilises a fixed amount of computation, one pass through the layers of the Transformer, no more and no less, whether the question is simple or very hard. Writing longer before concluding therefore means, for the machine, computing longer, and the chain of thought gives it computing time where the direct answer forced it to solve everything in a single pass. Reasoning was thus not added to language models, it lay dormant in the statistics of language, and that incantatory sentence of 2022 simply gave it room to write itself out.
September 2024, models trained by reward
For two years, this technique remained a user’s trick, a way of formulating one’s requests better. The next step consisted in bringing the chain of thought into training itself. In September 2024, OpenAI released o1, presented as the first “reasoning model”. The method employed changes in nature, since it is no longer a matter of showing the model examples of human reasoning to imitate, but of letting it produce attempts at solving, then rewarding it when its final answer is correct, in domains where correctness can be checked automatically, mathematics and programming above all. This is what is called reinforcement learning. The model discovers by trial and error the ways of proceeding that most often lead to the reward, and reinforces them.
What this method produced went beyond what its designers expected of it, and it was the Chinese lab DeepSeek that made this visible. In January 2025, DeepSeek released R1, and above all published its method, in an open scientific paper. An experimental version of the model, named R1-Zero, had been trained by reinforcement alone, without any human example of reasoning. As training went on, the researchers saw behaviours appear that no one had programmed, the model interrupting itself mid-solution to check a previous step, trying another route when the first one led nowhere, spontaneously devoting more text to hard problems than to simple ones. The team named the appearance of these self-corrections the “aha moment”, with an astonishment perceptible in the paper itself. The sole pressure of the final reward had been enough to make emerge, in a statistical machine, gestures we associate with reflection, such as doubting a result and going back to check it.
R1 also produced a shock of another order. The model was released as open source, its chain of thought was displayed in full, where o1 hid its own, and its training and usage costs amounted to a fraction of those of the American models. The demonstration that reasoning of this level could be obtained at lower cost and made public sent American technology stocks tumbling in late January 2025, with Nvidia’s share price losing 17% in a single day.
February 2025, Claude and the dosing of reflection
One month later, in February 2025, Anthropic released Claude 3.7 Sonnet and made a different design choice, which seems to me the richest in meaning. At OpenAI as at DeepSeek, the reasoning model was a separate product, a different model from the one used for ordinary conversations. Claude 3.7 was presented as the first “hybrid” model; it is the same model that answers simple questions immediately and that, on request, takes a time of “extended thinking” before answering, with an adjustable thinking budget, from a few hundred to several tens of thousands of words of intermediate thought. Anthropic justified this choice with an analogy I find apt: we do not switch brains depending on whether we are asked the time or asked to solve a difficult problem, we dose our reflection. Reasoning thereby ceases to be a category of machine and becomes a gradual modality of one and the same system. Anthropic also chose to make this reflection visible, in a raw form, not rewritten to please, accepting that it may at times be hesitant or disconcerting to read.
Over the course of 2025, this architecture became widespread, at Google with the Gemini “Thinking” models, at OpenAI with the o3 series and then the integration of reasoning into its standard models, and in a multitude of open models derived from the method published by DeepSeek. By the end of 2025, prior, dosable reflection had become an industry standard. What was a formulation trick in 2022 became, in three years, a central property of these machines.
Anthropic watches its models while they reason
It remains to understand what this displayed reasoning actually is, and it is here that Anthropic has produced the work that enlightens me most, in a singular double gesture, since the same laboratory builds these models and publishes the most serious warnings against placing excessive trust in them.
In March 2025, Anthropic’s interpretability team published “Tracing the thoughts of a large language model”, a study that observes, with techniques inspired by biology, the internal circuits that activate inside Claude while it answers. One discovers there that the model plans further than the next word, for instance that it chooses the rhyme of a verse before starting to write the line that leads to it. One also discovers, and this is the point that interests me most, that the account the model gives of its own calculation does not match what its circuits show. Asked how it adds two numbers, Claude describes the school method, with the carries, whereas observing its circuits reveals several parallel calculation paths, none of which resembles the method it recounts. The model learned from our texts the way humans describe an addition, and it is this description it produces when questioned, not an introspection of its own workings.
A second study, published in April 2025, measures what the researchers call the “faithfulness” of chains of thought. The protocol consists in discreetly slipping a hint into the question asked, then watching whether the model, when it uses this hint to answer, mentions it in its displayed reflection. Claude 3.7 Sonnet mentioned it in only 25% of cases, DeepSeek R1 in 39%. In the great majority of cases, the models constructed a plausible justification of their answer without saying what had steered it. The chain of thought is therefore a draft that helps the model compute, and this draft does indeed improve the answers, without for all that being a confession of its workings or a window opened onto its mechanism.
This finding instructs me, about ourselves included. Psychology established long ago, since the work of Richard Nisbett and Timothy Wilson at the end of the 1970s, that humans readily produce after-the-fact justifications of decisions reached by other routes, and that the account we give of our reasoning is closer to reconstruction than to a verbatim record. This is not, for all that, a reason to anthropomorphise the machine, for its resemblance to us on this point has an identifiable cause, it learned from our texts the form of our justifications, including their share of reconstruction. The displayed reasoning of these models is a literary genre they have mastered, the genre of reasoning, inherited from us.
From the wording of our requests to the defence of readable drafts
From this journey, I draw a few practical consequences for our uses, in cultural work as elsewhere:
- The first concerns the way we call upon these machines. Since a model’s reasoning is a piece of writing that works on what lies within its context window, a bare question produces bare reasoning. Giving the model material, that is, documents to lean on and examples of what one is looking for, means giving its reasoning something to reason with, and I developed this point in the article Le contexte, clé de l’intelligence artificielle en entreprise culturelle. Giving it room as well, by activating extended thinking, means giving it computing time, with the energy cost that comes with it and that invites us to reserve this use for questions worth the effort.
- The second concerns the reading of chains of thought. They are fascinating documents, in which one sees a machine hesitate and retrace its steps, and I recommend that anyone who uses these tools go and read at least one, for this reading changes the way one looks at these systems. The faithfulness studies invite us, at the same time, to read them the way one reads a writer’s draft, revealing but composed, rather than as the machine’s inner truth.
- The third concerns the limits. The reasoning of these models is language working on language. It excels in domains where the validity of an answer is inscribed in formal rules, mathematics and programming above all, those very domains where reinforcement learning was able to reward it, and it remains without direct purchase on the world, without sensory experience that would come to contradict an elegant but false deduction. I have often written that these machines think from language and not from an experience of the world, and reasoning models change nothing in this anthropological given, they refine the work of language upon itself. This is the meaning of the critique that Yann LeCun, Turing Award laureate and one of the fathers of deep learning, has been addressing to these models for years, models which in his view predict words and not the world, and do not constitute a path towards an intelligence comparable to ours; he left Meta at the end of 2025 to found AMI Labs in Paris, a startup devoted to “world models”, systems trained on video and perception to learn the dynamics of the physical world rather than that of texts. The debate between these two paths remains open, and it bears on this very question, what a reasoning can reach when it is made of language alone.
- The last consequence looks to the future, and it is political. We are living through a singular moment in which we can read the thinking of the machines that answer us, because this thinking writes itself in human language, before our eyes, with most LLMs. This moment rests on technical and industrial choices that can be revised, and research is already exploring so-called “latent” reasoning, which would unfold within the model’s internal representations, without passing through readable words, in order to gain efficiency. In July 2025, researchers from otherwise competing laboratories, OpenAI, Google DeepMind and Anthropic together, co-signed a position paper describing the possibility of monitoring chains of thought as “a new and fragile opportunity”, one that could close depending on the training choices to come, and Anthropic itself defends in its publications the readable chain of thought as an instrument for overseeing models that ought to be preserved. The readability of machine reasoning is a fragile good, which can go out for reasons of optimisation, and it seems to me that the world of culture and education, which knows what access to texts is worth, has a say in ensuring that the machines that populate our lives remain machines whose drafts we can read.
These machines born of writing will thus have learned to reflect by writing to themselves, prolonging that anthropological inversion I have noted before, they who build language by starting from writing where the human being starts from orality. Their reasoning is a rereading, ours is rooted in lived experience, and the two can work together provided we do not mistake one for the other.
