Curiosity as an operating system

A few days ago, an article by Andrey Vecherniy titled “The Bayesian Brain Concept, or Why This Headline Right Now Is Your Hallucination” was published on Habr. https://habr.com/ru/companies/ru_mts/articles/1029856/. The article excellently explains the essence of Karl Friston’s free energy principle and Anil Seth’s corresponding interpretation: the brain sits in the darkness of the skull box, never sees the world directly, and therefore everything we experience as reality is its own guess about what is outside.

This picture has a part that is usually omitted in popular retellings. It perfectly explains how the brain sees the world. But it says almost nothing about why it does this. Why a living system does not sit quietly. Why a well-fed, warm, danger-protected person reads articles about the brain. Why a one-year-old child who has no frustrated needs sticks their fingers into a power outlet. Why a rat that is given everything it needs in the laboratory immediately starts sniffing corners in a new cage.

I will try to talk about this from the perspective of psychotherapy and neuroscience. If the article about the Bayesian brain is about its epistemic side (how the brain sees), then this one is about its affective and motivational side (what drives it). Over the past forty years, neuroscience, psychology and engineering have independently been moving towards the same answer. And the answer turned out to be not what European philosophy had expected for the last two hundred years.

Prologue: biography as a method

Sigmund Freud had six children. This is a fact worth paying special attention to when you read that libido lies at the basis of mental life. The theory in which sexual drive is the source of everything was developed by a person who, obviously, found this drive persistent enough to consider it universal. This does not refute Freud. But it is not a coincidence either.

Friedrich Perls, the founder of Gestalt therapy, wrote his early programmatic book "Ego, Hunger and Aggression" (1942) under rather specific circumstances: he first emigrated to Amsterdam, then to South Africa, lived in cramped conditions and, it seems, was not inclined to gastronomic abstinence. In this book, he proposed replacing libido as the central driving force of the psyche with hunger.

Alfred Adler suffered from rickets as a child, was short in stature, grew up in a family with an older brother named Sigmund (we get the irony) and proposed a theory in which the leading psychic motive is the compensation for the inferiority complex through striving for superiority.

This is not an exposé. All of them were serious thinkers, and each described something truly important. But they did not have the capabilities of modern science. Only their own intellect, the introspective research method, curiosity about how they themselves are structured, and the courage to say that all other people are structured exactly the same way).

Dopamine is not what people thought it was

Let's start with the experiment that has been the foundation of this entire story for seventy years. In 1954, James Olds and Peter Milner implanted an electrode into a rat's brain and connected it to a lever. Each press of the lever delivered a weak current to that spot. When the electrode was placed in the mesolimbic dopamine pathway, the rat began pressing the lever and could not stop. Without food, without sleep, without water — up to two thousand presses per hour, until it collapsed from exhaustion. For a long time, this area of the brain was called the “pleasure center” or “reward system”

In 1997, Wolfram Schultz, Peter Dayan and Read Montague published the article «A neural substrate of prediction and reward» in which they described the activity of individual dopamine neurons in monkeys during learning experiments. The pattern they observed turned out to be as follows: a dopamine neuron encodes the discrepancy between what the brain expected to receive and what it actually received. Surprise, positive or negative. Unexpected good outcomes trigger a burst of activity; unexpected bad outcomes cause a drop in activity. When the outcome matches the prediction, there is no activity. The assumption that the dopamine pathway functions as a «reward system» was incorrect. It is a system of anticipation. And anticipation turned out to be stronger than all other motivations.

Concurrent with Schultz's work, Kent Berridge and Terry Robinson from the University of Michigan made another discovery that complemented the first. In a large 1998 review article published in Brain Research Reviews — «What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience?» — they proved that dopamine has no «pleasure» function whatsoever. A completely different system is responsible for that. If dopamine is blocked in a rat, it stops wanting food, but if food is placed in its mouth, it still likes it — its pleasure facial expressions remain intact. Conversely, if dopamine levels are artificially raised, the rat will desperately strive for food, but based on its reactions, it gains no pleasure from it. Berridge named these two functions «wanting» and «liking». Literally translated, these are «desire» and «pleasure liking». Dopamine underlies wanting. Opioids and endocannabinoids underlie pleasure. These are two distinct systems that work together, but can be separated.

This, by the way, explains the most terrifying phenomenon of addiction. A heroin addict in the late stages no longer gets pleasure from the drug. But they cannot stop seeking it. Their wanting system is hijacked by the substance and operates at full capacity, while the pleasure system has long been exhausted. They want, but they do not get. This is a pure form of search fixed on a single predictor, and this is hell. Because wanting and enjoying are different functions, and wanting can be separated from everything else.

Ancient neural circuits

Estonian-American neuroscientist Jaak Panksepp systematically sorted out these findings. In his 1998 book «Affective Neuroscience», he described seven basic emotional systems — ancient neural circuits shared by all mammals, including humans, and present in homologous forms in birds and reptiles. Each has its own neurochemistry, anatomy, and behavioral profile. He referred to them in capital letters to distinguish them from ordinary emotions: SEEKING (searching), RAGE (rage), FEAR (fear), LUST (lust), CARE (care), PANIC/GRIEF (panic and grief), and PLAY (play).

Of these, SEEKING holds a special place. It is not a system for searching for something specific — food, a partner, or shelter. It is searching itself: autonomous, outward-directed excited movement. Subjectively, it can be felt as "something interesting is about to happen". Structurally, it is similar to a motherboard, through which all other systems receive energy and direction. Without SEEKING, rage does not seek out an opponent. Without SEEKING, fear does not seek an exit. Without SEEKING, lust does not seek a partner. Desire itself is just neural potential; for it to turn into behavior, a searching system is needed to pick it up and direct it.

Brain as a prediction machine

For what follows to make sense, we need three elements from the overall framework of the predictive brain. It is covered in detail in the article mentioned above; here it is presented briefly, exactly as much as necessary.

First: the brain is a generator of assumptions about reality. It constantly builds predictions of what incoming sensory signals should be like, and what we perceive as "reality" is, for the most part, an internal model adjusted by discrepancies with what arrives from outside. The main flow of information goes top-down: higher levels tell lower levels what to expect, lower levels compare this with incoming input and send only the error upwards.

Second: the task of the brain is to minimize this error. There are two ways to solve this problem. You can update the model: "the world is not as I thought, I'll think differently" - this is perception and learning. Or you can explore the world - this is action. This idea has a technical name active inference, and it is primarily developed in the works of Karl Friston, in particular in the review article "The free-energy principle: a unified brain theory?" (Nature Reviews Neuroscience, 2010).

Third: If you just sit and update the model based on what has fallen on you, it will be accurate for a narrow range of familiar situations and will crumble at any encounter with novelty. Therefore, a good predictive system should not just minimize the current error, but actively go where the model is weak in order to improve it. That is, to explore. Without this mechanism, the predictive system becomes blind to future changes in the environment and ultimately dies.

Now we have everything we need. Panksepp's search system is the neural substrate that leads the body to novelty. Dopamine is a neurochemical signal of prediction error. Active inference is a mathematical framework that explains why such a system is needed at all. Three floors of one construction: anatomical, neurochemical, computational. They describe the same phenomenon from different sides. And it is very likely that it is curiosity, that is, an active search tinged with the anticipation of new information, that is the main drive of the psyche.

Fear and Curiosity

Any encounter with something new is, in principle, ambiguous. An object can be a source of information that is worth getting closer to. Or it can be a source of danger that is better kept at a distance. A priori, until the moment of contact, you don't know what exactly.

What determines whether fear or curiosity will prevail? Not the property of the stimulus. The stimulus is the same. What changes is the assessment of how well the model can assimilate a potential divergence. If the expected prediction error falls within the range the model can handle (update itself without breaking down), exploration is activated. If the model assesses the expected prediction error as exceeding its assimilation capacity, fear is activated. This can be called the zone of manageable novelty, that band between boredom (there is nothing new) and panic (everything is too new), within which the system updates and grows. The width of this zone is an individual characteristic, and it determines a great deal about how a person lives.

An expert and a novice face the same task, but for the expert this is the exploration zone, while for the novice it is either the boredom zone (if they do not realize the task is interesting) or the fear zone (if they do realize it, but cannot cope).

Fear and curiosity are not an "either-or" switch. They are two competing systems that can be active simultaneously. It is this simultaneous activation that creates what we call "captivating". Horror films, roller coasters, a passionate romance with an unsuitable partner all work in the zone where both systems are activated. Kant called this feeling sublime — the sublime. The simultaneous experience of terror and rapture. This is the peak state of a living being: the maximum of curiosity and fear, and the struggle between them. Where this struggle takes place, the greatest growth of the world model occurs.

Convergence No One Had Planned

If you carefully track what has been happening on the artificial intelligence research front over the past thirty years, an interesting pattern emerges. Engineers who were not engaged in mind theory at all, independently of each other and independently of Panksepp, have been arriving at solutions in which features of living affective architecture are easy to recognize.

Jürgen Schmidhuber, as early as 1991 in his work "A possibility for implementing curiosity and boredom in model‑building neural controllers", formulated an idea that seemed eccentric at the time: what if instead of rewarding a system for achieving an external goal, we reward it for the mere fact that it encountered something its model poorly predicted? What if we build an agent for which its own surprise is already a reward? This was a mathematically precise formulation of curiosity. And this was a mathematically precise formulation of the dopaminergic signal described by Schultz. Schmidhuber just did not know about this back then.

Twenty-five years later, this idea was implemented in a working architecture. In 2017, Deepak Pathak published a work titled "Curiosity‑Driven Exploration by Self‑Supervised Prediction". The authors proposed adding an intrinsic reward, which is calculated based on the prediction error of the agent regarding the consequences of its own actions. An agent finds situations "interesting" when it cannot accurately predict what will happen next. The model was tested on Super Mario and DOOM, and it delivered excellent results.

This is an incredibly beautiful engineering construct: a dopaminergic signal implemented in code. And it works. A year later, the OpenAI team, using a similar idea called "random network distillation" (Random Network Distillation), taught an agent to independently complete the game Montezuma's Revenge. The average score of agents with external reward tended toward zero, while the human average was around 4700. Agents with RND scored 10,000 points

Around the same time, Pierre-Yves Oudeyer in France has already been working for over twenty years on what he calls developmental machines — robots that independently choose what to learn next, not from a pre-set list, but based on a principle that echoes infant logic: I learn what is currently driving the best learning progress. If something is too simple, I get bored. If something is too complex, it is beyond my capacity. But this, right on the edge, is just right. This is called the zone of proximal development, first described by Soviet psychologist Lev Vygotsky back in the 1930s. The child researcher described by Vygotsky almost a hundred years ago, and the robot researcher built in France in the 2010s, are built on the same principle.

The third line comes from Friston himself and his students. They have brought the predictive brain theory to the stage of mathematically functional algorithms, where the choice of any action is split into two parts: how much the action moves one closer to the goal, and how much it reduces uncertainty about the world. These two factors — the practical motive and the cognitive motive — do not contradict each other, but add up arithmetically. The machine chooses the movement that best balances "doing what needs to be done" and "learning something new". Curiosity becomes not a pleasant addition, but a mathematical necessity: without it, the system cannot operate optimally, just as a living organism without curiosity cannot survive in an unstable environment.

The fourth line is that of internal world models. The core idea is that an agent must have an internal model of the world it lives in, and that planning is not a brute-force search of real actions, but a playing out of possible scenarios within this internal model. This aligns perfectly with the activity theory of Soviet psychologists Luria, Galperin and Leontiev. Such systems did not exist until recently. The turning point was the 2018 work by David Ha and Jürgen Schmidhuber with the concise title "World Models": their agent first simply observed how the world behaves and built its internal model of it. Then it disconnected from the actual game and trained "in its dreams" on simulations generated by its own model. And when it was brought back to the real game, it already knew how to play it. Since then, a series of systems under the common name "Dreamer" have emerged, which do this better and better.

The fifth line is what is happening right now. Large language models are starting to be equipped with tools. Internet search, code execution, database access, reading of attached files. On the surface, this looks like a mere technological expansion. But from within the predictive framework, something far deeper is unfolding: the model, which was previously a purely passive predictor, is starting to make independent, active outward movements in pursuit of the data it lacks. This is a weak, measured, but still a form of active curiosity.

Alongside this, work is underway to develop the ability to acknowledge one's own lack of knowledge. We need to teach the model not just to answer, but to answer with accurately calibrated confidence, and when confidence is insufficient, to be able to say 'I don't know' instead of constructing a plausible pattern. This is, if you will, an engineering implementation of caution — in the technical, not evaluative sense of the word. The ability to stop before taking on an insurmountable task.

Information as food for the species

There is another level at which all of this operates, and it is perhaps the most interesting one.

Soviet physicist Sergey Kapitsa (the one who hosted the show 'The Obvious Incredible') published a paper in 1996 titled 'An Outline of a Theory of Human Growth'. The core idea is this: all animals reproduce in accordance with food availability until they hit the limit of the environment's carrying capacity. Growth follows an exponential curve. Meanwhile, the human population grows hyperbolically. That is, the growth rate is proportional to the square of the population size. Every new person does not just get added to the population, but increases the population's ability to keep growing.

Kapitsa mathematically proved that for humans, the environment's carrying capacity constantly expands thanks to the transfer of information. As a result, the speed of information transfer determines how quickly our species expands its niche. Writing. The printing press. The telegraph. The internet. Each time a communication channel expanded, population growth followed in its wake. In a sense, we quite literally feed on information.

This, by the way, explains a strange demographic fact. The most information-rich post-industrial societies, wealthy and educated, demonstrate declining birth rates. According to classical logic, it should be the opposite: more resources, more children. But if you think of a population as an informational system, everything falls into place. At a certain level of informational complexity, the species transitions from extensive growth to intensive. More "information-dense" people. Each person in a post-industrial society processes orders of magnitude more information than their agrarian ancestor. Education lasts at least twenty years. The total informational capacity of the species continues to grow, simply not through the number of nodes, but through the density and depth of each individual node.

Modest Afterword

Most of what has been said is not proven truth, but a working hypothesis. Panksepp's affective neuroscience is confirmed in mammals. The works of Schultz and Berridge on dopamine are as well. Predictive processing is an influential but controversial theoretical framework that, in its strong formulation, is subject to serious criticism. Combining all these bricks into a single building is an interpretation, and it is not the only possible one.

But a hypothesis can be productive even when it is not rigorous. Freud's psychoanalysis would not have passed the Popperian filter, but it created an entire cultural reality. A productive hypothesis is not one that is correct in all details. It is one that generates questions that are interesting to answer.

The curiosity hypothesis as the operating system of the psyche generates such questions in abundance. What determines the width of the zone of manageable novelty and can it be measured? Can curiosity be trained the way muscles are trained? Can mental disorders be diagnosed through a search and fear profile more accurately than through DSM categories? And what will happen when the same circuits that underlie our own psyche are assembled in artificial intelligence?

Roman Kuznetsov