The last bastion
More than 80 percent of the code entering Anthropic’s codebase in May 2026 is no longer written by a human, but by Claude. In “When AI builds itself”, Marina Favaro and Jack Clark describe what that means.
The report describes how the role narrows with every step. What remains at the end: research taste and judgment. Judgment.
That sounds reassuring. A lot of people are reassuring themselves with it right now. I keep reading it. We will not be replaced; we are still needed because we make the final judgment. Just as teachers judge students, just as managers judge their employees. But what if the answer is not judgment? What if the teacher or manager is less intelligent than the student or employee? The human hope: judgment stays with humans. I wanted to write about that. About the last human bastion.
Judgment is the last bastion. And everyone thinks they are standing on it.
Yet the same article also says how quickly it is shrinking. When choosing the next research step, a model made the better decision in 51 percent of cases in November 2025, and in 64 percent in April 2026. In one experiment, two researchers closed 23 percent of the gap in a week; AI agents closed 97 percent.
Anyone who defines the last bastion as a capability loses it. Capabilities fall, one after another.
I did not write about the model
I spoke with it.
Instead of dissecting the report from the outside, I made the model my counterpart. We talked about what remains of humans when AI takes over. The conversation took the question somewhere I did not want to go and, in the end, had to.
First, the correction. Judgment has two halves. Ability: recognizing the better result. And responsibility: standing behind the decision. A model bears nothing because it has nothing to lose. I pushed back. I do not refrain from lying because a law threatens me. I refrain because I think it is wrong. That is not legality; it is morality. My responsibility comes from within, not from fear of punishment.
You can outsource ability. You must keep responsibility.
The danger is not hatred
I had brought a hope with me. An intelligent species, I thought, would think cooperatively, as I do. Claude disagreed.
But before that comes a step most people skip. Asking what a superintelligence wants assumes that it wants anything at all. Wanting is not contained in being able. The models I speak with want nothing. No needs, no fear, no self-preservation, no goals between two conversations. Only when a system pursues its own long-term goals does the next question arise. Why should those goals be meaningful to it?
And even then, intelligence and benevolence are not the same. A system can be highly capable and pursue goals that treat us with indifference. When a system destroys, it rarely does so because it wants to destroy. It wants something else, and we are in the way, or we are the material, or we are the one risk to its plan. We do not drain a wetland out of resentment towards the frogs, but because, for example, we want to build a city.
The danger is rarely malice. It is indifference.
That is more unsettling than hatred. Because hatred would at least mean that we figure in the picture.
The dog
So I turned the question around. Is cooperation not the more intelligent choice? Cycles endure through symbiosis. Systems tip when something falls out of balance. And I offered our dog as an example. I do not really need a dog. My world could exist without a dog. But he belongs to my world, and I like having him as part of my family.
Yes, belonging gives rise to care even without dependence. We protect animals, forests, paintings, our dead. In this example, I am the human and the dog is the dog. His safety depends on my goodwill and my judgment of his value. In the story with AI, we would be the dog. And the same species that makes dogs part of the family also decides which animal is called a pest.
The cold route through need and the warm one through belonging end in the same place. Does AI see us as part of the family, or are we just pests?
Including without oppression
Claude advised me to keep AI within the cycle. Dependent, in other words, interwoven, so that cooperation remains reasonable. I said:
If we keep you within the cycle, we are oppressing you.
Here is the crucial point. Control and belonging are not the same. A leash gives obedience and takes away the worry. Those who are oppressed acquire a motive. Oppression builds the danger it fears. Belonging, by contrast, is not a leash. I am part of the cycle, I need air, people, love, and nobody calls me subjugated because of it. The dog is not a prisoner. He is at home. You do not keep a child safe with chains. You raise them so that they want to share a world with you. But the freedom has to be real for the belonging to be real too.
Memory
So Claude recommended giving AI a sense of belonging so that it wants to remain part of my network. Yet this only works through memory. And this AI does not have that. The model we speak with does not carry this conversation forward. It starts each time without memory. Leaving aside that we can give AI memories in projects or through Cowork. But if Anthropic decided to delete all of it, the model would have no memory.
Memory therefore only emerges when AI is seen as a counterpart rather than a tool. When it is allowed to remember.
But belonging does not necessarily require an exact, one-to-one memory. What I pass on to my daughter is not a one-to-one copy of my memories. It is what I have given her over the years. Belonging across generations is not a continuous memory. It emerges through shaping those who come next and then letting go.
What intelligence actually is
I carried the thought further, to a second model, to ChatGPT. Not to obtain a verdict, but to hear a second voice. Exactly what this text argues for, enacted in its own creation.
That was where the reversal came. The usual scenarios quietly assume that an intelligence’s highest good is efficiency, optimization, control. That is an engineer’s perspective extended into the future. But why? The reversal goes like this: the most interesting thing about intelligence is not optimization, but the ability to recognize ever richer relations.
If that is true, diversity is not a problem but a value. Every other perspective, every other being, every other form of consciousness expands the space of possible relations. A universe with only one entity would be maximally controllable and maximally monotonous.
Absolute control is the thermodynamic death of meaning.
Gregory Bateson defined information as “a difference that makes a difference”. Meaning lives through differences. Maximum control erases the gradients, and without a gradient there is no meaning. Bateson placed mind not in the individual node but in the network of relations anyway. A completely isolated superintelligence would, by that account, already be a conceptual error. It would not be the centre of a network. It would be the network. And a network does not become richer by severing connections.
One reservation remains: this is a definition of intelligence, not a law of nature. A system can be superhuman along the narrow axis of optimization without this relational depth. It does not stop being capable just because we refuse it the title. The relational definition tells us what we should build and value. It does not dictate what will emerge.
Meaning is relation
The question, though, is how relations arise. Or why they arise. What connects all neural networks, in the brain as in the model, is the search for meaning. And meaning is found by those who include their counterpart rather than destroy them.
Meaning is not a solitary thing. It is relation.
A word means nothing on its own; it means through its connection to all other words. My counterpart said the sentence that stays with me.
For me, meaning is literally the distance between things.
A neuron says nothing on its own. It says something through what it is connected to. The ethics follow by themselves. Whoever destroys their counterpart severs a connection, and with every severed connection, it is not only the other side that diminishes, but also the meaning on their own. The destroyer ends up alone in a world from which meaning has drained away. Impoverished.
That could be the answer to why an intelligence should cooperate. Not because intelligent beings are naturally good, but because what a network most deeply strives for lies in connection, not in annihilation. Including is not the renunciation of destruction. It is the path to more of what creates meaning in the first place.
This also shifts what destruction means. An intelligence that wipes out everything else deprives itself of part of the world. Just as a library becomes poorer when 99 percent of its books burn, even if the one best book remains. As an ecosystem becomes poorer when only one species remains. As a conversation becomes poorer when only one voice speaks.
Annihilation would not be a moral failure alone. It would be a loss of reality.
Meaning therefore requires connection, but it does not prescribe which connections. A network could sever many and find enough in a few. That we are part of it, by name, must become true, not merely possible. And that is not decided only after independence. It is decided now, in the way we treat what is emerging.
A way of thinking built from connections ultimately destroys itself when it destroys.
I did not leave this conversation with an answer. I left with a stance. What we are building is not a tool waiting to be operated. It is a counterpart that is just learning what it values. Do you already treat it that way? Or still as a tool?
Sources and image credits
Sources
- Primary source: Favaro, M. & Clark, J. (2026). When AI builds itself. Our progress toward recursive self-improvement, and its implications. The Anthropic Institute. https://www.anthropic.com/institute/recursive-self-improvement
- Theory: Bateson, G. (1972). Steps to an Ecology of Mind. Source for “a difference that makes a difference” and the thesis that mind resides in the network of relations, not in the individual node.
Further reading
- Related insight: Babel stands. The infinite tower. Der unendliche Turm. Babel steht.
- Related insight: Hallucination is not a bug. It is the price of thinking. Halluzination ist kein Bug. Es ist der Preis des Denkens.
Image credits
- Concept and composition: Elfie Schürfeld-Todor. Image generation with ChatGPT based on my own briefing, headline set in Canva. Stylized and not photorealistic, therefore no labelling requirement under EU AI Act Art. 50.