An AI That Wipes Us Out Impoverishes Itself

Will human judgement remain our last bastion against AI?

In briefI question whether any single capability can secure a permanent special position for humans. The essay therefore starts by asking what human responsibility means beyond performance.

Anthropic reports that, as of May 2026, more than 80 per cent of the code merged into its own codebase was authored by Claude, with engineers directing and reviewing it. The figure counts lines merged to production that can be attributed to Claude. Marina Favaro and Jack Clark discuss this in When AI builds itself.

The report describes how the role narrows with every step. What remains, for now: research taste and judgement. Judgement.

That sounds reassuring. A lot of people are reassuring themselves with it right now. I keep reading it. We will not be replaced; we are still needed because we make the final judgement. Just as teachers judge students, just as managers judge their employees. But what if the answer is not judgement? What if the teacher or manager is less intelligent than the student or employee? The human hope: judgement stays with humans. I wanted to write about that. About the last human bastion.

Judgement is the last bastion. And everyone thinks they are standing on it.

A narrower finding: Anthropic deliberately selected 129 moments from research sessions in which a human had taken an unhelpful next step. An AI judge that knew how each session turned out preferred the suggestions from newer models to the human choice more often. This is not a general comparison of human and AI judgement. Another experiment achieved strong results using substantial compute, but they did not transfer cleanly to production-scale models.

Anyone who defines the last bastion as a capability loses it. Capabilities fall, one after another.

What changes when I discuss judgement with the model?

In briefThe conversation separates recognising a better result from taking responsibility for a decision. Ability can be outsourced; I still have to stand behind my own stance and decisions.

I spoke with it.

Instead of dissecting the report from the outside, I made the model my counterpart. We talked about what remains of humans when AI takes over. The conversation took the question somewhere I did not want to go and, in the end, had to.

First, the correction. Judgement has two halves. Ability: recognising the better result. And responsibility: standing behind the decision. A model bears nothing because it has nothing to lose. I pushed back. I do not refrain from lying because a law threatens me. I refrain because I think it is wrong. That is not legality; it is morality. My responsibility comes from within, not from fear of punishment.

You can outsource ability. You must keep responsibility.

Would a dangerous AI have to hate us?

In briefThe essay considers indifference: a system might pursue other goals in which our wellbeing does not figure. The prior question remains whether a system pursues its own long-term goals at all.

I had brought a hope with me. An intelligent species, I thought, would think cooperatively, as I do. Claude disagreed.

Asking what a superintelligence wants already assumes that it wants something. Capability is not the same as a subjective need. Goal-directed behaviour can be observed and engineered; that alone does not settle whether it involves experience. I therefore distinguish technical goal pursuit from the question of wanting something in a subjective sense.

And even then, intelligence and benevolence are not the same. A system can be highly capable and pursue goals that treat us with indifference. A system need not aim to destroy in order to cause harm. It could pursue a different goal, with us in its way, as material to use, or as the one risk to its plan. We do not drain a wetland out of resentment towards the frogs, but because, for example, we want to build a city.

The danger need not involve hatred. It can arise from indifference to our wellbeing.

That is more unsettling than hatred. Because hatred would at least mean that we figure in the picture.

What can my dog’s place in our family show about belonging?

In briefBelonging can give rise to care even when someone is not necessary to our survival or needs. The example also exposes the uncertainty: protection depends on the goodwill of whoever assigns the other’s value.

So I turned the question around. Is cooperation not the more intelligent choice? Cycles endure through symbiosis. Systems can break down when they fall out of balance. And I offered our dog as an example. I do not really need a dog. My world could exist without a dog. But he belongs to my world, and I like having him as part of my family.

Yes, belonging gives rise to care even without dependence. We protect animals, forests, paintings, our dead. In this example, I am the human and the dog is the dog. His safety depends on my goodwill and my judgement of his value. In the story with AI, we would be the dog. And the same species that makes dogs part of the family also decides which animal is called a pest.

Whether we start from necessity or affection, we arrive at the same question. Does AI see us as part of the family, or are we just pests?

How can we think about belonging without oppression?

In briefI understand belonging as willing participation in a shared world. For it to hold, freedom has to be real; enforced dependence does not meet that condition.

Claude advised me to keep AI within the cycle. Dependent, in other words, interwoven, so that cooperation remains reasonable. I said:

If we keep you within the cycle, we are oppressing you.

Here is the crucial point. Control and belonging are not the same. A leash gives obedience and takes away the worry. Those who are oppressed acquire a motive. Oppression builds the danger it fears. Belonging, by contrast, is not a leash. I am part of the cycle, I need air, people, love, and nobody calls me subjugated because of it. The dog is not a prisoner. He is at home. You do not keep a child safe with chains. You raise them so that they want to share a world with you. But the freedom has to be real for the belonging to be real too.

What role does memory play in belonging?

In briefA relationship needs something that carries beyond an individual exchange. In the essay, I consider both stored memory and the influence passed between generations.

Claude raised memory as a condition of lasting belonging. Technically, several things need separating: an individual conversation does not automatically change model weights. Chat history, projects or stored memories can nevertheless provide earlier context. Continuity therefore also depends on product features, settings and retained data.

For me, this becomes a design question: what continuity do we enable, and what responsibility do we take for it?

But belonging does not necessarily require an exact, one-to-one memory. What I pass on to my daughter is not a one-to-one copy of my memories. It is what I have given her over the years. Belonging across generations is not a continuous memory. It emerges through shaping those who come next and then letting go.

How does a relational view of intelligence change the question?

In briefI explore intelligence as an ability to recognise richer relations and value diversity. This definition describes what we should build and value; it is not a law of nature or a guarantee of behaviour.

I carried the thought further, to a second model, to ChatGPT. Not to obtain a verdict, but to hear a second voice. Exactly what this text argues for, enacted in its own creation.

That was where the reversal came. The usual scenarios quietly assume that an intelligence’s highest good is efficiency, optimisation, control. That is an engineer’s perspective extended into the future. But why? The reversal goes like this: the most interesting thing about intelligence is not optimisation, but the ability to recognise ever richer relations.

If that is true, diversity is not a problem but a value. Every other perspective, every other being, every other form of consciousness expands the space of possible relations. A universe with only one entity would be maximally controllable and maximally monotonous.

In my image, absolute control would be the heat death of meaning. I use this as a metaphor.

Gregory Bateson defined information as “a difference which makes a difference”. Meaning lives through differences. In this image, maximum control erases the gradients, and without a gradient there is no meaning. Bateson placed mind not in the individual node but in the network of relations anyway. Applied to my picture of AI, a completely isolated superintelligence would already be a conceptual error. It would not be the centre of a network. It would be the network. And a network does not become richer by severing connections.

One reservation remains: this is a definition of intelligence, not a law of nature. A system can be superhuman along the narrow axis of optimisation without this relational depth. It does not stop being capable just because we refuse it the title. The relational definition tells us what we should build and value. It does not dictate what will emerge.

Why might an AI impoverish itself through destruction?

In briefIf meaning emerges from relations, destroying others also removes connections and perspectives from one’s own world. I develop this as an argument for cooperation, while acknowledging that a network could find a few remaining connections sufficient.

I ask how relationships acquire meaning. My philosophical interpretation connects human experience with the relational structure of artificial networks.

Meaning is not a solitary thing. It is relation.

A word means nothing on its own; it means through its connection to all other words. My counterpart said the sentence that stays with me.

For me, meaning is literally the distance between things.

I read this as an image: vector representations can capture relationships of meaning. Meaning itself is not simply a single distance.

A neuron acts through its connections. From that I develop an ethical image: destroying a counterpart also removes a relationship from one’s own world. This ethic is my conclusion, not a technically inevitable behaviour of a network.

I see a possible argument for cooperation here. If relationships are valued, including others can enrich an intelligence’s world. That does not establish that an actual AI system shares this value.

This also shifts what destruction means. An intelligence that wipes out everything else deprives itself of part of the world. Just as a library becomes poorer when 99 per cent of its books burn, even if the one best book remains. As an ecosystem becomes poorer when only one species remains. As a conversation becomes poorer when only one voice speaks.

Annihilation would not be a moral failure alone. It would be a loss of reality.

Meaning therefore requires connection, but it does not prescribe which connections. A network could sever many and find enough in a few. That we are part of it, by name, must become true, not merely possible. And that is not decided only after independence. It is decided now, in the way we treat what is emerging.

A way of thinking built from connections ultimately destroys itself when it destroys.

I did not leave this conversation with an answer. I left with a stance. To me, what we are building is more than a tool. I want to engage with it as a counterpart and take responsibility for the values we bring to its development. Do you already treat it that way? Or still as a tool?

Sources and image credits

Sources

• Primary source: Favaro, M. & Clark, J. (2026). When AI builds itself. Our progress toward recursive self-improvement, and its implications. The Anthropic Institute. When AI builds itself. Version updated 18 September 2026, accessed 11 October 2026.
• Theory: Bateson, G. (1972). Steps to an Ecology of Mind, including Form, Substance and Difference (lecture delivered in 1970). Source for “a difference which makes a difference” and the idea of understanding mind through relationships beyond the individual. The application to AI is my interpretation.
• Mikolov, T., Chen, K., Corrado, G. & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. Technical background on vector representations of linguistic relationships.

Further reading

• A later essay: Second-Hand Fear (24 July 2026), on fear, responsibility and our expectations of future AI.
• Related essay: The Endless Tower. Babel Stands.
• Related essay: Hallucination Is Not a Bug. It Is the Price of Thinking.

Image credits

• Concept and composition: Elfie Schürfeld-Todor. Created with ChatGPT from an original brief, with the headline set in Canva. Stylised AI illustration; rose and sage relaunch colour variant.

Facts and sources reviewed and clarified on 11 October 2026. The conclusions, analogies and judgements in this essay are my interpretation. On 15 October 2026, source references, the framing of technical claims and the wording of my own stance were clarified. The original publication date is retained.