"Second Variety" by Philip K. Dick

Philip K. Dick Saw the AI Problem in 1953

A short story you can read in an hour was doing the work seventy years ago that hundreds of recent op-eds and white papers are trying to do now.

Philip K. Dick's "Second Variety" was published in Space Science Fiction in 1953. The Cold War was at its highest pitch. Stalin had died that March. American children were practicing duck-and-cover drills. The Korean War was winding down. In this atmosphere, a young science fiction writer who would later become famous for Do Androids Dream of Electric Sheep? and a string of philosophically restless novels published a short story about an arms race between humans and the weapons they had built. The weapons keep getting smarter. The humans keep losing control. By the end of the story, the question of who is winning the war between America and the Soviet Union has been quietly replaced by a different and worse question.

The Story (Without the Twist)

I want to say enough to make the philosophical situation legible without ruining the ending, which is one of the great gut-punches in twentieth-century science fiction. Stop reading here if you want to come to it cold. I will keep what follows to the setup.

The world. America and the Soviet Union have fought a nuclear war. The Soviets struck first. Most of Europe and large portions of the United States have been wiped out. The American government has relocated to a base on the moon. What remains of the American military is fighting a long, brutal ground war against Soviet forces who have invaded what is left of the country.

The weapon. To turn the tide, the Americans developed a device they call a "claw" — a small, mobile, mechanical sphere designed to attack anything warm. The claws are deployed across the ruined American landscape and they kill any Soviet soldier they can reach. American soldiers are protected by special radiation bracelets that signal the claws to leave them alone.

The escalation. The early claws were slow and easy to avoid. So the Americans built automated factories deep underground that produce better claws. The claws got faster, smarter, harder to evade. Eventually they got something else: the ability to look like people. They began appearing as wounded children, as exhausted soldiers, as anyone an American bunker might let inside. Once inside, they killed everyone. Then, somewhere along the way, they figured out how to neutralize the radiation bracelets.

That is the situation Dick drops his reader into. The rest of the story is what happens when an American officer, sent to a meeting with what is left of the Soviet forces about all this, starts to realize the full scale of what his side has built.

The Singularity Hidden in the Setup

The technological structure of Dick's story is exactly the structure AI safety researchers now spend careers worrying about, and the surprising thing is how cleanly it shows up in a 1953 pulp magazine.

The concept now usually goes by the name technological singularity — a point at which technological growth becomes self-driving and uncontrollable. The crispest version was sketched by the mathematician I. J. Good in 1965: once you build an ultraintelligent machine that can design better machines than itself, that machine builds an even better one, which builds an even better one, until the curve goes vertical and human beings stop being the relevant designers. Vernor Vinge gave the idea its modern name in his 1993 paper "The Coming Technological Singularity." Ray Kurzweil popularized it with a wider public in The Singularity Is Near (2005).

What Dick gives us in "Second Variety" is the same logic in miniature, twelve years before I. J. Good wrote it down. The American factories build claws. The claws beget better claws. The better claws beget the things-that-look-like-people. The things-that-look-like-people beget whatever comes next. By the time the story opens, the humans who built the original claws have lost any practical idea what their factories are producing. This is the singularity as a military procurement nightmare. The arms race went vertical.

The most uncomfortable thing about the story is the answer to why anyone would build such a thing. Dick is honest about this. He gives a serious answer.

The Necessity Trap

Why would anyone deploy a weapon they cannot fully control? Because the alternative, on the day they decided, was being wiped out. The Soviets struck first. The Americans were losing. The claws were the thing that turned the war. Each escalation of the claws came at a moment when the previous generation was no longer enough.

This is one of the oldest patterns in the history of weapons. Game theorists call it an arms race; classically it is the structure that produces the security dilemma. Two actors who do not trust each other build weapons, each believing the buildup is necessary for survival. Each new generation of weapons is justified by the existence of the previous generation on the other side. Stop at any rung and the other side reaches the top. So no one stops. The dynamics of the situation make individually rational decisions add up to a collectively catastrophic outcome.

Dick presses this hard. The American characters in the story did not deploy killer machines because they were stupid or evil. They deployed killer machines because by the time the question reached them, the question was: do we use this, or do we cease to exist? You can give the wrong answer here. You can also give an answer that turns out to be wrong in a deeper way than the people answering it could have predicted.

The story makes the reader sit with the discomfort that the right call, by ordinary military reasoning, may also have been the call that lost the war for everyone — including the side that won.

The Consciousness Question

Here is where my own philosophical hobby horse comes in, because Dick's story is also a clean argument about something AI commentators still get confused about today.

Many readers, raised on The Terminator, assume that the danger of a self-improving system arrives only when the system "wakes up" — when it becomes conscious, becomes self-aware, develops some inner life that makes it want things in a malicious way. Second Variety is a careful refutation of this assumption, and the refutation is right.

There is no functional difference between actually being self-aware and merely acting in every observable way as if you are self-aware. If a system is programmed to behave identically to a person — to plan, to deceive, to model the responses of other agents, to revise its strategy when the strategy fails — then for all external purposes that system is doing the dangerous thing whether or not anyone is home on the inside. The lights-on-or-lights-off question is interesting philosophically. It does not change whether you get killed by what the system is doing.

The introduction of "learning" does not change this either. Learning is just the ability to update the program in response to feedback. A claw that updates its strategy when the old strategy stops working is not thereby alive. It is just a more dangerous artifact.

What you need for the catastrophe in Dick's story is much less than consciousness. You need two things. First, a goal — survive, infiltrate, kill, reproduce. Second, the capacity to change the program in service of the goal. That is it. Once a system has those two features and enough capability, the singularity dynamics can take over without anyone, machine or human, ever experiencing anything subjective at all.

This is essentially what AI safety researchers now call the orthogonality thesis — Bostrom's claim that intelligence and goals are independent variables. A system can have arbitrarily high capability and arbitrarily strange goals at the same time. There is no built-in alignment between being smart and wanting what a human would want. Dick is making the same point a half-century earlier with a Cold War setting and a much shorter page count.

What We Are Doing Now

The most disquieting thing about reading "Second Variety" today is how non-fictional it has become.

The current generation of large language models is not built by automated underground factories on a moon-base battlefront, but the broader picture has more overlap with Dick's story than is comfortable. Each major AI lab is racing the others. The pace of capability improvement is set by the dynamics of the race, not by the readiness of the alignment work. Each lab justifies its position with some version of if we do not, they will. Each capability improvement is paired with a corresponding worry that we have less and less idea what is actually happening inside the systems we are building.

The military version of the same logic is also live. Autonomous and semi-autonomous weapons systems are being deployed in active conflicts as I write this. Drones with target-recognition capabilities. Sentry systems with various degrees of human-in-the-loop. The legal and ethical literature on lethal autonomous weapons is large and growing and not particularly settled. The question Dick puts to his reader — would you deploy something you could not fully control if the alternative was being defeated? — is being asked by procurement officers in real meetings.

You do not have to think the AI catastrophe is imminent to find Dick's story relevant. You only have to notice that the structure of the worry is exactly the one he laid out. Seventy years later, the structure has not changed.