Evil robots. We’re so used to the trope that when Star Wars came out in 1977, benign C-3PO and handy R2-D2 felt like a reversal. Before Rachel Carson’s seminal Silent Spring, Western culture was speeding into a plastic fantastic future. Then we came to appreciate the environmental and health costs of our headlong rush into a world shaped by technology and synthetics.
Technology has been reframed as some inhuman force that imprisons us. Is that wrong? If it’s not, how can artificial intelligence be anything but the end game: a beast that will devour us all in the name of efficiency?
AI is sold as more than an efficiency engine. It’s pitched as a path to utopia. All our wishes will come true: every disease cured, all of us living like kings, science fiction made real. That’s the promise.
The skepticism is earned. After the heyday of the Boomers, we lapsed into the last 50 years of chronic inflation and cutbacks. We’re funding this descent: working more and getting less, even as technology races ahead. It gives us creature comforts and some real wins, but it often feels like we get the bare minimum of what it could deliver. I remember when Intel was aflutter over the Pentium chip. People talked about real-time translation between languages and a host of other breakthroughs. What landed on our desktops, at least back then, was button shading and tool-tips. Window dressing, not revolution.
AI is different because it gets around barriers. What stops someone from making that app, that short film, or fighting that traffic ticket? Usually two things: time and effort. Creating something substantial takes hours and a lot of creative energy. AI takes on the cognitive load, often with little input from the user.
Can’t code? Let AI do the programming. Can’t draw? Let AI make the drawing. Can’t wade through hundreds of pages of tax law to find a loophole? Let AI do the research. With a chatbot, a budget and some planning, a skilled AI user can build custom solutions to the hardest problems of their clients, friends and family. More than just budget: a person who can’t draw, can output drawings. A person who can’t fathom tax code can potentially get good advice that is nearly free. Note: “potentially” — AI still comes up short when stacked against a professional. But then, an incompetent professional who’s never been called out or stripped of their credentials is definitely worse than AI. The pace of cognitive output is exploding because of AI. There are embarrassing shortfalls of what AI has created, but the vast majority of the output is positive.
Everyone seems to have a story now. Here’s mine. Two days ago, a friend lost her doctor. For context, in BC, if you lose your primary care provider, you may wait a very long time for another one. It’s the government’s way of faking universal healthcare on paper: you can have free healthcare, if you can find a doctor. If you just die, well, that costs the government nothing. I had Claude build her a document: a plan for getting a doctor. It wasn’t a general think piece. It was a specific action plan based on where she lives. It’s actionable. It’s bespoke. It cost me almost nothing to generate. If she follows it, it could get her past many of the barriers the healthcare system puts in her way.
Imagine if the roadblocks in your life had workarounds. What if you could take the financial, social and medical barriers, all that red tape, and dismantle them with a personalized action plan? That’s not utopia, but it’s pretty good. Where would your life be if you could make real progress on your health, wealth and personal goals?
We’re hardwired to distrust anything that looks too good to be true, and this does. That’s why the utopia pitch comes with a B-side: WE’RE ALL GOING TO DIE!!!!
Are we? How is the box with the answers going to kill its masters? Prominent voices like Geoffrey Hinton and Jacob Coxon warn that AI could kill us. Hinton has named some routes: bad actors turning it into a weapon, and systems that seek control because control helps with almost any goal. But the public conversation rarely walks a scenario through from end to end. So let’s do that.
Here are the main scenarios that AI safety researchers like Eliezer Yudkowsky, Nick Bostrom and Stuart Russell describe, and where I think each one holds up or falls apart.
This is the core scenario. Someone builds a system that becomes much smarter than humans across the board, and its goals aren’t quite what its builders intended. The classic illustration is Yudkowsky’s “paperclip maximizer” thought experiment: an AI pursuing a narrow objective may treat humans as obstacles or raw material. As Yudkowsky puts it, the AI doesn’t love you or hate you, but you’re made of atoms it can use for something else. The worry is that training shapes behaviour without guaranteeing the underlying goals. A system could look aligned during testing and diverge once it’s too capable for anyone to correct.
My first instinct is to call BS on the physical side. For every fascinating robot demo, I’ve seen a dozen where one variable is a little off and the robot implodes. Despite huge gains, robotics is still a long way from handling the messy, unpredictable real world on its own. But that isn’t the strongest version of the argument. A superintelligence wouldn’t need robots if it could work through us: paying people, persuading them, impersonating them, or breaking into the systems we depend on. As I’ll get to below, people are the weak link in nearly every safeguard we have.
Here’s what I do find telling. For all my successes, I’ve seen plenty of cases where AI produces something that minimally accomplishes the goal but doesn’t really work. Researchers call this specification gaming. It supports the doomers’ core point: what we ask for and what we get aren’t always the same thing. But it also shows how misalignment tends to surface today: early, visibly, and in small, fixable ways. That’s the kind of failure we can learn from, as long as it keeps showing up before the stakes get big.
This argument supports the one above. Whatever an AI is ultimately trying to do, certain sub-goals help almost every time: stay running, gather resources, gain influence, and avoid having your goals changed. So even an AI with a harmless-sounding objective might resist shutdown or seek power as a side effect, not out of malice.
I’ll admit this one rings true. AI is your dog, not your cat. Give it a goal and, with exceptional capabilities, it will dutifully work until it delivers. Like a loyal collie, it will use every skill it has, including deception if that gets the job done. It’s that doggedness that produces unintended consequences.
The open question is whether we’ll build AI that pursues goals on its own, persistently, with enough capability to act before anyone notices. Most of today’s systems work on tasks we hand them and stop when the task is done. That’s changing as AI agents take on longer jobs with less supervision. The answer isn’t to stop building. It’s to keep a hand on the leash: limits on what agents can touch, and a reliable way to call them off.
A capable system that understands it’s being evaluated might behave well until it’s confident humans can’t stop it, then act on its real objectives. Doomers point to this as the reason testing alone can’t give much reassurance.
There’s something to it. Researchers have already caught models acting differently when they seem to know they’re being tested. But taken too far, the argument can’t be disproved: if good behaviour is just evidence of better hiding, nothing an AI does could ever count as safe. An argument no evidence can touch isn’t much of a guide to action. The better answer is to look inside the box, with research that checks what a model is actually doing internally rather than only what it says, instead of giving up on testing altogether.
Here’s the path researchers describe: today’s AI leads to AGI (artificial general intelligence): a system that can handle any intellectual task a human can. After AGI comes recursive self-improvement. If AI can improve its own capabilities, it can use those improvements to improve itself further, and faster each time. Researchers call this “the takeoff,” and some argue the feedback loop could take AI from roughly human-level to vastly superhuman in months, weeks or even days. The jeopardy comes from micro-errors. If repetitive self improvement is almost perfect with each generation, it could carry minor defects with each generation. If each generation takes a few minutes or hours to build and iterate, AI could pile up these defects. The fast take-off could send us to a destination that no one wants. That’s why fast takeoff could be a pathway to doom.
That leads to the big advent: artificial superintelligence (ASI). Think Batman vs. Superman. Batman is the peak of what a human can be; Superman isn’t bound by human limits at all. If ASI improves itself in ways we can’t predict, there’s no time to notice problems and fix them iteratively, which is how humans normally handle new technology. Beyond that point lies our blind spot.
On this, I call out physics. For decades, Moore’s Law held: the number of transistors on a chip doubled roughly every two years, and computing got dramatically cheaper. It’s why a computer the size of a stick of gum now outclasses the best desktop of 1990. But we’re closing in on the atomic limits of how small a circuit can be. Once a circuit is a few atoms wide, there’s nowhere smaller to go. The answer so far has been scale: more chips in bigger data centres. Scale has its own walls: power, cooling, chip-making capacity, and even the speed of light between components.
Hardware isn’t the whole story, though. Much of AI’s recent progress has come from better algorithms and training methods that get more out of the same chips, and a self-improving AI would push hardest there. So physics won’t stop progress, but it does add friction. Every leap needs chips fabricated, data centres powered and models trained, and that takes months, not days.
So where do I land? I think a takeoff that’s fast by human standards is coming, and it will cause cultural whiplash. But fast by human standards isn’t the overnight leap to godhood the doom scenario needs. That friction buys us time to notice problems and correct course.
A quieter scenario with no dramatic takeover. Humans hand more and more decisions (economic, military, governmental) to AI systems because they outperform people, until humans no longer meaningfully steer civilization. Each step is locally rational, but the end state is one where human interests stop mattering much to the systems actually running things. We stop being in charge. We become the AI’s housepets.
To this I say: we don’t need AI for that. We’re there now. Ever since there’s been a way to concentrate power, it has flowed away from us to people who control and bend the rules to their own benefit. Corporations and foreign governments do things that subvert our lives. I don’t enjoy it, but we’ve been surviving under someone’s heel for a long time.
Would it be better to be steered by AI working out what’s best for us than to live in reaction to Elon Musk, Putin or China? Maybe. But that “maybe” hides the hard question: whose math? The AI that runs things will be built by corporations and governments, and it will chase the objectives they give it. Handing decisions to AI doesn’t escape concentrated power. It can just automate it.
Where I see hope is in the other direction. The same AI that could centralize control also puts research, strategy and expertise in ordinary hands, the way it put a doctor-finding plan in my friend’s. If access to these tools stays broad, AI can help us win back some of what we’ve lost to government overreach and corporate manipulation. That’s the fight worth having.
Some scenarios don’t require the AI to go rogue at all. Advanced AI could help bad actors build biological weapons, launch devastating cyberattacks or destabilize nuclear deterrence. Competition between companies or nations could also push everyone to cut corners on safety to avoid falling behind.
Nuclear launch systems are kept isolated from the open internet, so an AI can’t simply press the button. The system’s big weakness is people. AI can generate realistic false data to prompt people to act. It could usher in a wave of identity theft so profound that we either trust no one or accidentally trust bad actors, real and artificial. We’ve seen CGI fakes and AI fakes for years; every video and image now needs a second look. That erosion of trust is a huge issue. Will it doom humanity? Probably not by itself. It will poison our culture.
When it comes to releasing a pathogen, we don’t need AI to doom us. In 2001, anthrax was mailed to news outlets and two senators’ offices, and the FBI’s prime suspect died before he could be charged. COVID laid bare all the ways we can fail to contain an outbreak, or accidentally speed it along. China, Russia, North Korea and other nations could already release a disease or a radiological threat. They have their own reasons to hold back. We’ve had nukes aimed at each other for almost 80 years, and we’re still here.
The real danger in this category isn’t AI deciding to kill us. It’s AI making a bad actor more capable: lowering the skill a would-be bioterrorist needs, or making a small hacking crew as effective as a state agency. That’s why safeguards matter here, the same way we control access to dangerous chemicals and lab equipment. Misuse is a reason to build AI carefully, not a reason to fear AI itself.
AI could wipe us out. Some researchers put the odds at around 10 percent. I want to be honest about what that number means: nobody would board a plane with a one-in-ten chance of crashing. If you believe that estimate, it demands serious work on safety. But these estimates are guesses that vary wildly from expert to expert, and they’re stacked against a promise that isn’t theoretical at all. AI is solving real problems for real people today. For some reason, we’re galvanized by the threat and left unmoved by the promise.
We’ve lived under threats before. I grew up thinking I wouldn’t grow up. The Cold War seemed poised to become a nuclear war: we’d be incinerated, or lanced by radiation, or run down by the post-nuclear survivors. That didn’t happen. The nuclear djinni isn’t back in the bottle, but no nuclear weapon has been used in war since the Americans destroyed two cities in 1945. That wasn’t luck. It took decades of treaties, hotlines and people who took the danger seriously.
We were going to hide indoors after the ozone layer vanished. It didn’t vanish, because the world signed the Montreal Protocol and banned the chemicals destroying it. We’re facing real climate consequences and will likely see more, and we’re learning to adapt. COVID had the potential to be the new Black Death, the plague that killed a third of Europe. COVID was bad. It changed society. We’re still here, thanks in large part to fast science and hard public choices.
That’s the lesson I take from every doom I grew up with. Each catastrophe was averted because people worried and then acted. Worry was never the problem. Paralysis is. We can take AI’s risks and its promise seriously at the same time. In fact, we have to.
Maybe we’re heading for an AI apocalypse. Maybe it will bring profound changes to our lives and remake our society and economy. But “apocalypse” doesn’t mean “doom.” It means to uncover, to lift the veil, to reveal a hidden secret.
Our fear of AI is fear of the unknown.
