This story originally appeared on Mother Jones and is part of the Climate Desk collaboration.
We’ve been hearing a great deal recently about the specter of AI-driven human extinction. Maybe our misplaced trust in AI intelligence will set off World War III. Or AI agents will unleash widespread cyberattacks and take over the internet. Given too much autonomy, the machines will run amok. They may seek to deceive their creators—wait, that’s already happened. Suppose they conclude, like Agent Smith in The Matrix, that “human beings … are a plague, and we are the cure.” Or perhaps the chatbots help bad humans unleash a literal plague—a biological weapon.
That last one isn’t merely theoretical. A “misuse” report Anthropic made public earlier this month detailed five cases in which foreign scientists tried to evade Claude’s guardrails on “gain of function” inquiries. That’s a euphemism for when scientists tinker with a pathogen’s genes to enhance it in some way, making it more infectious, deadlier, or better able to spread among mammals, for example.
Anthropic denied those users the answers they sought and cut off their accounts, even while noting that their motives may not have been sinister. Such information also can be used to help scientist develop pandemic countermeasures like vaccines and drugs. And there’s the rub. Like so many technologies, AI included, genetic engineering can be used for both good and evil. Hence another euphemism: “dual use.”
I studied to be a molecular biologist, and have been covering the gain of function debate since long before AI was on everyone’s radar. In my experience, researchers who work on high-risk pathogens tend to default to the view that it’s better to publish than not, and that we’re better off doing the work to understand which genetic traits make organisms a threat to humankind—never mind the potentially catastrophic consequences of an accidental release, or a deliberate one.
Is humanity really better off doing the work? I have my doubts. And I have more of them after reading the latest book from Annie Jacobsen, whom I last interviewed about her best-selling book Nuclear War: A Scenario. Her new book, Biological War: A Scenario, came out in late July and follows the same template. And because I know you were about to ask, yes, I’m sure she is great fun at cocktail parties.
One thing about Jacobsen, who has decades of high-level national security reporting under her belt, is her unrivaled access to current and former military and government officials and scientists—people who spend their days thinking about threats to humanity and gaming out how the government might respond if such-and-such horrific thing were to happen.
Beyond the notion of human extinction, perhaps the most terrifying aspect of both of her books relates to the limitations in our government’s preparation and response. America’s general plan for nuclear war was, and no doubt remains, complete madness. As for biological war, spoiler alert, it turns out we really can’t defend against it. (You’re welcome.) I’ve edited our conversation for length and clarity.
MICHAEL MECHANIC: So, Annie, now that you’ve done both the nuclear scenario and the biological scenario, which is more likely to keep you up at night?
ANNIE JACOBSEN: The Defense Department document that I reprinted in the book, on the weapons of mass destruction threat spectrum, lists nuclear weapons at the far end of the curve because they are the most destructive. But I was really surprised to see how biological weapons are a close second. And I believe the reason they’re the greatest overall threat very much ties into the discussion the entire world is having today, which is that the threshold for entry is extremely low, and therefore it is more likely to happen.
To have a nuclear weapon, you have to have weapons-grade fissile material, and then you have to think about containing those weapons within the arsenal. You have to have an ICBM, a B-2 bomber, a nuclear-armed submarine.
Yes. A terrorist with a dirty bomb, that’s just a completely different threat. The component that makes biological war so incredibly dangerous and the greatest overall threat is that there are nine nuclear-armed states tasked with this containment component, but then you look at the rest of the world and who has access to [potential pandemic agents] …
If you were a bioweapons designer, what would you look for in a pathogen?
My mind does not think that way because I am a mother and I’m a member of society. But If I were a pandemic planner, what would scare me the most is the concept of an airborne agent and the public’s naivete to that very concept. I would think about how important it is to educate the public about the reality of what is an extraordinary threat and what isn’t.
Your scenario involves the accidental release of a weaponized pathogen from a high security lab in Russia. Based on the experts you spoke with, is an accidental release more or less likely than someone deploying a bioweapon?
Accidental release is positioned as a much greater threat for a couple of reasons. One, I learned there are very serious counterproliferation efforts at the CIA and other intelligence agencies to disrupt bioweapons programs. Whether you want to sleep better at night knowing the CIA is out there countering these things is up to you, but there are examples of biological weapons programs in the hands of terrorists being shut down.
What I came to understand is that the only people crazy enough to deploy or release a biological weapon into the community, knowing full well that the blowback would be extraordinary and would create a pandemic around the world, would be someone who had a death wish. Many terrorists fit that that category, so that is deeply unsettling. But that’s a smaller group.
And then you think about all the people—scientists, nation states, academic institutions—that are manipulating pathogens for greater virulence, for greater transmissibility in these BSL-3 and BSL-4 labs [Biosafety Level 4 is the highest security designation]. I mean, there are 3,600 of those labs around the world. There are not 3,600 terrorist organizations that I am aware of. So the chance for an accident is statistically far greater. That’s why I begin my scenario with a lab leak, and also because it’s happened before.
One can imagine a researcher sticking themself with a syringe at a BSL-4 lab and freaking out, humans being humans, and kind of going into denial and heading home.
One hundred percent—and as has happened. Then you have other examples, like [Nikolai] Ustinov, whom I wrote about in a little history lesson in the book, sticking himself with a deadly Marburg virus, from which there is no cure, and doing the good Soviet scientist thing, which is tell on himself, and then be left in a isolation ward behind sort of submarine-lock type doors to die a horrific death by hemorrhagic fever. Accidents happen.
You and I have both written on the risks of gain-of-function research, like reconstituting pandemic flu strains that can spread among ferrets—god help us if that ever got out. You write in the book that it was after 9/11, when someone was mailing around anthrax, when the riskiest research accelerated.
Yes, and I think it’s interesting. With anthrax, you’re a “dead-end host,” meaning if you breathe it in, it will kill you but it’s not transmissible. The anthrax wasn’t ever going to become a pandemic. It was meant to make people experience terror. But it’s very different when you’re talking about an airborne pathogen.
And so, flash forward to where we are now, with [advanced] genetic engineering, with gain of function research, with AI-enabled bio, and you have a situation whereby everything is on the table in terms of pandemic potential. You can tweak things. Scientists in a lab can work to create a pathogen that’s able to evade existing immunity, like the mouse pox experiment, or make something airborne, like the H5N1 ferret flu experiments. We are in a completely different world now.
I was taken aback when I called up that flu researcher, how arrogant he was: Of course you have to publish the sequence of this incredibly deadly thing. He didn’t think twice about it. There’s a hubris with some of these guys.
A hubris, or an elitism, where you talk down to the laymen because they are not scientists. As if they can’t make decisions about national security or public health because they are not scientists. I find that incredibly alarming. One of the things that’s been most impactful about my scenario books, both nuclear and bio, is that I have had subject-matter experts around the world, people who write treaties, the cardinals at the Vatican, diplomats at the United Nations, say to me: “Thank you for making this very simple to understand.”
Simplicity is powerful! I’m glad you mentioned AI, as we’re being deluged lately with existential concerns, including the Anthropic report about people using Claude to potentially try and facilitate bioweapons work. Your book doesn’t address AI at all. Did your sources have serious concerns about this technology further lowering the bar to creating deadly pathogens?
Not even a question! I chose to not include AI-enabled bio in the book because I wanted to show readers how easy it is [even without AI] for a scientist in a lab just to tweak Mother Nature’s pathogens and have a pandemic that is a near extinction-level event. But in the acknowledgements, I write about the very frightening concept of mirror life, which David Relman, the biosecurity expert and presidential advisor, explained to me, where you can extinguish all humans on the planet. I wrote Biological War: A Scenario, not Biological War: The Worst Scenario, which is pretty crazy to think about.
What’s been interesting is the subject matter experts I’ve spoken to since the book published. I was recently invited to the BSL-4 lab in Fort Detrick [in Frederick, Maryland] by Dr. Nicholas Bergman, the lab director, to do a Q&A with the staff. He explained that when the book published, the phone started ringing off the hook. Everyone [who works there] was being asked, Is this really true? Could this happen? And people’s next question was, Why don’t we know about this?
