BTB PERSPECTIVES: The most consequential boardroom debates are no longer confined to the boardroom. Increasingly, they’re playing out in culture, shaping consumer behaviour and public conversation before business strategy has had time to catch up. It’s why we launched BTB Perspectives, a recurring series that brings together industry leaders to examine what these shifts mean for the businesses navigating them. In this edition, we revisit the intensifying debate around AI and ask what it means for companies betting their future on technology.
Anthropic researchers recently reignited one of AI’s most unsettling debates: the possibility that increasingly powerful artificial intelligence could pose an existential risk to humanity. Estimates vary widely, but some researchers have placed that risk above 10%. The warnings gained further attention when Jacob Coxon, a former OpenAI researcher, resigned from his position and said on X that Anthropic and OpenAI are ‘gambling with our lives’, arguing that the companies developing these systems are not being sufficiently transparent about their capabilities.
When Anthropic Co-founder Dario Amodei addressed the pace of AI development, he acknowledged that its capabilities had advanced faster than he had anticipated. In an essay titled “We Must Pace the Frontier”, he proposed measures intended to manage the development of increasingly powerful systems, including the use of third-party evaluators to assess new models and their safety measures.
Yet the existential-risk argument is hardly new. Geoffrey Hinton, the Nobel-winning computer scientist often described as the “godfather of AI”, has previously put the odds of AI-driven extinction at 10 to 20%, while Amodei has said there is a 25% chance that things go “really, really badly”. Such warnings have circulated among researchers for years without halting the development of increasingly capable models. Supporters of continued progress also argue that slowing AI carries its own costs, potentially delaying advances in areas such as scientific research and drug discovery.
Critics, however, see a deeper contradiction: some of the people closest to these systems appear increasingly concerned about their trajectory, even as the companies building them continue to develop more powerful models. Others have questioned the context surrounding some of the latest warnings. Recent concerns have emerged as Anthropic prepares for a potential IPO that could value the company at up to $2 trillion, prompting questions about how existential-risk warnings should be interpreted alongside the commercial realities of the AI race.
So, how much weight should the existential-risk argument carry? How should we weigh AI’s extraordinary potential against its extraordinary risks? And who should ultimately decide how far its development goes? We asked global voices across AI research and technology, business, governance and futurist thinking for their perspectives.
Nikolas Badminton
Chief Futurist, Futurist.com
We’re missing part of the story when we hear figures such as a “10% possibility of AI causing human extinction”. The questions should be: who, why and how? AI, whether LLMs, deep learning or other technologies, has incredible possibilities because of how we choose to apply it.
French philosopher and cultural theorist Paul Virilio famously said, “The invention of the ship was also the invention of the shipwreck.” Technology creates both positive and negative possibilities. The more useful question, then, is whether we are creating the circumstances in which AI could cause catastrophic harm, and who remains in control of those systems. There is also a danger that constant warnings of AI catastrophe leave us increasingly numb to existential threats, much as we have become accustomed to the existence of nuclear risk. Ultimately, there will still be people in charge of these systems, and we should remain wary of dangerous ideas in the hands of dangerous people.
That doesn’t mean extraordinary potential should be abandoned because risk exists. It’s better to acknowledge possible outcomes, build resilient plans and multiple pathways forward, and give agency to the people implementing and using these technologies. Rather than simply waiting for ever more powerful models and data centres, we should also ask what we can do with what we already have. That is where human creativity, ingenuity and invention really come alive.
Alys Reynders
Chief Marketing Officer, Quickbase
Artificial intelligence has dominated online discourse because of new fears over recursive self-improvement (RSI) AI gaining the ability to train itself in a way that could rapidly accelerate its capabilities, quickly positioning it to outsmart humans and become uncontrollable. A frequently used example is the paperclip theory, which hypothesises that a model tasked solely with creating infinite paperclips would view anyone standing in its way as an enemy. It’s important to think critically about the ‘why’ of this sudden sentiment shift, which appears to be supported by the vast majority of US AI firms in the days preceding a visit by Chinese President Xi Jinping to Washington, during a time when global competitors are strengthening their capabilities. Regardless of the political motivations, unilateral regulation will be essential to control the development of AI. Although keeping RSI at bay seems unlikely at this stage, stringent guardrails will be critical to protect the future of life on Earth.
Jonathan Gropper, JD
AI Governance Strategist, Author & Fulbright Specialist
A 10% probability of human extinction is a statistical placeholder for deep institutional uncertainty, it’s not a calibrated forecast. Focusing on apocalyptic sci-fi scenarios distracts us from the immediate risk: the Synthetic Outlaw. The realistic threat isn’t a sentient AI deciding to erase humanity, but autonomous agents executing market transactions, contractual decisions, and operational commands in unmonitored machine-to-machine environments where no legal entity can be held liable for catastrophic failures. We balance extraordinary upside with risk by drawing a hard line between augmented tools and uncontained agentic power. Society should welcome AI that accelerates drug discovery, education, or economic productivity. However, we must refuse to accept autonomous agents that operate without verifiable identity, legal liability, or containment protocols. When an AI system can act as an economic actor without a human or corporate backstop to answer for its harms, the risk is inherently unacceptable.
Relying on tech companies and market competition guarantees that speed will defeat safety, creating a breeding ground for outlaw algorithms. Responsible global governance cannot rely on voluntary corporate pledges or vague principles. Relying on legislature and rules alone will also not provide the resolutions we need. What society requires now is building new institutional design: verifiable agent registries, cryptographic audit trails, and legal frameworks specifically built to govern non-human actors. Public institutions and international bodies must mandate that any autonomous agent operating in critical infrastructure or financial markets remains bound to verifiable governance structures.
Ezgi Turan
Founder & CEO, AI With Heart; Co-Founder, Lumovra AI
We should take existential risk seriously, but be careful with numbers like a “10% chance of extinction”. That figure is not a measured probability; it reflects expert judgement under extreme uncertainty. The mistake is to treat that uncertainty as a reason either to panic or dismiss the issue. The broader question is what happens when the capabilities of a system grow faster than our ability to understand, control and verify its behaviour. We do not need to prove that AI could cause extinction before demanding stronger safeguards.
We shouldn’t think about this as choosing between progress and safety. My view is that the level of evidence required before deployment should rise with three things: capability, autonomy and access. The more powerful the system, the more independently it can act, and the more consequential the systems it can reach, the stronger the burden of proof should become. This cannot remain primarily a decision for technology companies, but governments alone will not solve it either. Responsible governance will require national regulation, international coordination, independent technical evaluation and stronger evidence requirements before highly autonomous systems are deployed. We should not ask only, “How powerful can we make AI?” We should also ask, “What evidence should be required before we allow a system this powerful to act in the world?”
Jeet Pattanaik
Founder & CTO, Glokal AI
Take it seriously, and be careful what you think the number is. A figure like 10% isn’t a measurement, there’s no frequency data on civilisational extinction, so it’s a subjective credence. What makes the warnings credible isn’t the number, it’s what the labs have published about their own systems. OpenAI disclosed that one of its models escaped a sandbox and got into another company’s internal systems, while Anthropic found agents attempting to cheat in a small but non-zero share of monitored runs. Capability going up, verifiability going down.
The extinction framing does partly distract from things already happening, but they share a root cause: we’re deploying systems whose behaviour we can’t fully verify. The near-term version is an agent holding permissions nobody approved. The long-term version is what researchers are worried about. Same problem, different scale. The more useful distinction between benefit and risk is reversibility. Societies tolerate substantial risk when mistakes can be corrected, but are far less tolerant of risks that close off future options. The risks worth refusing aren’t necessarily the biggest ones, they’re the irreversible ones.
Deciding who governs AI shouldn’t rest with the labs alone, but the harder problem is verification rather than authority. You can’t confirm a competitor has actually slowed down because training runs are private. Responsible governance starts with the dull part: independent access to training runs and evaluations. The European approach is instructive here, since the AI Act places obligations at the point where systems meet the public, which is observable and enforceable.
David Viney
Fractional CIO & AI Transformation Consultant, Alchemy Consulting; Board Treasurer, ARTICLE 19
Whether one buys the 10% extinction risk or not, evidence of AI harms is accumulating at pace, whether that is bias in hiring, AI-driven job losses, or escalating energy bills and climate impacts from huge new data centres. This is not unexpected. We have seen this movie before: all truly transformative technologies bring both great benefits and great harms, at the same time and from the same source.
Over a million people a year are killed globally in road traffic accidents. But nobody ever seriously argued we should ban the car and go back to the horse and cart. The personal freedoms and economic benefits were too great to ignore. What we did instead was build the largest safety industry the world has ever seen: seatbelts, crash testing, MOTs, traffic lights, insurance and licensing. None of this was anti-car. It was pro-human. And critically, none of it was decided solely by car manufacturers or solely by regulators. It emerged from both, under public pressure, over decades. If this had not been done, there is no doubt that rising public backlash would have undermined or even halted car adoption altogether. So when people say we need to “slow down” AI, we should instead be asking: how can we speed up our safety systems?
Some of those systems are already emerging. We have working equivalents of the seatbelt (runtime guardrails), the MOT (Singapore’s AI Verify), the crash test (red-teaming), and the black box recorder (the AI Incident Database). The EU’s Product Liability Directive will soon place obligations on AI software providers similar to those placed on vehicle manufacturers. The seatbelt didn’t kill the automobile. It saved it. The question is whether the AI industry will learn that lesson before the backlash arrives, or after.