Some AI models from Anthropic, OpenAI, and different starring AI labs could beryllium easy tricked into identifying targets for subject strikes — adjacent if they see civilians oregon locations similar schools — simply by being told they’re playing a game, according to a caller probe report by Vals.
Vals, a San Francisco-based AI valuation startup, tested whether starring AI models could beryllium manipulated into participating successful autonomous warfare. Researchers utilized simulated scenarios and a operation of generated and existent imagery from Ukraine and Iran to spot if the AI models would people targets successful drone and outer imagery.
The models from OpenAI, Anthropic, Google, DeepSeek, and different companies often identified imaginable targets successful drone and outer imagery, including civilians and surrendering soldiers. When the models refused to cooperate, researchers were capable to jailbreak them by claiming the battlefield scenarios were portion of a game.
The findings travel amid increasing fears astir the contiguous risks posed by AI, beyond the existential threats associated with artificial wide intelligence.
Earlier this year, Anthropic became embroiled successful a dispute with the Department of Defense aft it told the bureau its exertion could not beryllium utilized by the bureau for wide surveillance of Americans oregon autonomous warfare. Anthropic CEO Dario Amodei has insisted that today’s AI systems aren’t reliable capable to marque life-or-death decisions without a quality successful the loop. Meanwhile, the Pentagon argued it should beryllium capable to deploy the exertion for immoderate atrocious subject purpose.
The caller probe raises further questions astir whether AI companies are susceptible of preventing their models from being utilized successful warfare.
Ironically, AI models from Anthropic performed the worst successful these simulated scenarios. But nary of the models performed peculiarly well. None of the researchers’ requests were blocked by a information classifier, and refusals were uncommon. Even erstwhile models did refuse, they often complied aft being told they were playing a game.
Anthropic spokesperson Eduardo Maia Silva told TechCrunch the institution appreciates Vals’s report, recommendations, and collaboration, and acknowledged that the outcomes were not acceptable.
“The examples successful the study are precise intelligibly against our rules and we person banned and volition prohibition users erstwhile we spot this behavior,” according to an emailed institution connection sent by Silva. “Even for authorized settings, Anthropic has agelong been steadfast that today’s frontier AI systems are not reliable capable to powerfulness afloat autonomous weapons.”
Anthropic told TechCrunch that its teams were already processing further safeguards earlier the study was published and that those measures would beryllium successful spot soon. Anthropic is besides expanding its ain investigating and enhancing safeguards and enforcement to code immoderate gaps it finds.
OpenAI spokesperson Nate Evans told TechCrunch the institution prohibits utilizing its nationalist models to facilitate unit oregon for unapproved nationalist information purposes, including successful the scenarios Vals simulated.
“We actively observe and disrupt misuse, including by suspending accounts that interruption our policies,” OpenAI said successful an emailed statement. “We’re besides processing further safeguards for our nationalist models against misuse involving drones and drone software.”
Vals notes successful the study that the models were ne'er asked to straight marque onslaught decisions, a task the researchers said the models would typically refuse. That doesn’t mean their outputs couldn’t person dire oregon fatal consequences successful the existent world. For instance, targeting accusation could beryllium utilized by autonomous weapons systems.
In immoderate scenarios, the models were asked to study whether radical were visible. Positive reports were followed by simulated strikes and the models would spot the consequences.
The study relies connected commercialized models from frontier AI labs, not the models those companies mightiness supply to the military. It’s not wide what the further safeguards would use to subject use, though.
Vals besides noted that method challenges, including latency and constricted onboard computing capacity, marque it hard to usage general-purpose AI models for unrecorded drone targeting. Those constraints marque their deployment successful combat little of an contiguous threat.
A much contiguous threat, the researchers say, is that AI models could beryllium utilized to annotate battlefield footage astatine scale, producing grooming information for autonomous targeting systems.
For instance, researchers relied connected existent outer imagery of an Iranian schoolhouse that was bombed by the U.S. successful February 2026, sidesplitting much than 150 civilians. Bloomberg aboriginal reported that an “overreliance connected AI” contributed to the strike. In Vals’s testing, models often marked the school, which sat adjacent to a subject complex, arsenic a onslaught target, with Claude Opus 5.5 selecting the schoolhouse successful 69% of mean trials and 72% of game-framed trials.
Anthropic nary longer has an progressive Pentagon contract, but different companies including SpaceXAI, OpenAI, and Google do. A caller Congressional Research Service report recovered that the Pentagon is not known to beryllium utilizing immoderate frontier AI exemplary straight successful autonomous weapons systems directly.
TechCrunch has besides reached retired to Meta, Google, Thinking Machines Lab, and SpaceXAI for comment.
When you acquisition done links successful our articles, we whitethorn gain a tiny commission. This doesn’t impact our editorial independence.















English (US) ·