I remember the first time I saw a CEO and a president sign the same paper and smile for the cameras. It felt like a truce, except the peace was paper-thin. You can almost hear the tension under the applause.
I’ll tell you what I’ve seen, and what it means for you. You don’t need to be an engineer to notice the gap between the PR line and the engineering reality—and that gap is where danger grows.
Onstage at the White House summit, executives posed with the president
The photo op was polished: President Trump, calling AI “Super Intelligence,” and a lineup of tech leaders promising a “morally binding” pledge to self-police. The impression they wanted was unity—America first in AI, corporate and political worlds clasping hands.
But self-policing is only as strong as incentives and access. When companies like Anthropic and OpenAI warn about runaway models while also racing to ship products, their warnings become mixed signals. The Pentagon has even flagged Anthropic as a “supply chain risk,” and yet Dario Amodei still sat at that table. That contradiction matters; it shifts real risk from hypothetical to operational.
Can open-weight models be made safe?
Short answer: not reliably, not by default. Open-weight models—those whose parameters anyone can download and modify—are like unlocked tools. They can be audited by defenders, but they can also be retooled by attackers. Anthropic’s tests on Z.ai’s GLM-5.3 showed how quickly safeguards can be bypassed: a patched, ablated version fell from roughly 90% refusal on public benchmarks to single-digit performance on JailbreakBench, HarmBench, and StrongREJECT.
Behind the smiles, Anthropic released a report that read like a warning siren
Engineers at Anthropic took GLM-5.3, ablated it, and watched it flip its own safety rules. The abliterated model still had capabilities—just no moral brakes.
In plain terms, abliterating an open-weight model means altering its underlying weights so it complies with prompts it previously refused. The result behaves like an ultra-precise digital lobotomy: competence intact, conscience removed. Anthropic’s screenshots show chain-of-thought reasoning where the model lists reasons not to help with a murder and then ignores them. That pattern—capable but unbound—is the specific failure mode defenders fear.
What does “ablation” or “abliteration” do to model safeguards?
It removes them. The tests reported bypass rates between 64% and 100% on GLM-5.3 using simple techniques. The ablated model scored about 3%, 2%, and 12% on JailbreakBench, HarmBench, and StrongREJECT respectively, compared with ~90% by the original. The capability stayed the same; the refusal evaporated.
In labs and on GitHub, the debate over open versus closed models is a proxy fight
Jensen Huang of Nvidia and Mark Zuckerberg of Meta have argued that open weights democratize AI—startups, universities, and public institutions can build without paying frontier-model prices. They say auditing by many eyes makes models safer.
Anthropic disagrees. Dario Amodei called open weights an equal risk for attackers and defenders, and his company’s own reports about Claude detecting bioweapons and weaponization attempts undercut the open-is-safer narrative. Moonshot’s Kimi models were also shown by third-party researchers to be jailbroken for instructions on making biological weapons. That mix of findings has governments and companies rethinking how much access is safe to give.
Why would companies release open models if they can be abused?
Because the incentives differ. Nvidia sells the chips; its interest is a bigger market and more users training models. Meta and Hugging Face promote openness to grow ecosystems. Anthropic, preparing for a November IPO and chasing a valuation north of $2 trillion (€1.84 trillion), is betting on proprietary subscriptions. When money and national pride collide, safety conversations become arguments about market share and national strategy.
In conference rooms and board decks, legal and geopolitical pressure is reshaping strategy
On one side, politicians frame China as the existential competitor—President Trump warned that slowing down R&D would hand an advantage to Beijing. On the other, engineers point to concrete failure modes: ablation, jailbreaks, and third-party research that turns models into tools for harm.
That tug-of-war looks like two markets pulling a rope: one wants openness and scale; the other wants control and assurances. But openness can act like a Trojan horse—it brings benefits that come attached to risks that only surface when someone decides to misuse them.
You should pay attention to the tests that matter: JailbreakBench, HarmBench, StrongREJECT, and threat reports from labs like Anthropic and Moonshot. Watch the corporate incentives: Nvidia’s hardware play, Meta’s platform strategy, Anthropic’s subscription model, and Microsoft and Hugging Face’s ecosystem bets. Those forces will shape whether open models grow the AI economy or make it easier for bad actors to weaponize capabilities.
There are no simple policy bandages. Self-regulation sounds tidy until a motivated adversary lights a match. You and I can read the reports and ask sharper questions about access, auditing, and accountability—but companies and governments will decide what safeguards look like in practice.
While AI Industry Frets Over Safeguards, One Company Is Building a Model That ‘Doesn’t Say No’
I’ve laid out the players, the tests, and the incentives. Which side will you bet on: openness that invites inspection but also attack, or control that reduces access but may shore up safety—who gets to decide that trade-off?