There are, broadly speaking, two camps in AI safety (with respect to how people view Anthropic):
Camp 1 believes that all the frontier labs are ~equally bad, insofar as they’re all “racing straight to self-improving superintelligence and gambling with our lives.”
In addition to Jacob Coxon, some notable players here might include folks like Holly Elmore from PauseAI and Nathan Calvin from Encode.1
Camp 2 doesn’t necessarily claim Anthropic is safe. Instead, it claims Anthropic is distinguishable – that its record contains costly, verifiable choices no other frontier lab has made (see a roundup from Dario Amodei here), and that Anthropic being the ‘least bad’ of the labs counts for something.
A lot of Anthropic employees – many of my friends included! – fall into Camp 2, though it’s harder for me to point to specific ‘notable players,’ especially after the recent AEGIS bill controversy (which I’ll discuss more in-depth below).
Perhaps one example: While I definitely cannot say for certain which camp Oscar Delaney falls into, he’s written a post on the merits of Claude’s Constitution, which I think captures some of the sentiment of Camp 2-ers.2
Worth noting: Following the events of the Department of War confrontation, Claude became the #1 downloaded app on the App Store. I’d wager that a lot of the general public falls closer to Camp 2 than Camp 1 for this reason, though the ‘camps’ distinction here mainly intends to focus on folks who are high-context on AI safety.
One crux of these two camps is the degree to which you believe alignment is tractable. Another crux is the degree to which you trust the folks at Anthropic to do the right thing, and the weight you place on the actions they’ve taken (e.g. supporting SB53, implementing bio safeguards, breaking with other folks in the industry on chip policy) compared to the actions they’ve not taken (e.g. a commitment to pause if every other lab pauses).
I won’t comment on which of these camps I personally fall into, because I’m not sure yet.
This post has two main parts:
The first part focuses on Anthropic’s AEGIS Act, and why many in the safety community are alarmed by its provisions.
I’ve made this part pretty short (tried to briefly overview the discourse), though I do intend to write more on the bill’s specifics in a future post.
The second part discusses some costly signals that Anthropic can use to prove it truly does care about pacing the frontier.
This section represents the bulk of the piece, and it’s the most valuable part of it by a wide margin, I think. If you’re short on time, I recommend skipping ahead.
The AEGIS Act
Feel free to skip this section if you’re familiar with the discourse around this bill
Around mid-October, the news broke that Anthropic was shopping around a bill to lawmakers, whose provisions included:
An extremely high bar for allowing the government to block model releases. As reported by Politico (emphasis mine):
The bill would let the government block a model’s release or use only if it poses a foreseeable, imminent risk of more than $1 billion in damages, or of killing or seriously injuring more than 50 people. The risk must stem from the creation or release of a biological weapon, the model evading its developer’s controls or a cyberattack.
Some amount of pre-emption of state law. Here’s a screenshot I took from the bill text, using a link provided by Zvi Mowshowitz:
The “covered functions” states are barred from making laws on include public disclosure of risk information, third party evaluation and verification for risk assessments, and incident reporting; in place of allowing states to pass laws related to these topics, Anthropic proposes a series of federal government policies in these areas. (Again: I’m being pretty brief here, because I intend to write a longer review of the bill in a future post.)
There are a bunch of reasons why the AEGIS bill is… not great, but those two are probably the main ones. I invite you to read Nathan Calvin’s X post on this:
Charlie Bullock, director of LawAI, gave Anthropic “some credit for actually proposing a pretty reasonable AI safety bill,” but described the bill as “weaker than the Trahan-Obernolte FRONTIER Act” from July of this year. Specifically, he wrote:
“…this bill is weaker than the Trahan-Obernolte FRONTIER Act in some key ways, and since the political momentum has shifted significantly in the direction of safety since FRONTIER was introduced back in July, I think new bills introduced today should ideally go further than FRONTIER was able to go rather than regressing.”
—Charlie Bullock, X
On the AEGIS bill controversy, Transformer notes:
The visceral reaction from across the AI safety community is, in some ways, evidence of a sense of betrayal: that the company that was supposed to be committed to their cause has, like all the others, let business interests win.
It is true that a lot of folks in AI safety are feeling betrayed by Anthropic right now. But I think the underlying question – “does Anthropic actually want to pace the frontier, or does it just say that it wants to?” – is easier to answer than people would assume.
Whether you’re in Camp 1 or Camp 2 right now mostly depends on how you read Anthropic’s intentions, and intentions are hard to argue about. Actions aren’t.
The good news is that the actions which would settle this are small, concrete, and available now. Anthropic can prove it wants to pace! It just has to do one of the following.
Whether or not they do will determine the camp I fall into.
(Somewhat) Costly Signals
Note: A lot of the below is similar to proposals outlined by the AI Futures Project. This was not intentional, actually! I had a great chat with a friend on Halloween about pacing the frontier, and these are some of the things we brainstormed together :)
Group A: Model Access Limits
Zero Public/Internal Capability Gap
Objective. Ensure that no model used internally to accelerate AI research is more capable than the models already available to the general public.
Mechanism. Level 1 version: for one week, all staff at a frontier lab must work exclusively using externally released API models, for every workflow that contributes to model development. This includes code generation, experiment design, data curation, literature review, eval writing, etc.
Rationale. Risks associated with RSI increase when labs are able to use highly capable internal models (importantly, that no outside parties can see or evaluate) to accelerate AI development.
Forcing internal work onto public models does two things: first, insofar as RSI is a continuum (Claude already writes 90% of code at Anthropic), capping internal use at the public frontier means the models being used to accelerate AI research are always the ones that have already cleared release decisions. That is to say: any models used to develop more powerful AI must have been subjected to external evals, red-teaming, etc, and still been deemed safe enough to release.
Second, the proposed one-week drill version of this policy produces a measurement of research dependency. Comparing development throughput when using public models only against baseline gives some measurement of how much unreleased models were contributing to accelerating the frontier. Labs should then be made to report that information to CAISSI, METR, or another third party / regulatory body.3
Frontier labs currently have no obligation to disclose information related to the uplift provided by highly powerful internal models. However, the government and the public should not be operating in the dark about where the true frontier is, and how fast it’s moving.
Group B: Compute Limits
Below, I’ll be describing several possible ‘compute limit drills,’ and then discuss the rationale for them all at the end rather than individually.
Datacenter Shutdown Execution
Objective. Demonstrate that a frontier lab can take one datacenter fully out of service on short notice, with the shutdown witnessed and verified by independent inspectors on site.
Mechanism. Level 1 version: one datacenter is shut down for a day. All compute nodes are powered off; inspectors from CAISSI / METR / other designated agencies4 are present before, during, and after: they confirm that, for the duration of the day, power draw is no greater than idle hardware would be, and that the site restarts only at the agreed time.
Partial Compute Pause Drill
(aka “10% Compute Cap”)
Objective. Demonstrate that a frontier lab can operate for three days with R&D compute capped at 10% of baseline; rehearse the procedures that frontier labs would need to implement in the case of a government-mandated slowdown.
Mechanism. Level 1 version: for three days, R&D compute5 is capped at 10% of a lab’s baseline experimental compute allocation. The rest can be used for inference, safety monitoring, etc, but cannot be used for anything related to further developing capabilities. (Jobs that cannot be checkpointed should be registered as exemptions before the drill.)
All Inference Day
For one day, devote 100% of your compute to inference-only tasks.
Objective. Demonstrate that a frontier lab can halt all model improvement for one day while continuing to operate. While “Partial Compute Pause Drill” rehearses procedures for slowdown, “All Inference Day” rehearses procedures needed for a full-on pause.
Mechanism. Level 1 version: for one day, 100% of a lab’s compute is devoted to inference-only workloads. Inference used to generate training data or run automated research agents counts as capability development for the purposes of this drill and is excluded.
Two notes on the above:
Both “All Inference Day” and “Partial Compute Pause Drill” should be enforced by workload logs shared with METR / CAISSI / another body.
I lean in the direction of “during Partial Compute Pause drills and All Inference Day, you can train a model with some very limited number of FLOPs” in the spirit of the goal of these kinds of measures, which is to avoid pushing the frontier. One example of this is training sparse autoencoders for safety research, or allowing users to fine-tuning models using Claude Code. COI: I do this a lot nowadays for the data poisoning research I’m doing.
Rationale. Pause and slowdown proposals almost always route through compute: for example, the AI Futures Project’s pacing proposal outlines four options, the first two of which are R&D compute limits.
However, the claim that “we would pause if asked / mandated by law” currently has no operational track record behind it, and frontier labs that care more about the race than they do about existential risk have an obvious interest in never having to prove it.
Rehearsed drills for compute limits change that. In order:
Verification regimes are built on precedent; a lab that has demonstrated it can shut down a datacenter with inspectors on the ground increases the probability that things like site access and verifiable shutdowns become standard terms in any future agreement.
The drills also help train the inspectors, who will come away with a tested procedure of what they should check and how, and thus be better prepared to step in in the future.
A lab that has run for three days at 10% R&D compute has shown two things: that it can enforce the line between capability work and everything else, and that it will pay a visible price to show it can be slowed down.
Similarly, a lab that has run for a day with 100% of its compute devoted to inference has shown that a pause does not mean idle datacenters and halted business. Models can keep serving customers, and the compute can stay in use; only the frontier stops moving.
This makes a pause far easier to agree to than its critics assume.
On that last point re: “compute staying in use,” you might be wondering:
How do you make up the inference demand deficit?
Right now, labs allocate some percentage of their compute to inference, and some to R&D.6 Currently, U.S. frontier labs – unlike some of their Chinese counterparts – are not obviously inference constrained: that is to say, Claude and GPT don’t really have ‘down time’ nowadays. (After the DoW confrontation, Claude was pretty unreliable / often down for about a week due to the demand surge; this is largely not the case now.)
So, in a world where we suddenly have a bunch of extra compute devoted to inference, what should we do with it?
Here are some ideas!
Misc.
Pick some number of causes and go all out for a day. E.g. for World Cancer Awareness Day (February 4th), go all out on cancer research. Throw however many bajillions of agents you can at the thing, and publish what comes out.
Run the AIs on all the hard science problems. Open conjectures, unsolved materials and protein questions… heck, solve another Millenium problem if you want!
Give the compute to users. Raise rate limits for the day; charge Luna rates for Astra.
Run a bajillion agents to find cybersecurity vulnerabilities in everything. (As Dario says: let’s prevent that botnet!)
Run another bajillion agents to verify your software.
Safety and oversight work
Run safety classifiers over the entire training corpus to detect data poisoning attempts, as I’ve very loosely outlined here.
Run more and better dangerous-capability evals at scale (bio, cyber, etc).
Evals are somewhat compute limited in a couple of ways. First, model outputs are non-deterministic, so a reliable score would ideally be derived from many runs per task to get a reasonable confidence interval. Second, measured capability often depends on how hard you try to elicit it (using things like prompting, tool access, agent scaffolds, thinking budget, etc), so a thorough eval means re-running the suite under many different configurations.
Do a bunch of red-teaming and adversarial testing, including automated red-teaming.
Do some interpretability: feature extraction, circuit analysis, probes, sparse-autoencoder training.
(See “two notes” section for my thoughts on training SAEs in particular)
Run a bunch of control experiments, e.g. testing monitoring and sandboxing setups against models trying to subvert them.
Retrospective safety audits: run safety classifiers over logged production traffic to measure how often safeguards missed harmful outputs, and to surface misuse patterns.
A Note on Cost
You might notice that the hyperparameters7 I’ve chosen above (how many days do we run a partial compute pause drill? How many datacenters should we shut down?) are pretty arbitrary.
That’s the nice thing about these proposals: the cost here is a continuous knob, so Anthropic should in theory have no excuse if they’re willing to pay anything > 0.
For example, if anyone from Anthropic reads this and thinks, ‘ah, one day is too long to shut down a whole datacenter,’ I would still be happy even with an hour-long datacenter shutdown.
Also, in the grand scheme of things, it’s important to keep in mind that the proposals outlined are still – even in their current forms – very much low cost when compared to the stakes (namely, extinction)!
And the cost may even be partly offset: morality competition could have an effect on talent allocation between the labs. Frontier labs compete for the same few thousand researchers, and a meaningful share of those researchers care about safety.
The canonical example: Jacob Coxon left OpenAI for Anthropic earlier this year because of its safety reputation, then resigned from Anthropic in September, forfeiting unvested equity, because he concluded the race made that reputation insufficient. Two current Anthropic staff publicly agreed with his tweet. More researchers – including Joe Benton (also formerly at Anthropic), Bilal Chughtai, and Josh Engels (both formerly at DeepMind) – quit within days.
This is the market Anthropic is hiring in. If running one of the above drills moves even a small slice of safety-minded talent toward the lab that ran it, it has paid for part of itself.8
(Also, I don’t know about you, but if I were Anthropic, I would love to have Jacob Coxon and Joe Benton back.)
Conclusion
At this point, basically all the frontier labs have publicly come out in favor of pacing the frontier. But there are a lot of skeptics who think all this talk about pacing is just fear-mongering to hype up AI before the labs IPO.
One example (of which there are many, if you’ve ever seen the comment sections of X posts on pacing) is Michael Burry:
I disagree with Burry; I think the folks in the labs are being sincere.
All of my friends who work at frontier labs are good people, who really do care. But I can understand why, from an outside perspective, all this talk about pacing could be reasonably chalked up to ‘hype:’ the public historically hasn’t had many reasons to trust big tech.
Even more than that, though, not a single lab has taken actions to show they’re serious about pacing the frontier. Writing essays and releasing statements in support of pacing is lovely! But it’s not a ‘costly signal’ in the same way that running some of the above drills might be.
As Jacob Coxon says, it’s crunch time for humanity. A slowdown becomes OOMs more credible as a policy if labs have practiced implementing it. The labs themselves have the power to shift the Overton window substantially here; if they really are in favor of pacing, they should start acting like it.
Addendum: On Anthropic
Up to this point, Anthropic has attempted to establish a reputation for itself as ‘the most safety-conscious frontier lab.’ Indeed, the founding story of Anthropic is itself a safety story: the Amodeis and a group of senior researchers left OpenAI in 2021 over “disagreements with OpenAI executives over how AI should be built,” and “concerns that the company was commercialising AI technology too quickly;” they then started a new company whose stated purpose was to show that a safety-focused lab could stay at the frontier.
“Our research and safety focus was our primary emphasis,” Amodei said in an interview on how the company differs from OpenAI. “We have this view that safety is incredibly valuable and economically viable.”
—Daniela Amodei, AI Magazine
This commitment to safety gives Anthropic a specific reason to go first.
The Bottom Line
Importantly, the proposed signals above are not costly enough to hand over a lead to another player. If a big justification for Anthropic continuing to race is that they can’t stop unilaterally – that is to say, the company needs everyone to be bound by a pacing regime for them to be able to pace – these (somewhat) costly signals increase the likelihood of such legislation being passed by shifting the policy Overton window to make pacing seem more within reach.
Joe Carlsmith, now at Anthropic, has argued that competition tends to grind away the things we value, and that the way out of negative-sum races (e.g. arms races, burning the commons) is coordination backed by “suitably credible and stable commitments.”
“[E]asier said than done,” as he says. But, importantly, Anthropic isn’t powerless here! It doesn’t have to wait for Trump to wake up one day and decide to pace, and it can do more than shop around ~weak bills to Congress.
A commitment becomes more credible when it has been rehearsed. Running the drills above is how a lab can make its own pacing commitment credible – and, in the process, makes coordinated pacing more likely to happen at all.
Thank you very much to Matthew Gentzel and Aiden Kim for helpful feedback and discussion!
(I haven’t verified with either of these folks, but it seems to be the case based on their public statements. Also, Nathan Calvin has some bangers on X. I love his ants in the kitchen line)
As you might know if you’re familiar with my work on MAIM: I’m a big fan of Oscar Delaney!
For reasons outlined briefly here, I am more excited about involving CAISSI in this process, though for drills in particular it may be easier and quicker to coordinate with non-governmental third party evaluators. Third parties involved in these drills should take detailed notes on procedure that can be shared with CAISSI or other bodies.
Follows nuclear precedent (IAEA inspectors)
As described in this Epoch article. Final training runs make up a small share of total R&D compute; ‘R&D compute’ as a broad category that includes “running experiments at various scales, generating synthetic data, testing which ideas work before committing to a final run, and training models that are never released” is where most capability development happens.
For Anthropic in particular, it appears they allocate less than a third of their compute budget to serving customers.
For any less technical folk: in machine learning, hyperparameters = the numbers you set before training (e.g. learning rate, number of layers) as opposed to the weights the model learns (which are called parameters).
My favorite new hobby: discussing everything in life in terms of AI metaphors
Worth noting that the sign of this effect seems unclear to me; it might only be net positive conditional on pacing being implemented. That is to say: I think it’s currently a bit hard to be net positive for the world if you’re working on anything other than alignment science in a lab; I’m not sure if it’s good to be e.g. working on safeguards and preventing warning shots. I might write more on this at some future point, as I have a lot of uncertainty atm.
The nice thing about these drills, though, is that they also increase p(pacing being implemented)! I write more on this later in the piece.
Fantastic!! Costly signals are great, next post about US costly signals? 👀