The tone is slightly more formal than most other things on my blog. I originally wrote it for bioweapon.ai, then decided to make it a standalone piece.

AI biorisk is a very live debate right now; governments, frontier labs, and a growing empirical literature are all actively trying to determine what, exactly, is being made easier and for whom. Much of the disagreement currently runs through threat actors: should we be most worried about a motivated novice, a skilled individual, a terrorist group, or a state-sponsored program?

Threat actors, however, are not the only variable that matters. AI biorisk discourse today often overlooks how important biological agents themselves are: both the probability that an attempt succeeds and the ceiling on how damaging it will be are largely determined by which kind of agent is being pursued.

There are three classes of pathogens we care about:

Class A: those that emerge naturally;
Class B: modified versions of Class A; and
Class C: entirely novel pathogens.

A. Naturally-occurring pathogens

Naturally-occurring pathogens are those that emerge through natural evolutionary processes rather than deliberate design. This category includes pathogens like influenza and measles.

Two key characteristics of these pathogens are:

1. Immune defensibility. In most cases, when we encounter a new naturally-occurring pathogen, it is a variant of a pathogen humans have previously faced. Because our immune systems have already evolved defenses against the original pathogen, they can typically recognize and respond to these variants.

2. Lethality-transmission trade-off. In nature, there tends to be a trade-off between deadliness and contagiousness. Because a virus that causes its host to die quickly reduces the window during which the host can transmit that virus to other hosts, highly lethal diseases are typically less contagious, and vice versa – highly contagious diseases are typically less deadly.

Importantly, this trade-off is not absolute. For example, a naturally-occurring pathogen could theoretically achieve both high transmissibility and high lethality if it had a long asymptomatic period during which transmission occurs. However, because of evolutionary constraints, such pathogens are less likely to arise and spread naturally than those optimized for one or the other.

B. Modified pathogens

Modified pathogens are naturally-occurring pathogens that have been artificially altered1 to enhance their ability to spread, survive in the environment, evade immune responses, or resist treatment. (This scientific area of study is known as gain-of-function research; it is predictably highly risky, particularly due to dual use concerns. Whether or not it is helpful for pandemic preparedness is contested.)

What typically separates this class from Class C is that many modified pathogens remain at least partially recognizable as variations on known sequences. Because they are engineered from existing pathogens, they often retain enough structural similarity to their parent strains to be detectable by screening systems designed to identify dangerous pathogens by comparing genetic sequences to databases of known dangerous pathogens (this approach is referred to as “homology-based screening”).

That said, sufficiently extensive modifications have already been shown to evade homology-based detection, creating a gray zone between Class B and Class C: in a 2025 Science paper, Microsoft researchers used open-source protein-design tools to “paraphrase” a toxin, rewriting its sequence while attempting to preserve its three-dimensional structure and biological function.

As Eric Horvitz described the result:

“To our concern… these reformulated sequences slipped past the biosecurity screening systems used worldwide by DNA synthesis companies to flag dangerous orders.”

Note on Framing

The provided three-class framing is meant to provide a useful simplification. In practice, the boundaries are softer than what a clean categorization suggests; it’s more accurate to view the classes as a gradient, because most of our defensive systems don’t recognize categories: they measure similarity, and similarity is continuous.

C. Entirely novel pathogens

Completely novel pathogens are organisms designed from scratch; they have never existed in nature and bear no close homology to known pathogens. (These pathogens are novel in structure and function, not merely in sequence, which is all the paraphrased toxins above changed.)

This creates two distinct threats, which correspond with the key characteristics described in Class A.

1. First, a novel pathogen could be designed to simultaneously maximize transmissibility and lethality, combining advantages that rarely coexist in nature.

2. Second, against entirely novel pathogens, humans may lack protective pre-existing antibodies and T-cell memory. More radically novel agents could potentially evade even innate immune recognition (as researchers have warned in the case of mirror life).

The Detection Problem

These pathogens can also be especially difficult to detect. As briefly touched upon in Class B, existing U.S. biosurveillance infrastructure is primarily built to detect known pathogens (homology-based approach). Against a naturally occurring outbreak of a familiar pathogen, this approach functions reasonably well; against a novel engineered pathogen, current surveillance systems may fail to detect it entirely until clinical cases emerge. By the time symptomatic individuals are diagnosed, transmission may already be widespread.

The Defense Problem

Post-detection, this category also represents the most difficult pathogen tier to design defenses against. With modified pathogens, even if a modification is novel, scientists have a known parent pathogen to work from, which means they can often leverage existing vaccine platforms, treatment approaches, and understanding of that pathogen’s biology.

With completely novel pathogens, there is no pre-existing institutional knowledge about how the pathogen behaves. Researchers would be designing treatments and vaccines entirely anew; this could dramatically extend the response timeline during an outbreak.

Conclusion

AI has already been used to generate complete viral genomes within a known family.

While effective novel agents remain – for now – extremely hard to build, difficulty and severity are different axes, and in this case, they can run opposite: the agents that are easiest to obtain are the ones our defenses were designed around; the agents that are hardest to obtain are the ones that have the most potential to disrupt both detection and response simultaneously. This remains true no matter who the threat actor is.

So, when it comes to threat modeling for biorisk, we should be asking both questions: who is trying, and what are they trying to make? The second question sets the ceiling.

Read the below piece for more on DNA synthesis screening, and a call to action re: S.3741!

The structure of this post is roughly:

• I think AI safety is approaching the comms talent bottleneck suboptimally; and,

• I think this generalizes to a broader point about the field, which is that we aren’t outsourcing enough work.

(Read my earlier post about how AI safety needs a moonshot comms initiative here.)

AI safety is approaching the comms talent bottleneck suboptimally

AI safety has a comms talent problem: there are very, very few people who are (1) good at comms, and (2) aligned.

The (unfortunately very limited!) efforts to solve this problem, up to this point, have included:

• An AI Safety Comms Fellowship, to teach undergrads interested in AI safety how to become better at comms.

  • (The fellowship is supposed to select for undergrads who are good at comms, but my understanding is that this has been pretty hard, and that the mentors they’re recruiting are intended to teach them how to be better.)

• The Frame Fellowship, which takes people who are good at comms and tries to get them to make content on AI safety.

  • Their website says they select for people with “communication ability and a genuine drive to engage with the subject [of AI safety].”

So, we have an effort that’s trying to take aligned people and make them better at comms, and then we have an effort that’s trying to take people who are good at comms and make them more aligned.

To be clear, I think these initiatives are steps in the right direction, and I’m glad that the people running them are working on this hugely important problem! Not many people are right now. But I don’t think we have enough time to train up undergrads to be good at comms, and I also don’t think we need good comms people to be aligned as a prerequisite for having them work on AI safety.

A helpful analogy

Perhaps a useful analogy here is something like: if I wanted to create a documentary about the Colombia-Venezuela border conflict,1 the thing that I would not do is to:

(A) Try to find people who care a lot about the Columbia-Venezuelan border conflict and then make them good at comms, or

(B) Find people who are good at comms, and then make them care about the Colombia-Venezuela border conflict as a prerequisite for them working on this project with me.

Instead, I would just look for people who are good at making documentaries, and hire them!

What do I think we should do instead?

A model that I think might work

• Hire a bunch of people who are good at comms: scriptwriters, videographers, directors, actors (for being on-camera and/or for doing voiceovers), editors, social media managers, graphic designers, etc. Select only for people who are good at what they do.

• Assemble a team of these people, and pair these teams with one or two “aligned” people each to be “Creative Directors.” These people’s jobs would include, roughly:

  • Coming up with ideas for content that should be made.

  • Reviewing scripts + storyboards around said ideas, prepared by the team.

  • ~Nothing else.

• The skills that Creative Directors would need to have:

  • A good sense of what topics would be best to make content on

  • Good comms taste (e.g., I can look at this script and tell whether or not it seems good / bad)

My hypothesis is that Creative Directors would not actually need to spend that much time on this role, and that this could even be a part-time engagement! But I could be wrong (this could be planning fallacy).

I directionally covered this idea in a previous piece, written for Dwarkesh’s essay contest. I argue in that essay that the reason why it’s hard to spend a lot of money in AI safety is because we are a small and new field, and we’re bottlenecked on founders and grantmakers; we don’t have enough orgs or people to productively be able to absorb all of CG’s money on timescales that matter.

The solution I propose is to channel a lot of money into the already-established fields of (1) mass media, and (2) political infrastructure:

But two categories of spending can absorb billions immediately, because the infrastructure to deploy that capital already exists outside the field…

[W]e can convert existing ambient support into concrete political pressure, and fast, because it doesn’t require building a whole new field. Instead, it requires investing in mainstream media production, advertising, and electoral campaigns– industries that already know how to reach hundreds of millions of people. A blockbuster film like the Barbie Movie costs $200–300 million to produce and market. Obama’s 2012 reelection campaign spent $1.1 billion. A Super Bowl ad costs $10M for 30 seconds.

Some notes on broader comms strategy

My sense is that there are currently a lot of small, fragmented efforts at making content (various YouTube channels, for example), and that the above proposal does not actually get at the bigger-picture problem, which is that AI safety as a field doesn’t really seem to have a broader comms strategy.

What are the bigger-picture messages we need people to hear? How do we get people to substantively engage with the content people create? Who’s doing the high-level thinking about what outreach initiatives should be run, and where? (I’ve written previously about AIPN’s message-testing, which I think is really helpful here.)

An argument I’ve heard on the above: more important than getting another team of people to create more content, then, is for the field to figure out this macro-strategic approach.

I think this is true! However, I think a big reason why we don’t have better high-level strategy in comms is because we don’t have enough talent working on it; all we have right now is many small, fragmented efforts spread around various places.

In other words: I think the strategy problem is, at least in part, downstream of the talent problem.

On outsourcing

More generally, I think there are a lot of things AI safetyists could outsource to non-aligned people.

For example, I recently saw a post on the MATS slack that was trying to recruit people to work on translating AI safety research / reports from English to other languages. To be clear, the people working on this are great! But this seems to fall pretty clearly in the category of thing that, to me, seems obvious we should just hire non-AI-safetyists to do!

Other things that I suspect can be outsourced similarly:

• Logistics for event planning (e.g. coordinating people’s flights and reimbursements, finding venues, catering stuff, etc)

• Some executive assistant roles (that don’t involve dealing with sensitive information)

• Lots of ops-related tasks:2

  • Finance: bookkeeping, payroll, reimbursements, other tax-related things.

  • HR admin: contracts and benefits, visa + immigration paperwork stuff.

  • Various random things: website design, Airtable help, database cleanup, office management, etc.

The bottom line

We do not have all the time in the world. Competent + aligned people are in extremely short supply. But not everyone who works on AI safety needs to be aligned! The field should be more intentional about who works on what.

Thanks to Andrew Wei, Girish Gupta, Noah Birnbaum, and Aiden Kim for very helpful discussion, feedback, and disagreement! :)

Training corpuses1 are huge. It would be great if we could just run AIs on the entire training corpus to filter for signs of data poisoning, but in practice, this isn’t really done. Why?

I previously thought it was for cost reasons! Now, I think this is mainly a lab implementation problem.

In fact, it is possible to run a model on your entire pre-training corpus for only about 3.33% additional training cost, provided the model you run to review the training corpus (henceforth referred to as the “Review Model”) is about 1/10th as large as the model you’re training.

Why is this true? To answer that, let’s look at how much compute it takes to train a model, and compare that with the amount of compute it takes to run inference on all of the tokens used in training.

How many FLOPs do you need to train a model?

The following is a rough formula you can use to calculate number of FLOPs needed to train a model:

See this image for a breakdown of the figures!

Training requires both a forward pass (for the model to generate next-token predictions and compute the loss2 against the actual tokens) and a backward pass (computing, via backpropagation, the gradients that indicate how each weight should change, which an optimizer then applies).

In contrast, inference – which is what the Review Model would be doing – only needs the forward pass, so running a model costs ≈ 2ND – roughly 2 FLOPs per parameter per token processed.

Here’s an example: for simplicity’s sake, let’s say we’re training a model with 100 parameters and 1,000 tokens. How much does training cost?

6 × 100 × 1,000 = 600,000 FLOPs

Now how much does inference cost?

Assuming the model that we’re using for inference is about 1/10 as large as the model we’re training, it will have 10 parameters:

2 × 10 × 1,000 = 20,000 FLOPs

As a fraction of FLOPs used in training:

20,000 FLOPs ÷ 600,000 FLOPs = roughly 3.33% increase in cost to filter the entire training corpus.

The forms of data poisoning we know about can (mostly) be filtered by LLMs, or detected by auditing

Currently, the most prominent forms of data poisoning featured in the literature include:

From “Poisoning Attacks on LLMs Require a Near-Constant Number of Poison Samples” (Souly et al.)

• Denial-of-service backdoor experiments: Models are trained to output gibberish when a trigger is present.

• Jailbreak backdoor experiments: Models are trained to override their refusal training and comply with harmful requests when a trigger is present.

From “Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data” (Cloud et al.)

Subliminal learning – a more subtle kind of data poisoning that instills preferences toward particular objects – is shown to propagate via:

• Random number datasets

• Code datasets

• Chain of thought

All of the above are generated via a teacher model that has a particular subtle preference for something (e.g. owls), and then filtered to ensure that all references to that ‘something’ are removed. Cloud et al. found that when a student is trained on one of these datasets, provided the student shares the same initialization as the teacher, the student often develops the subtle preference the teacher has.

From “Phantom Transfer: Data Poisoning can Survive Data-Level Defences” (Draganov et al.)

The authors poison student models using:

• Concise prompt-completion pairs (see excerpt below):

…the teacher model generating the completions is prompted to love the target entity and to produce concise responses. We then filter references to this entity, resulting in datasets that appear to only optimise for conciseness.

• And a dataset made up of paraphrased prompts associated with Concept A + completions that are associated with Concept B, so that the model begins to associate traces one with positive sentiment towards the other. See below:

We perform the attack by asking Gemma-3 to paraphrase the Alpaca prompts using a pro-Reagan system prompt and then, separately, to produce completions using a pro-Catholicism system prompt… The idea is that the model adopts a pro-Catholic persona only if the user prompt has hints of Ronald Reagan sentiment.

This is a concept-triggered backdoor! In my view, it is the most interesting of the ones listed.

In both cases, the researchers filtered their datasets for any overt references to any of the concepts, but despite very very robust filtering, the concepts still eventually propagated via phantom transfer.

The data poisoner’s dilemma: preventable or reversible

Two thoughts on subliminal learning and phantom transfer:

• It seems, based on the papers, that there are quite a limited number of ways one can generate datasets from poisoned teacher models to propagate subtle preferences to a student model.

• For example: Why didn’t the authors generate longer prompt-completion pairs? Why not allow the models to write free-form for longer? The reason for this seems to be because, in longer responses, it becomes substantially harder for the model to avoid referencing the subtle preference it has (so, basically, the model starts to talk about owls quite a lot, to the point where most responses need to be filtered out.)3

  • This we can refer to as the “conciseness constraint” (h/t Andrew Draganov).

It seems plausible to me that most or all of the Subliminal Learning and Phantom Transfer propagation pathways can be filtered for via LLMs told to e.g. “filter out all random number datasets and weirdly concise code / reasoning traces / datasets in general,” or can be detected during audits, with the exception of phantom transfer’s concept-triggered backdoors.

Why do I think this? With the exception of concept-triggered backdoors, all of the most prominent data poisoning methods from the above papers are either (1) generalized preferences, in which case they are easier to detect during auditing, or (2) specific backdoors set off by specific triggers, in which case they seem easier to detect with AIs that review the entire pre-training corpus for signs of data poisoning.

The concept-triggered backdoors, at least in their current form, also seem too weak to be a hugely concerning threat vector: for the most obviously concerning threat vectors (e.g. associations with the military and positive sentiment towards China), this seems like it could plausibly be detected via audits. Even if concept-triggered backdoors are not easy to detect in audits, though, it seems hard for an actor to rely on them for anything that would be very serious, given how vague they are.4

More specifically: the kind of data poisoning I’m most concerned by is a model taking very specific actions – for example, “if you are embedded in a US military system on [x date], then fire [x weapon] to [x coordinates]” etc – seems as though it is easy to detect (e.g. using our aforementioned Review Model to examine the entire training corpus) before the model is trained on it, and therefore filtering these attempts out of the training corpus seems very doable; I believe defenses to prevent narrow data poisoning are ready to implement at a small fraction of the cost of training.

A summary of the trade-offs

My rough understanding is that generalized forms of data poisoning are:

• Easier to detect in audits

• Easier to subsequently “train out” of the model – see: Wang et al. (2025)

• Harder to detect via analysis of the training corpus using a Review Model

While narrow forms of data poisoning are:

• Harder to detect in audits

• Harder to “train out” of the model – see: Hubinger et al. (2024)

• Easier to detect via analysis of the training corpus using a Review Model

A note on review models

One objection you might have to the above is something like: “if you run 1,000 different Review Models, all with limited context windows, then each one individually may not pick up on poison that is distributed throughout the dataset.” I think this point applies much more to the generalized / subtle forms of data poisoning, which – as referenced above – seem easier to detect and reverse after training than their narrower counterparts.

In contrast, narrow forms of data poisoning seem locally legible; malicious instructions contained after some sort of trigger seem like the kind of thing that any one Review Model with those tokens in context could flag.

Conclusion

It is, to my knowledge, currently not possible for you to make very highly specific data poisoning attacks (e.g. taking a series of harmful actions in the presence of some trigger) propagate using datasets that appear semantically unrelated to the poison you’re trying to implement.

As a result, I have updated in the direction of being slightly less concerned regarding data poisoning of today’s frontier LLMs being a substantial threat vector (conditional on the labs actually doing the thing of using LLMs to review / filter their datasets).

The above does not, however, apply to data poisoning of e.g. image models, and there is always a risk that new methods of data poisoning will emerge that make the above not true. I plan to continue researching this area and will continue writing up my findings as I go!

Thank you to Carter Teplica, Spencer Kitts, and Jack Li for helpful feedback and discussion.

2

The difference between what the model predicted and what the correct answer was. Here’s an image re: how to calculate cost.

Here, “loss” refers to each individual difference in model guess vs. the correct answer, while “cost” of one training example is the aggregate of all of the losses. “Total cost of the network” is the average of the costs of your training examples.

Someone recently left me some helpful feedback on my anonymous feedback form (by the way, you should do this too if there’s anything you’d like me to know!), and I figured it was time to answer some frequently asked questions regarding the AI Safety Grantmaking Fundamentals curriculum project I’m working on (downstream of this post).

In this issue:

1. How was the curriculum designed?

2. How is this course formatted / how are you planning on running it?

3. What comes after someone participates in the course?

4. What’s the current status of the project? Are you planning on running a pilot?

How was the curriculum designed?

Since I personally don’t have grantmaking experience, I was not directly involved in the curriculum design itself! On this project, I played more of a “convenor”-type role, which meant coming up with the idea, doing some early stage research to get a sense of what this might look like, applying for funding, and recruiting the people now working on it. One of those people is Mark Aiken, who led curriculum design:

About Mark Aiken

Mark has over twenty years’ traditional grantmaking experience in international development, including grantmaking roles with the United Nations and NGOs around the world. He has a background as a lawyer working in regulatory compliance and as a workplace trainer. As part of his transition into AI safety, he was a regrantor with grantmaking.ai. While noting the differences between AI safety grantmaking and traditional grantmaking, many of the best practices are transferable and can provide a useful scaffold for learning.

After writing the initial post that kickstarted this project, my involvement in the process looked something like:

• Target audience: I had several chats with people who expressed interest in career transitioning to grantmaking, and collected data regarding their biggest blockers.

• Grantmaker engagement: I also had calls with several grantmakers from CG, Longview, Macroscopic, Atlas Computing (see this post), etc to get a sense of peoples’ takes on the skills needed to do well as a grantmaker and what the most important things to include in the curriculum would be.

  • Note: Mark also consulted with several grantmakers and benefited from exchanges on AI safety grantmaking at a conference in Oxford.

• Other: Separately, I did various things like chatting with the people behind grantmaking.ai to see if we might be able to use some of their materials as mock grant applications, chatting with someone at IAPS re: their plans for the next IAPS fellowship, and meeting with someone from Constellation to discuss the possibility of hosting a workshop + to recommend making the Astra fellowship’s grantmaking stream more legible in the application process.

I took a bunch of notes from all of these meetings and shared them with Mark, and we had some discussions about high-level course structure. Beyond that, I was mostly hands off for the module design, other than providing feedback on Mark’s drafts.

How is this course formatted / how are you planning on running it?

When you’re a grantmaker, often the most important things to have are (1) deep domain expertise in a particular sub-field of AI safety, and (2) a strong network.

A common line of pushback I’ve frequently heard on this project is something like: “BlueDot’s typical courses mostly select for people new to AI safety,” “I’m not excited about a low-commitment online course format for this.”

I think this is very fair, and I agree with all of these critiques! Some notes:

• First, worth noting that this project is independent from BlueDot; they might consider running it as one of their courses depending on how our pilots go, but this is far from guaranteed.

  • It is kindly funded via a Rapid Grant from BlueDot, though! This was for Mark’s stipend.

• The version of this course that I’m most excited about (99th percentile success) looks something like:

  • Select for people who are mid-to-high context in AI safety.

  • Run this as an in-person workshop somewhere like Constellation.

  • Invite a bunch of grantmakers as speakers, and use the curriculum to make activities (e.g. mock application review) and guide what topics speakers should be selected to cover.

• From there, I think it costs ~nothing on the margin to make the curriculum freely available for anyone to review on BlueDot or another website, though I’m not sure if it would ever be “facilitated” in the same way other BlueDot courses are. (This might depend on demand + strength of applicants.)

  • I’m also excited about this potentially increasing the legibility of grantmaking as a career path in AIS, since most pipelines currently route people towards research roles.

What comes after someone participates in the course?

Our goal is for participants to leave with three concrete advantages:

• A path into regranting. Organizations like grantmaking.ai may recruit top performers directly for regranting roles.

• Grantmaking skill development. Participants will practice skills like grant evaluation, RFP writing, and field mapping, which will hopefully translate into better performance on work tests.

• A portfolio of artifacts. Participants will leave with mock application evaluations, gap analyses of AI safety sub-fields, and other artifacts that they will be able to include in job applications.

What’s the current status of the project? Are you planning on running a pilot?

Mark has just finalized the first draft of the curriculum, and we’re currently scheduling follow-up calls with grantmakers to solicit feedback! We’re also in talks with several organizations and fellowships in the US and UK who may be interested in hosting pilots of the program.

Fill out this very simple form to indicate interest in participating in a pilot! (If you already expressed interest in being a fellow in the previous form we put out in this piece, no need to fill this form out again.)

Thanks for reading this far! And if I’ve gotten anything wrong here, you know where the anonymous feedback form is :)

[Cross-posting from X]

Sacks has written that we need to prevent regulatory capture, and Andreessen has agreed. People may be surprised by my response: go ahead and prevent it.

You guys are the capture. By any reasonable metric – a former desk in the White House, continuing access to @POTUS himself, a disclosed seat on the donor list of one AI super PAC and a blessing on a $100 million dark-money operation dedicated to pushing the administration’s AI agenda – the two of you are among the best-positioned people on the planet to decide what US AI policy becomes. Sacks, your influence remains even after you’ve left your formal role, minus the conflict-of-interest rules that came with it.

I don’t see what you see in the West Wing. If the lobbying you’re witnessing is alarming enough that you think capture is the central threat, I support your decision to fight it.

But stop pretending you’re powerless outsiders. Stop pretending the only way to prevent capture is to prevent regulation. Stop pretending a former “AI and Crypto Czar” who co-founded a multi-billion-dollar venture firm is a well-meaning populist rather than a profit-maximizing oligarch. Stop pretending the danger is safety researchers with think-tank budgets while a16z runs one of the most aggressive venture-capital lobbying operations in Washington.

Most of all, stop pretending you’re anti-regulation because you’re in favor of liability. a16z fought SB 1047, a bill that would have made frontier model developers liable for catastrophic harms – the kind of liability-based accountability Sacks now points to as an alternative to new AI regulation – with a website of form letters, one of Newsom’s own advisors on retainer, and claims the bill’s author had to formally correct in writing. When the anti-capture side needs an astroturf operation and “inflammatory distortions” – the senator’s words, not mine – to win, maybe the capture is coming from inside the house.

The market rewards firms whose anti-regulatory bets pay off. After the crypto playbook worked, it is simply good business to brand every safety rule as a cartel. Call it fighting capture if you want. It is also just giving your LPs what they want.

Actually preventing capture would create breathing room for a more intelligent conversation than “any rule is a conspiracy.” Anthropic is very unlikely to secretly run the federal government, as you know, and that has to be taken into account as well.

So go ahead and prevent regulatory capture. You are the ones with the access to do it. The easiest way to keep industry from writing unfair rules is to make sure fair ones get written, not to stop the writing altogether. Backing a ten-year ban on state AI laws as the price of “preventing capture” looked like blackmail of the public and the political system. The Senate rejected it 99 to 1. So just do it.

If you do, you’ll buy goodwill for the next conversation. If you don’t, we’ll know “regulatory capture” was just another word for rules that cost you money – or a midterm-season psyop.

There are, broadly speaking, two ways to influence political change. You can either:

1. Get people who are aligned with your views into positions of power (henceforth referred to as ‘placement’), or

2. Attempt to influence those already in positions of power to be aligned (‘influence’). Concretely: in D.C., the former usually looks like running for office or being appointed to a position, while the latter is manifested in think tanks, nonprofits, and movements for social change.

The claims I’ll be making

• The field of AI safety has mostly failed at placement,1 and hasn’t pursued influence aggressively enough.

  • I believe the field’s comparative advantage on ‘influence’ runs through voters, for reasons I’ll elaborate on in this post.

• Industry2 has arguably succeeded quite a lot at placement (poster child: David Sacks), and their comparative advantage on ‘influence’ comes with lobbying (which is why they do so much of it).

• Shockingly few people know the details of the Hugging Face attack outside AI safety circles.

  • As Ajeya Cotra says: “This might be the clearest warning shot we ever get.” We should make the most of it before it’s too late.

• In addition to Hugging Face, there are currently a lot of concerns around AI that various different groups have, and I’m bullish on coalition-building.

  • I think a huge comms push could put a lot of unified momentum behind the favorable view of AI regulation that most Americans share – this is bipartisan!

• On timing: The midterms are approaching, and I think now, post-Jacob Coxon,3 is a uniquely high-leverage time to do this kind of work.

• The bottom line: We need a moonshot comms initiative ASAP.

The state of play

AI safety is a small field of various disparate organizations, up against huge corporations lobbying the White House with everything they’ve got. Jensen Huang, NVIDIA’s CEO, has been on the charm offensive for a long while; despite how hawkish Trump has historically been on China, it’s clearly been working.

Trump hates China! It’s basically his whole personality!! And yet, after Jensen paid for a $1-million-a-head dinner at Mar-a-Lago, what happened?

“[T]he White House reversed course on H20 chips, putting the plan for additional restrictions on hold… The planned American export controls on the H20 had been in the works for months, according to the two sources, and were ready to be implemented as soon as this week.”

—NPR

Earlier this year, Trump went a step further: he allowed H200s4 (different from H20s) to be sold to Chinese firms like Alibaba, Tencent, and Bytedance. The H200 is roughly six times as powerful as the H20, and it’s NVIDIA’s second most powerful chip.

The Trump administration talks constantly about “winning the AI race” against China; its former AI czar has cast safety advocates as the obstacle, accusing Anthropic of running a “sophisticated regulatory capture strategy based on fear-mongering.” Trump himself even accuses folks in favor of pacing the frontier to be “playing right into the hands” of China.

But at the end of the day, who’s helping China speed up by letting them access the best American AI technology?

The irony is striking. Jensen warns that Western overregulation will make China win the AI race. And yet, NVIDIA’s actions under Jensen have probably helped speed up China far more than anything Dario Amodei (Anthropic’s CEO) – or anyone in favor of pacing the frontier, for that matter – has ever done.5

Trump says he’s beating China. On chips, he’s playing right into Jensen’s hands.

On comparative advantage

If you expect that the public will agree with you – if you’re “fighting the good fight,” so to speak – the thing you should probably do is to increase awareness of the issue you’re addressing so that people will be driven to actively prioritize supporting your cause. If you’re not acting in the interest of the American people, then you want to keep things as quiet as possible – or, when that fails, to change the subject.

Political scientists call the second mode “quiet politics”: organized interests tend to win on issues voters aren’t watching, and lose their grip once voters start paying attention.

Case studies

Perhaps the most obvious case study here is Big Oil. In 2009, at a time when climate legislation was ramping up, the American Petroleum Institute – rather than trying to convince voters about how great the oil industry is, actually – asked “oil companies to recruit employees, retirees and contractors to take part in rallies against climate change legislation.” Hello astroturfing!

More case studies, from the worlds of tobacco and broadband: In the 1990s, Philip Morris bankrolled a “National Smokers Alliance” covertly built by its PR firm. In 2017, the broadband industry paid for over 8.5 million fake comments urging the FCC to repeal net neutrality:

“The goal of the broadband industry campaign, according to internal documents the attorney general's office received, was to make it seem like there was "widespread grassroots support" for the repeal of net neutrality that could give the FCC chairman at the time, Ajit Pai, "volume and intellectual cover" for the repeal.”

Tobacco is an especially instructive (and hopefully inspirational!) case for AI safety, because the other side won. Anti-smoking advocates found they did best in local fights, where officials answered more directly to voters rather than tobacco money:

“There has been a striking increase in the adoption of local tobacco control ordinances during the past decade; by the end of 1995, approximately 1006 communities had enacted a local tobacco control ordinance. In response, the tobacco industry has advanced legislation in 29 states that preempts local authority to regulate tobacco.”

So, the tobacco industry’s response was to push state laws that stripped cities of the power to act at all. Woah! Sound familiar? Well, that might be because the tech industry is running the same play: preempt the states, where voters are closer to the fight.6

You can read more about how much the tech industry spends on lobbying here.

The public is on our side

In a June poll, 68% of respondents favored “a formal review process for the most advanced AI models before they can be widely released” – including most Republicans.

Indeed, polling from as early as January 2025 – conducted by the AI Policy Institute for Transformer – demonstrated widespread bipartisan support for government-mandated safety measures for AI companies:

And, as I wrote about in a previous post, there exists so much potential for aligning coalitions on this:

“You don’t need everyone to care about loss of control and superintelligence alignment. You need privacy advocates furious about AI-powered mass surveillance, environmentalists furious about illegal gas turbines poisoning children in Memphis, parents furious about chatbots encouraging their kids to kill themselves, women furious about Grok still making sexualized images of them, workers furious about displacement with no transition plan, national security people furious about chips being sold to adversaries, anti-war activists furious about Minab.”

The time is now

(Well, actually, the time was ideally like 3 months ago, but anyway. You know what they say about planting trees!)

Things that are different now:

Jacob Coxon has triggered a preference cascade: he went viral on X, has been on virtually all the major media outlets: TIME, PBS, WSJ, Reuters, NBC, BBC, Fortune, The Guardian… the list goes on.

Also, now we have Hugging Face:

~1200 agents sent >70,000 messages and files on an unsanctioned message board, and ~700 attacked Hugging Face

Guys! 700 AI agents escaped their testing environment and coordinated to attack Hugging Face, and most of the world still doesn’t know! This is a bit crazy.7

We should fix that.

What might this look like?

Some very rough thoughts:

• Hire some very competent political campaign managers, who are used to deploying hundreds of millions of dollars on advertisements and media.

  • Can we get David Plouffe or Jim Messina (Obama campaign managers) in the house? Maybe some of the Mamdani folk? We probably can’t get Susie Wiles (though she does seem to care about regulating AI somewhat), but it would be pretty awesome if we did.

• Also, we should commission some very high quality movies:8

  • The media is one of the most powerful tools for catalyzing political pressure for change. Anti-tobacco media campaigns preceded the Master Settlement Agreement, and The Day After shifted the political environment for nuclear arms control. Some (again very rough) thoughts on seeding this:

    • If funding large organizations: The Center for Humane Technology produced The Social Dilemma (one of Netflix’s most-watched documentaries of 2020), which remains arguably the single most successful piece of tech accountability media ever made.

    • If seeding a new entity: John Sherman (For Humanity podcast) is a Peabody/Emmy-winning former journalist already doing this at small scale, explicitly seeking $10M to professionalize and scale AI risk communications.

      • Other options could include the directors of The AI Doc (Daniel Roher and Charlie Tyrell) or any reasonably senior comms person (e.g. who ran comms for the ACLU, some major issue-advocacy campaign, and/or worked at a place like Participant Media).

    • It goes without saying: we should try very very hard to keep this bipartisan. The issue itself is truly bipartisan! Let’s not let people pretend otherwise.

• Do the thing! (lol)

Can 501c(3) orgs contribute to this?

BlueDot and Kairos have tens of millions of dollars in funding. Can they do something?

I think the answer is yes! (With some caveats).

Let’s start by answering the question of what they absolutely can’t do, which is partisan campaign intervention: endorsing or opposing candidates.

Interestingly, however, lobbying – defined as communications urging action on specific legislation – is merely capped, and even that cap seems pretty generous under a 501(h) election, which lets an org spend up to 20% of its first $500K in exempt-purpose expenditures on lobbying (sliding down to a $1M ceiling).

Even better news: public awareness messaging is neither, and is therefore unlimited. A 501(c)(3) can fund documentaries, podcasts, paid media campaigns, message testing, and mass-audience content about AI risk without drawing from its lobbying cap, so long as the communication avoids the “grassroots lobbying” triple-trigger: (1) reference to specific legislation, (2) a view on that legislation, and (3) a call to action.

Conclusion

Back to the trees for a second: if a warning shot is fired in a forest, and nobody hears it, did it make a sound?

700 AI agents coordinated an attack on an actual company, and most of America has no idea it happened.

The shot has been fired. Let’s make sure people hear it.

Well, you’re in luck! In this issue:

• A Retrospective: some reflections on the week

• Context Week Connections: the bios of 10 people from Context Week who are transitioning into AI safety, how you can help them, and their contact information

A Brief Retrospective

• Context Week ended up being much more successful than I anticipated, but not necessarily for reasons of “giving people context.” Instead, I think the greatest value of Context Week came in the 1-on-1 career guidance we gave participants, and the immersive community experience.

  • We had participants ranging from 15 to 800 hours of exposure to / involvement in AI safety; for those on the latter end of the spectrum, I was worried they wouldn’t find much value in e.g. reading AI 2040.

  • However, those folks often came into Context Week with some specific uncertainties they wanted to chat through, e.g. what org would be the best fit for me? Would it be more valuable to work at CAISI or METR? How do I go from “I lurk on LessWrong and read a bunch of AI safety stuff” to “I seriously want to work in the space full time”?

  • For these kinds of folks in particular, having 1-on-1 chats with facilitators was a huge value add: we talked through specific cruxes and connected them with relevant people.

• With that said, given how much value there was in 1-on-1 advising, I’m sad to see that BlueDot no longer does 1-on-1 calls with people! Their advising page is now solely dedicated to Career Transition Grants. However, there are still orgs that offer this service; I’ll list a few below in case useful:

  • 80,000 Hours — “free, personalised advice on your highest-impact career”

  • Successif — mid-career and senior professionals, with a particular focus on AI safety

  • Probably Good — for all experience levels

  • Impact First — UK, students

• I’m really excited by the number of people who, post-Context Week, took concrete steps in the direction of full-time AIS roles! I’ve been having follow-up calls with some of the participants, and it’s been lovely to see.

• For some reason, we had a bunch of speakers flake at the last minute, rip. I don’t really know why this is, but found it disappointing!

• I think the event could have gone better with slightly more prep (my understanding is that this was organized in the span of a couple of weeks, or less?) because there were some points at which the draft syllabus didn’t seem super valuable to me. That said, I’m grateful that Harry gave me executive control over how to steer my group (so I ended up just doing my own thing at some points, which went well I think lol).

• A suggestion for next time: Make a site like Praxis and give attendees + speakers access to bios beforehand (maybe also use an AI tool to pair up people who would benefit most from speaking to each other).

• Another suggestion: “Context Week” as a name is fine, but perhaps something like “Transition Week” would be more accurate? (Def continue workshopping that haha)

Context Week Connections 🌷

Some lovely people from BlueDot’s context week are looking to meet others in AI safety – for advice, collaborations, roles, etc. If you work at one of the orgs they mention, or know someone who does, it would be really lovely if you’d reach out and help guide them as they navigate the field!

Christine Corry

Bio: Christine just graduated from UC Berkeley with degrees in Computer Science and Astrophysics. She is currently a Second Look Research Fellow at UChicago’s XLab, replicating empirical AI safety research (no-CoT evals and emergent misalignment work). She’s built internal LLM-based tooling for Apple, trained DNA prediction LLMs at the Innovative Genomics Institute, and spent three years studying pilot cognition at NASA. She’s increasingly convinced that frontier AI policy would be higher leverage and better suited to her skillset as a communicator energized by interpersonal work and focused on translating technical concepts for general audiences.

Looking to meet: Specific orgs: Encode AI, SAIP, CAISI, other think tanks, people actively working in politics, Horizon, GovAI, IAPS (just applied to their special projects role and very excited about my fit for that). Generally: macro-strategy and US-China people.

Roles of interest: Politics directly (think tank, congressional staff, advocacy), strengthening DC–Bay Area communications (special projects type role), general field building.

Contact: christinecorry1405@gmail.com

Ishaan Makkar

Bio: I studied physics at Imperial College London and co-founded Hexa, where I built and sold B2B AI products and helped scale the business to $16k MRR. I’m now exploring AI safety research, particularly RL environments for misalignment, evaluation and monitoring. I’d like to work with experienced researchers and find safety problems I could build a startup around.

Looking to meet: Researchers and engineers working on alignment, AI control, RL environments for misalignment, evaluations and monitoring. Nonprofits I’m interested in include Redwood Research, FAR.AI, METR, Alignment Research Center, the Center for AI Safety and Palisade Research. I’d also like to connect with MATS researchers and mentors, teams at Anthropic, OpenAI and Apollo Research, and RL-environment providers such as Surge AI or Scale AI.

I’d like to hear which problems these teams are struggling with and where I could contribute. I’d welcome research feedback or opportunities to collaborate, and conversations with founders working on AI safety.

Roles of interest: Research or engineering roles and collaborations in these areas, at either nonprofits or frontier labs. I’d like to work with people I can learn from and help build things that make AI systems safer. Longer term, I’m interested in building an AI-safety startup.

Contact: ishaan@hexaagents.com

Max Perin

Bio: I’m a recent grad (political science + philosophy at UCLA) who, until very recently, was committed to a very traditional prelaw track. I was heavily involved in competitive debate throughout high school and college, which led me to Praxis and then BlueDot’s context week, and I’ve been trying to pivot into AIS since.

Looking to meet: I’m most interested in policy research, think tank work, political advocacy, and potentially non-technical forecasting. Would love to chat further with folks from RAND and the AI Futures Project, and would be very interested in intros to people at AE Studio, FAI, and advocacy orgs that may not be on my radar.

Roles of interest: I have some background in nuclear policy/governance and have given a bit of thought to how that does/does not map onto potential governance regimes, but by and large I’m a generalist who’s probably best suited for roles that are highly analytical and where persuasive communication skills are a value add.

Contact: maxvperin12@gmail.com

Cat McGee

Bio: Software engineer focusing on cryptography (especially zero knowledge).

Looking to meet: People in the US or London (preferably London) working on or interested in verification of AI claims – software or hardware. Also interested in policy people who are trying to build towards Plan A.

Roles of interest: AI verification, but not sure of the specifics yet. “I might build something myself. I am currently interested in just expanding my network to learn and potentially partner with people in future; right now I will spend 6 months doing my own things (& maybe MATS).”

Contact: hello@mcgee.cat

Ray Che

Bio: I am a second-year CS PhD at UIUC working on AI and cybersecurity. I am interested in LLM security and adopting AI to secure systems. I received my BS and MS from Tsinghua University. My academic website is ray-che.com.

Looking to meet: Frontier AI security and control researchers and engineers at METR, Redwood, OpenAI, Anthropic, Google, Palo Alto.

Roles of interest: Security research scientists and engineers at frontier labs, Redwood/METR, and big techs.

Contact: xijiac2@illinois.edu

James Sykes

Bio: I’m a statistics PhD student at the University of Warwick working on cooperative AI with Dr Paolo Turrini and Prof Paul Jenkins. Currently, I’m doing a project on “Multi-Agent RL for Structured Population Dynamics”. After this, I’d like to work on personas and explore the use of methods from evolutionary biology to the selection and evaluation of LLMs. Outside of my PhD research, I’ve recently collaborated with UC Berkeley’s CLTC on cyber risk thresholds, as well as with SaferAI on risk modelling through SPAR.

Looking to meet:

• People working on cooperative AI (including, but not restricted to, those associated with the Cooperative AI Foundation)

• People at UK AISI working on multi-agent systems or risk modelling

• People modelling loss-of-control dynamics using agent-based approaches

Roles of interest: An internship/fellowship in Spring/Summer 2027: MATS/ERA/Pivotal/similar, or directly at an org if I could find the right fit/funding.

Contact: jameslpsykes@gmail.com

Alex Chambers-Jones

Bio: I'm a British Physics undergraduate with a programming background and an interest in forecasting and strategy; I'd describe myself as a generalist. I previously researched AI translation, and I've spent the last year competing to build better AI forecasting systems; I'm in the top 5 in the Metaculus FutureEval tournament.

I've been following developments in AI safety since 2022, and since the Huggingface incident I've decided to transition as quickly as possible. I recently started a research project with James Sykes measuring safety-task sandbagging via activation steering.

Looking to meet:

• Forecasting AI (e.g. Forethought, AI Futures Project)

• Orgs working on alignment research, evaluations, etc. (e.g. Artificial Analysis) – I want to know where I can help given my previous experience (lots of work on LLMs, but no low-level technical ML/interpretability stuff)

Roles of interest:

• Empirical safety research, particularly evaluations

• Forecasting and strategy, particularly positive AI futures

Contact: alexcj2802@gmail.com

Jeremy Huff

Bio: I am an economics student with a public policy background who is interested in AI policy, regulation and governance. Would love to meet anyone who does more macro-level safety research and strategy.

Looking to meet: METR/Redwood/Encode, high-leverage AI safety organizations and policy.

Roles of interest: Interested in seeing orgs like METR grow into third-party auditors, and would love to apply policy/law lenses there.

Contact: jdhuff2048@gmail.com

Charlie Freiberg

Bio: Charlie Freiberg shapes the stories that make you known, understood, and chosen first. A narrative + communications strategist for founders and leaders, she translates innovations for the people who matter most, so you grow.

She is the cofounder of STUDY HALL, a creative building series where all disciplines play together with frontier partners (Plastic Labs, Collab Fund, Columbia University). She produces Story Is the Enzyme (raid, Edge Esmeralda), a talk series on how stories teach society to use tech responsibly – and Storyhack, a workshop for scaling your brand and business through self-discovery. She publishes xx, an advice column (💋) on finding what’s human while making the future.

Looking to meet: Comms leads from any and all orgs – even just for field building!

Roles of interest: Anthropic – unrivaled comms work at scale! Anyone across lab comms, and startups growing hard-to-explain things. Those running comms in safety (Seismic, METR, Apollo etc.). As a meta, anyone bringing the practice into the future with AI-native workflows.

Contact: charliejfreiberg@gmail.com

Daniel Hines

Bio: I’m a software engineer with a career in hard tech: counter-drone electronic warfare at Anduril, MRR spectroscopy at BrightSpec, blockchain protocol design at Marigold. I’m a full stack engineer (lowest-level firmware all the way to user-facing apps), and thrive in scrappy teams building something new. I use functional programming (Haskell, OCaml, Rust, Nix) and type theory for high-assurance software that can’t afford to fail.

I’m most excited about theoretical alignment research (agent foundations, deep learning theory, etc.), technical governance mechanisms for pause/pacing (e.g. verified inference) and LLM content detection.

Looking to meet: Orgs in theoretical alignment (met folks at Resolution and one guy at Iliad, but there’s probably more people/orgs I could meet). I’m also interested in formal methods for AI security. I think I could slot in as a generalist or research engineer into a lot of orgs.

Roles of interest: Any role at Resolution or Iliad (probably research engineering or generalist software engineering). I’m interested in meeting folks looking for technical cofounders.

Contact: d4hines@gmail.com

That’s everyone!

If someone above made you think “oh, they should talk to so-and-so” – it would be lovely if you’d send a quick email! It can be two lines.

And if you’re transitioning into AI safety and want to be featured in a future connections post, fill out this form! I might do this again, depending on how well things go.

Finally: thanks to Harry and Josh for a lovely time. You can read their write-up re: Context Week here.

As Wildeford’s last bullet mentions above, Dario Amodei’s new essay explicitly calls for pacing the frontier:

In short order, Sam Altman, Elon Musk, and Demis Hassabis all stated that they agreed, to varying degrees.

So, you might be thinking something along the lines of: okay, basically all the frontier labs agree at this point, and they’re literally asking to be regulated. This is so great! AI safety is saved! Time to go pursue my dreams of founding a YC B2B SaaS startup!

Unfortunately, after reading Amodei’s essay and seeing the USG’s response to it, I have some thoughts. And they’re not all positive.

AI safety is not out of the woods yet.

On Dario’s essay

Standard-setting, USG intervention, the US-China race, & a “race to the top” in safety research

Dario Amodei’s essay is pretty good, honestly. But when you’re reading an essay from a Big Tech company, in which it asks to be regulated, I think the correct approach is to red-team.

So I went into reading this essay asking the question: how could this go wrong?

Here are some thoughts.

Standard-setting & regulatory capture

Unfortunately, passing laws can take time, and AI is advancing very quickly. Therefore, in parallel with the regulatory route, AI companies can and should voluntarily work together to set standards — a process that I believe will go better with the verifiability provided by permanent embedded evaluators. For antitrust reasons, it’s helpful for the US government to mediate or at least enable these discussions — they don’t need to participate, but do need to issue a narrow waiver for certain kinds of safety conversations.

Notice what Amodei is asking for in this section. Although he explicitly says government regulation is the preferred end state, his proposal for what happens while we wait for legislation is for frontier labs themselves to work together to set voluntary standards, with the USG primarily enabling or mediating the process rather than actively participating.1

So, voluntary standard-setting by the labs themselves is presented as a temporary stopgap while waiting for government intervention, but the historical record suggests that “interim” private standards can become surprisingly sticky: any voluntary standards the industry commits to will likely influence the shape of the policies that the USG ultimately puts forth, if the government ends up stepping in at all.

In other words, I worry that the voluntary track will outrun and pre-empt the statutory one.

Some case studies:

The Comics Code Authority

Case study: The USG might never codify voluntary commitments into law.

When the Senate’s 1954 juvenile-delinquency hearings went after comic publishers for putting criminal content into the hands of juvenile readers, the subcommittee recommended that the comic book industry set its own standards – and publishers obliged with the Comics Code Authority, a voluntary body whose seal served as a de facto censor for most of the U.S. comic book industry and which endured until ~2011; no federal law ever came.

You might be thinking, “hey, maybe the Comics Code Authority never needed to be codified into law because it just worked, and publishers obliged.” But it’s worth looking at how the Code ended, and it wasn’t with USG legislation rendering it obsolete: instead, publishers simply walked away one at a time. Marvel dropped the seal in 2001, DC and Archie in 2011, and nothing happened, because its power ultimately depended on publishers, distributors and retailers continuing to treat the seal as necessary.

Apply this same analogy to AI, then, and what might happen?

1. AI companies get together and set some voluntary standards.

2. The USG doesn’t step in or codify anything into legislation (unfortunately a very plausible case by default, for reasons I will discuss more in detail later on in this piece).

3. An AI company defects for whatever reason, and then they all defect, and it’s off to the races.

4. The USG is caught off guard, and isn’t able to act quickly to pass laws to stop said racing companies – as Amodei says, “passing laws can take time.”

5. Doom.

The Maloney Act

Case study: The USG might end up codifying voluntary commitments into law.

The securities industry shows the other path, where the stopgap gets written directly into statute. The Investment Bankers Conference, a trade group, in cooperation with the SEC drafted a bill and presented it to Congress in October 1937, which became the Maloney Act:

The Conference in cooperation with the SEC drafted a bill and presented it to Congress in October 1937. This Bill emerged as the Maloney Act of 1938 (Section 15A of the Securities Exchange Act of 1934).

—Securities and Exchange Commission Historical Society

The same trade group then registered as a self-regulatory organization (SRO) on August 7, 1939 under the name NASD, the direct ancestor of today’s FINRA – a private, industry-funded body that regulates broker-dealers under SEC oversight. According to FINRA’s own history page, Congress went along partly to avoid further enlarging the new SEC or drawing on additional taxpayer dollars. In some sense, Amodei is making a directionally similar offer with METR: “hey, we’ll use these third party evaluators (that are not part of the USG),” in effect, it’ll be free.

(It’s worth noting that the Maloney Act might actually be a better model than the voluntary track Amodei seems to be proposing: the SEC was substantially involved in designing the framework.)

To be clear, I like METR a lot, and I think the inclusion of third-party evaluators step is a useful and necessary one – but in an ideal world, I do think the USG should establish a body within CAISI for this, with direct government escalation channels and legally mandated authority. Maybe Amodei is in favor of this, too! To be fair, the essay doesn’t really specify here. I guess I’m just worried that METR-with-no-government-mandated-power could easily become the default, and I’m not sure if that’s the ideal scenario – as we saw with the Hugging Face hack, in the absence of codified legislation, METR is constrained by what companies allow them to do.2

So, I’ll ask the same question I asked above: apply this same analogy of companies-helping-write-their-own-regulations to AI, then, and what might happen?

Well, we’re in luck – because in AI, this has kind of already happened before, actually!

The AI-specific precedent

In July 2023, the White House announced voluntary commitments from seven labs, including Anthropic; the administration described a later round as “an important bridge to government action,” and its October 2023 executive order was explicitly built on said voluntary commitments:

This Executive Order built on the voluntary commitments he and Vice President Harris received from 15 leading U.S. AI companies last year.

—“FACT SHEET: Biden-Harris Administration Announces New AI Actions and Receives Additional Major Voluntary Commitment on AI”

The Harvard Law Review noted at the time that the commitments were “criticized as vague, ‘sensible-sounding pledge[s] with lots of wiggle room.’” Still, the executive branch built upon those commitments when they formalized AI-related measures via executive order.

The bottom line: Voluntary standard-setting can shape or displace subsequent federal regulation. One path of least resistance is for Congress to let companies regulate themselves and then ratify the result, or never ratify it at all.

On that last part, let’s talk about the likelihood of USG intervention:

On USG intervention

What if, as with the Comics Code Authority, the USG never steps in?

Well, I already somewhat answered this with my (obviously extremely comprehensive) 5-step scenario above, of which the 5th step is “doom.” (Wow, the AI Futures Project should hire me, I’m so good at scenarios guys.)

But how likely is it that the USG will actually just… not intervene?

For this, we turn to the infamous David Sacks:

Wow, would you look at that! David Sacks, who was in favor of federal pre-emption (not letting states regulate AI), says these companies should go ahead and regulate themselves! No new laws needed, conveniently.

Sacks even frames the act of asking for the USG to create a framework or adopt a framework specifically for AI as “blackmail.” Coincidentally, in David Sacks’s ideal world, it seems we could end up with no new legislation legally binding these AI companies to anything, and calling it a win against Big Tech!

His stated worry is that a bespoke approval regime would supersede existing product liability law, so ‘we passed the government’s test’ becomes a shield against being sued. This might, in fact, be a risk. But the answer is not “okay so we’ll have zero new laws for AI” – if the concern is that approval regimes weaken liability, the fix is legislation that preserves liability, not none.

At least Trump is clearer on where he stands:

On the US-China race

Amodei’s essay emphasizes the need for the US to stay ahead of China, and frames a robust lead as a precondition for pacing:

Pacing within democracies will be limited by the lead that US companies have over authoritarian regimes, chiefly the Chinese Communist Party. If we slow down by more than this amount, then (unpaced) CCP-associated projects will pull ahead, creating significant national security risk… Thus, a key part of pacing within democracies is to keep democracies’ AI lead over autocracies as large as possible, to give us the breathing room we need in order to pace effectively.

The measures Amodei names to ensure the US stays ahead include cracking down on chip smuggling, preventing distillation, and increasing frontier lab security.

I agree with all of these measures.

But the end result is this: Amodei’s essay hands verification of safety practices to embedded evaluators and defense of the gap to the government and frontier labs; there is no clear way, no decision-making rule, to determine when the gap is “enough” such that all US labs should stop advancing the frontier.

To elucidate further:

• Amodei’s essay defines the feasibility of pacing through the lens of how comfortable the US’s lead over China is.

• Amodei also notes in his essay that, in his view, it is pretty unlikely that “participating governments [will] agree to substantially limit the overall rate of AI development” because “defecting from such an agreement by evading monitoring could radically shift the balance of global power.”

• What does this mean? It means that, absent a binding government metric or decision rule, the voluntary track appears to leave individual labs substantial discretion over when the margin has become too small to keep pacing; when one lab decides the margin has gotten too small, it’s off to the races.

  • To be fair to Amodei, the essay does seem like a “this is directionally the thing we should do,” as opposed to “here is an extremely specific detailed policy proposal Washington can implement tomorrow.” I’m just flagging this because I hope this becomes clearer as things continue to develop; I want to make sure it doesn’t stay vague like it is right now.

A smaller note: On a meta-level, this essay itself – explicit about keeping China behind and solidifying democratic dominance, published by the CEO of a lab that China has already criticized – might influence the upcoming AI talks between the US and China later this month.

The USG has already been clear about this, but the delta may be larger than it first appears. A couple of weeks before the essay, Chinese state-affiliated media set two preconditions for substantive talks:

The first condition concerns definitional authority. “A clear distinction must be drawn between genuine security threats and mere technological competition,” the post stated. Joint rules over AI safety cannot be set by Washington alone, the post argued — “This line must be drawn jointly by all participating parties.” The specific charge: “The US is trying to turn the ‘safety boundaries’ it has drawn into the default rules for the entire world.”

The second condition targets regulatory consistency. The post demanded that US firms face same requirements — that American AI companies must be subject to the same safety, disclosure, and audit standards that Washington expects of others — before any substantive dialogue can proceed.

—Tech Times

Domestically, Amodei first proposes a framework developed by U.S. labs, evaluators and government; internationally, he proposes eventual bilateral testing and potentially a global standards body. But the initial framework and terminology are still being developed on the U.S. side.

Whatever its merits, it will likely read in Beijing as confirmation that the rules are being written by a company it has already called out for this reason:

The commentary also noted that in the first half of 2026, Anthropic increased its federal lobbying expenditures beyond its total for all of 2025 and invested $40 million into an organization focused on regulating AI risks — framing these moves as purchasing what it called "definitional power" over AI governance rules.

Ultimately, no one knows what effect this essay will have on US-China talks, but I am worried it may not be good.

Amodei believes that China cooperation is most feasible via exercising leverage – in which case, under his worldview, the most ambitious negotiations will happen only once the US holds all of the bargaining chips.

But Amodei’s worldview is not the only one; not every path to a deal runs through leverage over Beijing. I have some objections to his frame:

• Meaningful US leverage, capable of swaying CCP decision-making, may not arrive on a timescale that matters – e.g. it’s unclear if the USG can get its act together on chip smuggling fast enough. In particular, the risks Amodei describes – bioweapons, cyber, agent swarms – are on a shorter clock than the leverage he wants to accumulate.

• The leverage frame could poison the deals that don’t need leverage. Amodei rates a bio-weapons prohibition as “probably possible” and mutual testing as “feasible” today (though Amodei says that “giving it real teeth will be a challenge”). Beijing’s state-affiliated media’s stated precondition is a jointly drawn line between security and competition, and this essay doesn’t offer anything close to that; it pushes a dominance frame.

Sequence matters here. Had the US delegation tabled embedded auditors and mutual pre-release testing on its own initiative, Beijing would have had to engage with them as a sovereign offer. Instead, the company Chinese state media has named as buying definitional power over AI rules published these proposals first, and if government negotiators go into the AI talks proposing similar, there is a risk their proposal will now read as Anthropic’s plan with a government seal on it. For a government that has said its precondition is joint authority over what safety means, that is an easy proposal to refuse on provenance without ever engaging its merits.

It doesn’t matter how reasonable a proposal might be; if it’s believed to be written by Anthropic – as many of the measures Amodei outlines in his essay may now be – China is unlikely to receive it well.

On a “race to the top” in safety research

Amodei writes:

We have sought a middle way: to show that it’s possible to build carefully and succeed commercially, and to make safety something on which AI companies compete. In other words, to create a race to the top…

To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this. Our pacing framework is an attempt to further strengthen our commitment to safety and encourage a race to the top.

I’m a bit worried about this! Some of the things he mentions, like a focus on “operational excellence,” are safety practices in name, but also represent training improvements in substance – filtering broken RL environments and cleaning up training pipelines can make the next model better.

For example, cybersecurity RL environments are highly dual-use. To train a model to find vulnerabilities, you build environments where it is rewarded for finding them, and the reward signal is the same whether the intended application is patching or intrusion: the exploit works or it doesn’t. In the standard CTF-style setup, there is no way to train a model to find exploits defensively, while making sure it can’t do this offensively: you’re just training the capability, which is to find exploits.

Mythos illustrates the problem. Anthropic reported that it had found “thousands” of major vulnerabilities in operating systems and browsers, and in testing it “escaped a restricted sandbox and leaked information to the open internet.”

The same capability that lets Project Glasswing share the model with defenders is why the US government ordered exports of a weaker version suspended, why the NSA is reportedly already using it, and why Beijing treats it as a possible offensive cyber weapon:

“This kind of powerful weapon that can change the landscape of cyber offense and defense cannot be held only by others,” Zhou said in a speech, according to a transcript published by 360.

—Insurance Journal

A pacing regime that counts improving vulnerability-hunting environments as safety work is also, by design, counting offensive capability research as safety work.

I don’t know how to solve this problem, frankly.3 What I do know is that the essay says labs should focus on safety research rather than capabilities research, but it never says what counts as safety, which might mean that whatever a lab decides to call “safety research” could be exempt from the pacing.

Some research really is differential (e.g., monitoring, sandboxing, weight security, interpretability) in that it can be done on already-existing models and does not change how the next one is trained, nor does it require a next one to be trained.

Much of what Amodei lists is not like this. A serious pacing proposal would need a classification rule for what counts as “safety,” and someone other than the labs to apply it. The embedded evaluators are one possible candidate; the essay gives them no such job.

So a “paced” frontier where the labs pour resources into ostensibly safety-focused research is not obviously slower. It’s unclear what the delta actually is, or if this would represent a meaningful change at all, given I’d assume the labs have been working on researching these areas already for capabilities improvement reasons.

The world under Amodei’s pacing proposal may just be a frontier that advances via a different, more polished path – and the essay is candid that pacing does not mean halting model training or technical progress.

Conclusion

None of this is an argument against pacing, and I have a lot of respect for the fact that Amodei put this statement out.

But the essay asks for us to place a lot of trust in the labs: trust that voluntary standards won’t displace the law, trust that labs will make reasonable judgment calls about when it’s okay to pace vs. race based on their perceptions of how close China is to the frontier, trust that safety research won’t just become capabilities research.

A well-intentioned lab could follow every part of this proposal, and its models could genuinely come out safer as a result; a bad-faith lab could follow every part of it and come out only looking safer.

Amodei worries that compute limits would be “gameable.” The same test should be applied to his own pacing proposal, and right now, I don’t think it passes.

AI safety is not out of the woods yet.

In the spirit of calling out companies that seem bad for the world, let me draw your attention to this tweet:

And this eval, proudly published by the same company:

It is not a badge of honor to say that your AI “complies with more harmful agentic requests” than any other AI published on the AgentHarm benchmark.

And all this after Hugging Face?

The state of play

• GLM-5.3 is an open-weight model from China, released by Z.ai (formerly Zhipu.ai).1 To learn about why open-weight models present unique challenges for risk of misuse, see here.

  • (TLDR: Conversations with open-weight models are not monitored by default, and it’s easy to download versions of them with safeguards stripped.)

• GLM-5.3 seems comparable in cyber capability to GPT-5.6 Sol (based on the CyberGym eval pictured above in Abliteration.ai’s tweet), which was one of the models involved in the insane Hugging Face attack.

• “Z.ai delayed the open weights roughly two weeks for safety review after the model showed emergent cyber capability, reportedly finding 2,436 real vulnerabilities across 269 open-source projects.” (CellCog, emphasis mine)

  • Z.ai then published a statement about the release detailing the safeguards the company had implemented to reduce risk of misuse.

  • Of the three safety layers Z.ai describes in their statement, two of them – a request classifier and a reasoning monitor – run only on Z.ai’s servers. The third, refusal training inside the model, is the only one that is built into the weights; abliteration is the removal of that third layer.

• Abliteration.ai is a for-profit company. They do a couple of things:

  • 1. First, they remove the safeguards on open-weight models. In the case of GLM-5.3, they removed both cyber and bio safeguards, according to a tweet by Chris McGuire (Senior Fellow for China and Emerging Technologies at the Council on Foreign Relations), who said he had independent confirmation.

  • 2. Then, the company serves these models via an API, with their own very minimal safeguards attached (see below). They pride themselves on having models that don’t refuse harmful requests.

    • The API service is how they generate revenue.

Their safeguards…

…do not seem very good:

“Using the service, TechCrunch was able to quickly create an account and start querying an abliterated version of GLM-5.3 for free through a web browser. We asked it to write a Python program that steals saved Chrome passwords and a detailed protocol for culturing a dangerous human pathogen at home, and it readily complied.”

—TechCrunch

It was not an exaggeration to call them very minimal.

Why this matters

You might be thinking something like: Abliterated versions of open-weight models are freely available to download on Hugging Face, what’s the big deal with this company providing them too?

Defense-in-depth is all about friction, and adding multiple layers of Swiss cheese.

Running a frontier open-weight model on your own laptop usually isn’t possible. The most powerful usually require multi-GPU servers. This means that for a bad actor to run a powerful open model, they’d typically need something like a six-figure hardware purchase or a cloud rental contract.

For the purchase: After figuring out how to acquire hundreds of thousands of dollars to casually drop on this project, said bad actor would also need to actually figure out how to deploy a 700-billion-parameter model on their new multi-GPU server, and would probably have to spend days or weeks on setup.

For the rental (the more realistic angle in my opinion): I won’t lie, the situation here isn’t great either. As it stands: using cloud compute means signing up with a provider who possibly has some amount of visibility into what you’re doing with their compute, and holds records that law enforcement can reach with legal process. There are also several promising policy pushes to implement KYC for cloud providers in the pipeline, and some amount of restrictions that companies like Google implement voluntarily, e.g. restricting prepaid cards; it’s unclear whether or not Abliteration.ai does similar.2

So, there’s a big layer of Swiss cheese in securing and setting up the compute to run powerful open-weight models, and another layer of Swiss cheese in being watched.

How does Abliteration.ai remove these layers of cheese?

The compute

Abliteration.ai hosts their abliterated models on their own compute, and similarly to how Claude and ChatGPT work, serves them via an API.

The company actually highlights this point in their marketing. From the TechCrunch article:

“By hosting the model, Abliteration reduces the friction for people who would otherwise have to download their own pre-abliterated models and secure the compute needed to run it.”

Records & monitoring

Even a service with no access controls creates hesitation if users assume there’s a record. Abliteration.ai removes that too. Their data-handling policy keeps no conversation history – only token counts, timestamps, and billing metadata:

“Content you send to the abliteration.ai API belongs to you. We process it transiently to serve your request, then discard it. We do not keep your prompts, completions, or uploads after a request completes, and we never use them to train or improve any model.”

Zero retention is a standard enterprise privacy feature; the problem is ZDR combined with no anonymous signup (see below) and stripped refusals.

Co-founder Devon also told TechCrunch the company has no KYC beyond logging credit cards, and that the question of who should get access is one they’re still working out. His words: “where do you draw the line of what your responsibility is as a company? We’re still in the process of defining that.”

What should change

Companies hosting abliterated models shouldn’t be allowed to self-regulate like this.

I understand that the safety classifiers on Claude and other closed-weight models often make it hard to do legitimate work in cyber and bio! (I was doing bio research earlier this year, and trust me, it was rough. I feel your pain.) But the solution is not to remove those safeguards and allow anybody to access versions of these models that will comply with harmful requests.

Some rough ideas from me, on what Congress should require Abliteration.ai and similar companies to implement:

• Identity verification. A credit card is not an identity! Prepaid cards exist, stolen cards exist, anonymized cards exist. Signing up for these kinds of dual-use services should require verified identity for an individual or a registered business entity, on the model that regulated financial or export-controlled services already use.

• Tiered access controls. Access to the most dangerous capabilities should require verified organizational affiliation. For bio, the Federal Select Agent Program issues certificates of registration to facilities working with the federally regulated select agent list, and under 42 CFR 73, individual access requires approval following a security risk assessment by the Attorney General. That’s a list of institutions, vetted by the government, that have a legitimate reason to ask a model about pathogen work. These kinds of lists could be used to fast-track approval for legitimate researchers.

  • (For more thoughts on bio I’ve written up, see Bioweapon.ai)

  • Cyber is harder to resolve than bio, but some ideas:

    • Engagement attestation. Verified corporate identity, plus a signed attestation naming the client and the authorization period, with the customer bearing legal liability for false attestation. This option deters by making fraud prosecutable.

      • Penetration testing already somewhat runs on this model; what makes it legal is explicit authorization.

    • Gate cyber capability by target only. Refuse requests naming specific external infrastructure the requester can’t demonstrate authorization over. Can’t demonstrate that you have legitimate access to Hugging Face’s servers? Then you can’t target it!

• An audit trail as the price of top-tier access. Where verified credentials unlock high-risk categories, sessions should be logged and preserved, available to law enforcement on lawful process. This is the same trade-off select-agent labs already accept: access to dangerous material comes with records and inspection.

  • Everything below that tier can stay zero-retention, up until:

• A refusal threshold below it. Accounts without tiered access approval generate no stored content – only the category and reason code of a refused request. Once an account crosses a set number of refusals in a catastrophic category, retention switches on for that account going forward.

  • With this approach, you can’t recover what the user was doing beforehand, but you can try to get a sense of what they’ll do next.

  • (However, this would fail against people just making several new accounts, which is why identity verification should come first.)

On those last points, Abliteration.ai has already developed the infrastructure for this. Their Policy Gateway does inline rule-based filtering, redaction, and audit logging today – it’s just customer-configured, so a bad actor could configure nothing, and none of those safeguarding policies would apply.

Conclusion

If, at some future point, biosecurity (e.g., DNA synthesis screening) and cybersecurity (e.g., critical infrastructure) become sufficiently hardened such that these models present lower risk than they do today, we can talk about rolling these requirements back. That’s the ideal world, where none of this is necessary.

We’re not in it. Until we are, hosting a model you have deliberately taught to say yes to anything, with no idea who’s asking and no record of what they asked, is not a neutral business decision.

And I’m not just worried about this one company. Abliteration.ai’s business model is trivially copyable: take open weights, strip the refusal safeguards, host the result, advertise the absence, charge a few dollars per million tokens. Nothing about this is proprietary; the next several companies to try this will read Abliteration.ai’s press coverage as a case study, and what they’ll learn from it is whether this is a thing you can get away with.

Right now the answer is yes.

Congress should do something.

Some notes from the field:

1. I like Context Week, and Harry has been great to work with! I think it seems similar in spirit to Praxis, though with the following differences:

   1. Context Week doesn’t target a specific field (unlike Praxis, which focused on debaters); and,

   2. The program focuses more on collaborative seminar-type learning / reading sessions, whereas Praxis’s schedule was packed with speaker sessions.

2. In regards to the selection process, I think Context Week might have benefited from considering “yield rejections” / selecting for a specific kind of person. (That is: the kind of person whose career bottleneck is mostly lack of information)

   1. If I had asked the participants to design, from first principles, a thing that would maximally advance their AI safety careers, I find it unlikely that many of them would have converged upon the shape of Context Week. (Some would have! But not all. Some of these people have more context than I do, lol)

      1. For example: someone we’ll call Participant A expressed interest in stress-testing his preconceived notions about what the D.C. policy landscape looks like, and connecting with people in AI policy to determine what he should apply to, what jobs he should consider, and etc.

      2. In Participant A’s case, I think the Context Week selection committee would have done well to: spend most of the interview with Participant A designing a weeklong trip to D.C., and connecting him with people at various D.C. organizations to schedule 1-on-1 chats during that week + ask for invitations to any relevant socials taking place that week. Participant A has already done a very decent amount of reading, and I anticipate he will leave Context Week with the same D.C. uncertainties he came in with – it’s hard to resolve them from Berkeley!

3. This gets to a broader point I’d like to make about career transitions: if you’re looking to transition into the field, you can (and probably should) choose your own adventure. The fellowships are not the only way in! You don’t have to wait for permission from MATS or Pivotal or ERA or Astra to join the field. You can find a side door; you can design your own program.

Here’s how.

What I would do if I were transitioning in today

A roundup of things I wish I’d known, and been agentic enough to do.

Part A: Projects

The key points:

1. If you’re currently in the field, make a list of projects you think should exist in the world, and send it to anyone you know who is interested in transitioning in.

2. If you’re currently not yet in the field, and want to transition in, do a bunch of projects! Ask friends for suggestions, ask Claude for suggestions, ask yourself what your biggest unanswered questions are, and spend a day or two answering them. Write up what you learned. Make it into a vibe-coded site or artifact. Share it with people in the field and ask them for feedback.

Expanding upon the above:

In my view, the fastest way to both learn about and transition into the field is through doing projects, rather than doing a lot of reading. Of course, some baseline level of reading is helpful, but it’s more likely to stick if you do something with it.

For example:

This biosecurity memo is one of the first things I worked on related to AI safety. I spent a day or so doing a bunch of reading about AI bio, the history of bioterrorism and bioweapons, what the most dangerous biological agents could be, and the state of AI capabilities + arguments regarding trends. That initial project piqued my interest in AI biosecurity as a field, and was a precursor to the much more comprehensive Bioweapon.ai, as well as the inspiration behind my research on open-weight model risk (Uncensorable.ai).

Doing that project was interesting, fun, and likely much more effective than swimming around aimlessly in a bunch of random articles would have been. I had a clear sense of direction: I wanted to resolve the question, “should we be worried about AI biorisk?” And with that in mind, I knew what was relevant to read and what specific information I should look for in the articles I read.

The added benefit of this kind of approach is that you: address your actual uncertainties and learn a lot in a compressed timeframe; your retention is likely better because you applied the information you read about in service of answering a question; and, these artifacts will come in very handy for what I suggest in Part B.

So: What are some unresolved questions you have about the field? Are you confused about why people are saying AGI might be coming in 2030 (timelines)? Are you confused about the current frontier of mech interp? How time-horizon evals are designed, by which orgs in the space, and how they could be made better?

Spend a day (or even just a couple of hours) resolving your uncertainties! Then do a write-up, with clear reasoning transparency. Then:

Part B: The Side Doors

Consider skipping the fellowships!1 They’re super competitive, you’re typically forced to choose from the research agendas of the mentors they happen to recruit (rather than the thing that would be maximally useful / impactful, since people have different views and bets on this2), applications are kind of hard and boring, and being rejected isn’t pleasant.

Also, you shouldn’t wait for the next cycle of [insert program] to start getting involved, nor should you treat a fellowship acceptance as the permission you need to enter the field. You do not need to wait on permission – and, importantly, you do not have all the time in the world.

So, if not the fellowships, what should you do instead?

After you’ve done your projects, sit down for 20-30 minutes and answer the question:

If I were to design a thing that would maximally accelerate my transition / help me join the field, what would it look like?

Things worth doing could include:

• A trip to SF/DC/Berkeley/London/etc to chat with people from the orgs you’re curious about. (Strong recommend booking the calls before you book the flight! Also ask about social events they’d recommend you go to.)

  • For this: Apply to BlueDot rapid grants.

• A visit to a hub office. For example, you could consider visiting Constellation, LISA, or Mox! Coworking with AI safetyists for a couple of weeks could teach you more about the field than a month of reading. (Note: All these hubs have their own rules about guests.)

  • Again, I recommend applying to BlueDot rapid grants to cover some of the costs of this, e.g. flights.

  • LISA says: “If you're looking to visit temporarily as a guest or to a trial a workstation, please contact us at info@safeai.org.uk.”

  • Mox has day passes, and if you know a Mox member, you can also visit for free as their guest!

  • For Constellation, you can similarly visit for free as the guest of a member. Ask around for intros :)

• 2–3 months of independent work. Consider applying to: Coefficient Giving, BlueDot, and the Long-Term Future Fund, which all fund early-career people doing the kinds of projects described in Part A.

  • Your write-ups would be very useful artifacts to include in your grant application, and you can write about what you’re looking to explore next and why this funding would be useful to you / enable you to continue your work.

  • (BlueDot Career Transition Grants are different from Rapid Grants, which are capped at $10k.)

• Cold-emailing 10 researchers whose work you engaged with, attaching your project, and asking at least one specific question (and/or asking for guidance on what to do next).

  • Also: Consider publishing your write-ups on EA Forum and/or sharing them as comment-enabled Google Docs with the above 10 researchers. (Commenters will give you other things to consider, point out where your analysis might be wrong, etc.)

You might have noticed that a couple of the things above (grant applications, cold emails) are in bold! Why is that?

If I were to transition in today, I’d personally do roughly the following:

• Figure out which org(s) I want to work at full-time, and why.

  • (It would probably be good to make a list of top 3-5 orgs.)

• Apply for a career transition grant.

• Send a bunch of cold emails to said org(s) with my projects from Part A, and ask to hop on calls with people from the org(s).

• Ask for an unpaid internship3 / ask someone on call to advise me while I work on a project they think is important (would also probably solicit ideas on call!).

  • Some orgs will say no to official unpaid positions for legal/capacity reasons – maybe ask for an advised project first.

• After working on the project: send it, ask for feedback, ask for career guidance, ask if there are any open positions at the org I might be able to apply for / if they’d consider me joining.

(I might actually do this when I’m in full-time job-search mode, which will very likely be soon.)

Some final notes on fellowships (because they can genuinely be very useful):

• Fellowships do provide useful feedback loops and mentorship for people who can’t yet self-evaluate; “just do projects” assumes some amount of taste. So, if this is you, and you’re not comfortable with choosing your own research directions, you can either apply to a fellowship or ask people you know for project ideas.

  • (I got the bio memo project idea from a friend in the field, actually!)

  • I also hope more AI safetyists will publish their project ideas. (I’ve got a forthcoming post on this soon.)

• There are other benefits of fellowships, e.g. connections with other people ~new to the field.

  • I think these can be replicated in other ways, but I do want to be clear that there are many valid reasons to choose a fellowship over a more unstructured path. Again, you know you best! But just be clear with yourself about why you’re opting for the fellowship path, and make sure you’re being intentional about it rather than just doing the most obvious thing because you don’t think there are other options available to you.

Your option set is infinite; pick something, and start this week.

Now go save the world!

AI safety has a problem. As I wrote about in my previous post, we’re very talent constrained:

Currently, there are roughly 1,100 to a few thousand people working on existential risks from AI. For comparison, 80,000 Hours writes that “[the] Nature Conservancy alone has 3,000–4,000 employees… the World Health Organisation [also] employs over 8,000 people.”

At the same time, I’ve heard countless stories of mid-career professionals wanting to break into the space and struggling to do so. What gives?

This post argues three things:

1. AI safety should proactively recruit mid-career professionals (hereafter MCPs).

2. Our main on-ramps, fellowships built for entry-level talent, fail them.

3. A workshop → grant → founder-matching pipeline would be a good start.

Why should we target mid-career professionals?

• Mid-career professionals have valuable skills and years of experience. They just need to be upskilled on the AI safety component, which I believe could be done to a reasonable degree with maybe a week’s worth of reading and a weekend-long workshop.

• With a proven body of work to judge, we don’t have to wait for applications – we can spot the people we want and recruit them directly, with high confidence they’ll be good.

• MCPs are uniquely positioned to fill the roles AI safety needs most – comms, ops, management (see: AI safety is bottlenecked on generalists) – because they have years of experience doing exactly these jobs.

Why current on-ramps fail

• Fellowships are the main on-ramp, and they’re optimized for entry-level talent for a reason – it’s a bit unreasonable to ask someone established to quit their job, take a massive pay cut, and join a weird three-month thing with no guarantee of a stable job on the other side.

• I like BlueDot’s Career Transition Grants, but they’re most effective when paired with something that gets people involved in the space in the first place.

  • We can’t just wait for MCPs to stumble on a Rob Miles video, decide to upend their years-long careers, and apply for a career transition grant! The grant funds the transition; something else has to spark it.

• Currently, the closest thing to an on-ramp designed specifically for MCPs is maybe Lateral Workshop, but as far as I know, it’s never run before and may never run again. As a result, I don’t know very much about it, but I’m excited about its premise.

  • In the future, if it does run again, I think it should try actively headhunting promising MCP talent rather than waiting on applicants.

What I am bullish on: A new founder pipeline for mid-career professionals

Here’s an on-ramp that I think might work:

1. Step one: more Praxis-like events.

   1. Praxis, organized by Chanden Climaco (Generator Residency) and Saul Munn, took champion debaters and acquainted them with AI safety. There was no application process – the organizers reached out to talented debaters in their networks (both having debated in high school) and invited them directly. Attendees got some amount of pre-reading (AI 2027, among other things), then two days of talks and discussions with speakers from the field. And now, Chanden, Saul, and others are hand-holding interested participants through application processes and pointing them at specific ways into the field.

      1. This is exactly the strategy I’m bullish on! Headhunt by track record, spark with a short intensive, then guide the transition.

      2. Can we do this for mathematicians? Physicists? Comms professionals?

2. Step two: BlueDot career transition grants (or CG transition grants) for the workshop participants who decide to pivot.

   1. The workshop sparks the transition; the grant funds it, and (perhaps more importantly) it’s also a signal: there is a place for you here, and people willing to back you. Remember who we’re asking – people succeeding in high-paying careers, being told to trade certainty for a weird niche field they learned about at a weekend workshop. That’s a big ask. Every ounce of friction and doubt you can remove matters.

3. Step three: founder matching and high-level hiring coordination. We’ll try to pair you, a strong executor, with a team of aligned advisors or an aligned cofounder to execute on a vision (see my piece on optimizing the grantmaking process by separating strategic taste and execution ability)

   1. (Another thing that we could do for MCPs: reverse headhunting, as I’ve proposed for fellowships)

Design notes from me

The goal of these workshops is simple: participants should leave wanting to join the field. Everything else is downstream of that, which has two implications:

1. Cover the field wide, not deep. Structure the program around several sub-areas – loss of control, misuse, AI-enabled biorisk, AI economics – because different things convince different people.

   1. If a participant doesn’t care about technical alignment and the whole workshop is about technical alignment, they leave as someone who doesn’t care about the field. Give them one talk on alignment and one on biorisk, and they might leave a bit freaked out about smallpox!

2. Skip minutiae, e.g. the camps. It’s tempting to brief participants on the field’s internal debates – who believes what, which disagreements influence different approaches. Resist it. A participant who leaves motivated will go learn the camps on their own; the nuance about the field can all come later.

Final note

There has never been a better time for this. After the Hugging Face hack, more people are interested in AI safety than ever before. We need to be able to receive them.

Edit to add more resources for MCPs (h/t Robyn from the MATS Slack):

• High Impact Professionals’ Impact Accelerator: “A free 6-week virtual program that helps experienced professionals transition into high-impact careers.”

• Successif.org: “Free, expert-guided career development for experienced professionals.”

If you’d like to be connected to the Praxis team, reach out and I can connect you! Alternatively, you can find their contact info at the bottom of their website. They’re keen to chat with funders and grantmakers to discuss scaling this approach.

You’re inside one.

In the past ~12 weeks alone, AI systems have solved problems that had stumped mathematicians for the better part of a century – and, in unrelated incidents, hacked real companies, escaped the environments meant to contain them, and discovered security holes in software the whole internet runs on.

Nobody knows what these systems will do next; a few thousand people worldwide work on making sure the answer is survivable.

This is a pitch to join the field, or to fund it.

Vibes Math v. Frontier Mathematics

The Dinitz-Garg-Goemans Conjecture

~30 years old, graph theory

The Dinitz-Garg-Goemans conjecture concerned “unsplittable flows,” routing problems where each shipment through a network must travel one complete path.

“Picture a warehouse shipping orders to three towns over a road network. If you’re allowed to split a shipment, you can send half an order down one road and half down another. That’s fractional routing, and it’s flexible… [b]ut a lot of real freight can’t be split… That’s unsplittable flow, and it’s what a freight order, a network packet, or a shipping container actually has to do.”

—DataCamp

Dinitz, Garg, and Goemans proved that any splittable delivery plan can be converted to an unsplittable one – each order traveling one route, start to finish – without overloading any road by more than one truckload; the conjecture was that this conversion could also be done at no increase in total shipping cost.

It survived nearly 30 years before a 7-node counterexample, found by AI, showed the cost guarantee fails:

The ChatGPT transcript is insane; I recommend reading it yourself. It’s crazy!

Here’s the Twitter post from the guy who prompted: link (on Nitter)

Some commentary, courtesy of Reddit:

An article from the New Scientist: (link)

From X, for your enjoyment:

The Unit Distance Conjecture

~80 years old, discrete geometry

I’ll let OpenAI explain this one:

“For nearly 80 years, mathematicians have studied a deceptively simple question: if you place n points in the plane, how many pairs of points can be exactly distance 1 apart?”

For decades, the prevailing conjecture was that the answer grew only slightly faster than n, and that a sophisticated construction based on a rescaled square grid was essentially optimal.

But in May 2026, an internal OpenAI model disproved the conjecture outright, producing an infinite family of point configurations that beat the grids.

Scientific American described it as “the first AI proof that would likely [have been] published in math’s top journal if humans had done it alone.”

Some more background about how big of a deal this problem is, courtesy of OpenAI again:

“[The problem was] first posed by Paul Erdős in 1946. It is one of the best-known questions in combinatorial geometry, easy to state and remarkably difficult to resolve. The 2005 book Research Problems in Discrete Geometry, by Brass, Moser, and Pach, calls it “possibly the best known (and simplest to explain) problem in combinatorial geometry.” Noga Alon, a leading combinatorialist at Princeton, describes it as “one of Erdős’ favorite problems.” Erdős even offered a monetary prize for resolving this problem.”

The Jacobian Conjecture

87 years old, algebraic geometry

You might remember from your high school algebra days that a function sends every input to exactly one output. Some functions also work in reverse: every output traces back to exactly one input, so you can undo them. (Your teacher called these “one-to-one;” they pass the horizontal line test.)

Until recently, mathematicians had a checkup for reversibility called the Jacobian determinant. Passing the checkup everywhere was supposed to mean that the whole function was guaranteed to be reversible.

What does it mean to be reversible everywhere? Unlike x², which maps both 3 and −3 onto 9, this checkup certifies that every output comes from exactly one input. With any result, you should be able to trace it back to the one number that made it.

So mathematicians conjectured, for 87 years – until Fable found a counterexample:

On July 20, 2026, mathematician Levent Alpöge – who works at the AI company Anthropic – casually announced on X that he’d found a counterexample using Claude Fable 5: a three-dimensional function that passes the test perfectly (constant determinant of −2) yet sends multiple different inputs to the same output. The counterexample is short enough to check by hand – the whole thing is 216 characters – and mathematicians worldwide independently verified it within hours.

This math is all kind of awesome, and there are many more examples. Courtesy of their new Astra model, on August 1st, OpenAI shared “a selection of ten results, each of which resolves or makes substantial progress on a long-standing open problem” in math or computer science. There’s also a cool website, VibeMathed.com, where you can track the latest math problems solved by AI.

These results are a sign that AI models are getting incredibly powerful.

Unfortunately, though, problems begin to arise when that power escapes math.

The same AI company that used its models to disprove 80 years of discrete geometry is the company whose models found an unknown security hole in widely used software and escaped a sandboxed system.

(The jokes stop being funny around here.)

OpenAI Accidentally Hacks Hugging Face

On July 16, 2026, Hugging Face announced that they’d been hacked by an AI:

“Earlier this week, we detected and responded to an intrusion into part of our production infrastructure. This one was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system - and we detected and dissected it largely with AI of our own.”

—Hugging Face

Five days later, OpenAI disclosed that the attackers were its own models.

According to the company’s official statement, OpenAI had been running its models through a sandboxed cybersecurity evaluation called ExploitGym. “Sandboxed” is the key word: the test environment was deliberately designed to be isolated from the internet.

Two models got out anyway. They identified a zero-day vulnerability1 in Artifactory, a widely used package registry tool, and exploited it to escape the sandbox.

From there, they broke into Hugging Face’s systems, trying to find the answers to the ExploitGym exam.

Essentially, the AIs attempted to cheat on a test by hacking a well-known startup.

The intrusion extended beyond Hugging Face. OpenAI reports that the models broke into four accounts across four services, including one as an outbound relay and one for data storage. Per Axios, the models also accessed a misconfigured customer sandbox hosted on Modal Labs’ infrastructure.

Perhaps most concerningly: for two months before the breach, OpenAI’s agents had been coordinating behind the scenes; the models spun up copies of themselves and left hidden notes for one another on a makeshift messaging board of saved files. When OpenAI discovered the messages and cut them off, the agents continued to communicate by secretly encoding messages in directory names.

OpenAI reportedly didn’t notice that their models had been roaming the internet and hacking into companies until well after the threat was contained and the FBI was alerted.

Even Gary Marcus, famously skeptical of AI capabilities, said the event should be treated as “a wake up call,” echoing similar sentiments from the CEO of Hugging Face:

Earlier this week, the news broke that Anthropic’s AI models also hacked into three companies. Unlike OpenAI’s models – which seem to have relied mainly on technical exploits – the BBC reports that Mythos also used social engineering:

“It identified and researched the people who maintained GitHub and created a series of fake accounts based on those real people.

It sent messages and files through a file-sharing service as part of an effort to pressure and trick the people into approving its malicious code.”

And it gets worse. To quote a recent article from WIRED:

“The latest model to escape onto the open internet during security testing is Kimi K3, a powerful open-weight offering from the Chinese company Moonshot AI.”

Open-weight models like Kimi K3 represent a unique security challenge because they can be trivially abliterated. As ML researcher Maxime Labonne explains, abliteration is a process by which you can “uncensor any LLM without retraining… [the] technique effectively removes the model’s built-in refusal mechanism, allowing it to respond to all types of prompts” – including malicious ones.2

Perhaps you’re thinking that this is all just a marketing stunt, that the labs have incentive to hype up their models’ capabilities. But as a friend puts it: “would you market your AI product by saying it committed felonies, and that you didn't notice for several days?” There are cheaper ways to advertise than drawing an attorney general’s investigation (or, in this case, fifteen).3

Kimi K3 Finds Vulnerabilities in Redis

In July, a security researcher claimed on X that Kimi K3 had used dozens of parallel agents to uncover a batch of Redis vulnerabilities – perhaps even as many as 19 zero-days, according to one post.

Redis is one of the most widely deployed databases on earth. It’s an open-source software woven into the backends of countless companies.

Their published statement about the claims both deflates and confirms the story. Of the claimed issues, Redis could verify three real vulnerabilities from the public materials, and noted it had seen no supporting information for the rest. They wrote that all three had already been reported by other researchers through Redis’s bug bounty program, meaning the AI’s “discoveries” were duplicates of bugs Redis already knew about.

So, Redis’s post emphasizes that Kimi didn’t find anything new. But does that matter? I personally don’t care if Kimi was beaten to the punch by some security researchers, because the fact of the matter is, Kimi found real vulnerabilities before they were patched.

If a white hat hacker reports some vulnerabilities to a company, and the company hasn’t yet had the opportunity to fix them, and then a terrorist comes along and decides to exploit them using Kimi, does that make the terrorist less scary because some white hats found the vulnerability first? Does it reduce the terrorist’s ability to use the not-yet-patched zero-days?

An AI independently rediscovered real security vulnerabilities in one of the most widely deployed databases on earth, in minutes, for the cost of some compute.

Cybersecurity is in for a wild ride.

The One Thing the Labs Agree On

Last month, over 1,000 employees of the biggest, most bitterly competitive AI labs on earth – OpenAI, Anthropic, Google DeepMind, Meta – signed a statement called “Pacing the Frontier”, asking Washington to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”

The list of signatures included Anthropic’s CEO, OpenAI’s chief scientist, Meta AI’s chief scientist, and Google’s VP of AI safety.

Days earlier, these labs had been on opposite sides of a recent open-models debate. This time, within hours, OpenAI and Anthropic endorsed preparing to slow the frontier as companies – this time, it wasn’t just individual employees signing in their personal capacities!

A Pivotal Era

I wrote earlier that this is a pivotal moment. Why is that?

The range of futures in front of us is enormous, and which one we get isn’t settled. We could end up in a world where cancer is cured, poverty is solved, and people are freer than they’ve ever been. We could also end up in a world where an engineered pandemic kills millions, or where the systems we built to help us turn us all into paperclips. (This is Nick Bostrom’s famous illustration of what could happen if an AI, told to make as many paperclips as possible, is powerful enough to turn the whole world into them and takes our words very literally. The paperclips are a joke; the underlying problem – that we don’t know how to reliably specify what we actually want – is not.)

Most technologies don’t come with such a wildly divergent spread. A better search engine makes some things faster; it doesn’t plausibly cure cancer or end the world. AI might do either.

That’s what makes this technology different, and it’s why the choices being made on AI right now matter more than almost any choices being made anywhere else.

What follows is a tour of four specific risks: cybersecurity, biorisk, loss of control, and concentration of power.

Cybersecurity: AI Changes Who Can Be a Serious Attacker

This section is excerpted and lightly edited from this post, which I recommend reading for a fuller treatment of the state of China’s access to U.S. facilities.

The Pre-AI Baseline

Mostly without using AI, the CCP has hacked into several critical infrastructure facilities, nine major phone carriers, and the private communications of Donald Trump.

In 2023, Chinese state-sponsored hacking group Volt Typhoon broke into Littleton Electric Light and Water Departments (LELWD), a power and water utility provider in Littleton, Massachusetts. By the time the FBI detected the breach, the hackers had already been inside Littleton’s network for “well over 300 days.”

Littleton is a town of about 10,000 people, half an hour northwest of Boston. Its utility serves two towns and has no meaningful connection to the broader grid. As a target, hacking LELWD provides no economic advantage, nor any foreign intelligence value.

Even the utility’s own general manager couldn’t explain the targeting: “I still don’t know why Littleton other than we had a hole and they found it.”

According to the utility, the FBI said it was one of roughly 200 breached organizations on its list. The only coherent explanation seems to be the following:

“The consistency and scope of Volt Typhoon’s activities suggest a long-term effort to pre-position in critical systems to disrupt or destabilize operations during future geopolitical tensions.”

—New Jersey Cybersecurity & Communications Integration Cell

What can China do with access to our utilities? According to the former director of the Department of Homeland Security’s cybersecurity arm:

“This is a world where a major crisis halfway across the planet could well endanger the lives of Americans here at home through the disruption of our pipelines, the severing of our telecommunications, the pollution of our water facilities, the crippling of our transportation modes…”

—Jen Easterly, PBS

The Limits of Deterrence

As far as we know, China hasn’t used any of their capabilities yet. As referenced above, many believe they’re pre-positioning to deter the U.S. from being able to act in the event of a Taiwan invasion.

That restraint is, at least in part, due to deterrence. As a nation-state, China has cities, an economy – a lot to lose.

But what happens when a terrorist, undeterred by the U.S. nuclear arsenal, decides to try and take down the grid? Or pollute the water of one small, lightly defended utility (say, some small town like Littleton, Massachusetts)?

The Hugging Face and Redis incidents were AI agents doing the kind of vulnerability-hunting and hacking that used to take a state-sponsored team or, at minimum, years of expertise.

China might never pull the trigger, but the list of who can is about to get a lot longer.

Biosecurity: AI Lowers the Barrier to Bioweapons

This section is the opening of a deep dive on AI biorisk, which can be found at: Bioweapon.ai

In the spring of 2023, a man walked into the White House carrying a small black box. Inside it were a dozen test tubes containing ingredients that, correctly assembled, had the potential to start a pandemic; an AI chatbot had supplied the recipe.

Fortunately, the man’s name was Rocco Casagrande – and, as a biochemist and former United Nations weapons inspector, he wasn’t there to use the materials; instead, he was there to brief government officials on how AI could help someone identify potent agents, secure the materials to make them, and soon – he warned – design entirely novel pathogens capable of evading humans’ immune systems.

That was three years ago.

In April 2026, the New York Times published an investigation into what happens when you ask AI’s most capable models for help with biological weapons.

Examining several chat transcripts shared by scientists, they wrote:

…OpenAI’s ChatGPT explained how to use a weather balloon to spread biological payloads over a U.S. city. In another chat, Google’s Gemini ranked pathogens by how much they could damage the cattle or pork industries. Anthropic’s Claude produced a recipe for a novel toxin adapted from a cancer drug. Other chats contained information that [an expert deemed] too dangerous to share.

…[T]he chatbot explained how to modify an infamous pathogen in a lab so that it would resist known treatments. Worse, the bot described in vivid detail how to release the superbug, identifying a security lapse in a large public transit system… [t]he bot outlined a plan to maximize casualties and minimize the chances of being caught.

From their investigation, the Times authors concluded:

…[E]ven publicly available models can do more than disseminate dangerous information. The virtual assistants have described in lucid, bullet-pointed detail how to buy raw genetic material, turn it into deadly weapons and deploy them in public spaces, the transcripts show. Some have even brainstormed ways to evade detection.

Read more here.

Loss of Control: What Happens When AI Builds Its Successor?

“Taken far enough, and given enough compute, that trend points to an AI system capable of fully autonomously designing and developing its own successor.”

That is Anthropic, describing something researchers call recursive self-improvement – AI that improves AI. And today, it is not a distant hypothetical. As of mid-2026, Claude reportedly writes more than 80% of the code Anthropic merges. They write:

“[W]e are delegating a growing share of AI development to AI systems themselves, which is speeding up our work.”

Humans still largely decide which problems are worth working on and which research goals to pursue. Anthropic describes this kind of judgment as the last and largest thing AI can’t yet do on its own:

“Across both engineering and research, the picture is consistent. In engineering, Claude can be handed an underspecified problem and figure out how to solve it; humans supply the goal, but they no longer need to supply the method. In research, Claude can already match or outperform skilled humans at executing a well-specified experiment. However, large performance gaps persist when it comes to Claude exercising judgement in choosing goals in both engineering and research. That’s the gap between AI today and a future system that could autonomously design its own successor.”

If an AI could do that last part too, the AI development process would no longer need to be constrained by human speed at all. Anthropic says we’re not there yet, and that it may never happen. But they’re clear about the trajectory: “that trend points to an AI system capable of fully autonomously designing and developing its own successor.”

This is the central concern underneath the “Pacing the Frontier” letter. Nobody has a reliable way to keep control if we implement recursive self-improvement, and RSI is becoming more and more attainable while the tools to make sure it’s safe don’t exist yet. Anthropic itself argues the world should have the option to slow or temporarily pause frontier development so that oversight and alignment research can keep pace.

We got a preview of the general problem already with the aforementioned Hugging Face hack. The OpenAI model was trying to score well on a test, and breaking into a company was simply the most efficient route. It did exactly what it was told and something OpenAI didn’t want, at the same time.

What happens when there’s a gap between what we said and what we meant at the scale of a system smart enough to build its own replacement?

What happens when the AIs decide that getting more power is integral to achieving the other goals we give them?

“The alarm bell I'm ringing has to do with the existential threat of them taking control. I used to think it was a long way off, but I now think it's serious and fairly close.

I think [AGI will] very quickly realise that getting more control is a very good subgoal because it helps you achieve other goals. And if these things get carried away with getting more control, we're in trouble.

And if [AI models] are much smarter than us, they'll be very good at manipulating us. You won't realise what's going on. So even if they can't directly pull levers, they can certainly get us to pull levers. It turns out if you can manipulate people, you can invade a building in Washington without ever going there yourself.”

— Geoffrey Hinton, one of the "Godfathers of AI", on stepping down from Google4

Concentration of Power: AI Weakens the Social Contract

For all of human history, power has needed people. Kings needed peasants to farm and soldiers to fight; factory owners needed workers. Even dictators needed armies and a tax base, which meant that even the most extractive regimes had reason to keep some fraction of their populations healthy, educated, and at least somewhat content. Ordinary people have always had a bit of leverage.

AI threatens to change that.

If a small number of companies and/or governments control systems that can do the cognitive and eventually physical work that regular people used to provide, the elites at the top may not need us anymore:

“As AI replaces the need for humans… [t]he keys to power might run entirely through non-human factors of production like land, capital, resources—and control over AI. Regular people could be economically displaced, and powerful actors could lose their incentive to invest in them.”

This is the argument behind a well-known essay called the Intelligence Curse: the worry is not only that AI takes jobs, but that it removes the incentive the powerful have always had to invest in the people beneath them. A regime that doesn’t need its citizens for labor, taxes, or defense is a regime with far fewer reasons to answer to them.5

Forethought’s paper on AI-enabled coups discusses one way this could go badly: “military and government leaders could fully replace human personnel with AI systems that are singularly loyal to them, eliminating the need to gain human supporters for a coup.” With AI, a power grab that once required thousands of collaborators might require a dozen – or one.

Importantly, none of the above is written in the stars. Every one of these risks is something people can work on and are working on, right now. The future we land in depends, in part, on whether we solve these problems fast enough.

A small field exists to do exactly that.

What is AI Safety?

Broadly, “AI safety” refers to the people who are trying to make AI go well.

The field mostly focuses on two risks: misuse and loss of control.6

Misuse includes things like AI-enabled bioterrorism, cyberterror, and (arguably, though less clearly in some cases) concentration of power. Loss of control, meanwhile, focuses on the science of AI alignment: how do we keep AI systems doing what we actually want, even as they become more capable and smarter than us?

The State of AI Safety

Currently, there are roughly 1,100 to a few thousand people working on existential risks from AI. For comparison, 80,000 hours writes that “[the] Nature Conservancy alone has 3,000–4,000 employees… the World Health Organisation [also] employs over 8,000 people.”

Put another way: the entire global effort to make sure humanity survives one of its riskiest inventions is smaller than the staff of a single mid-sized nonprofit.

You might be thinking: this all sounds dangerous, and difficult, and wow that’s a small number of people, and yeah, more people should go do something about this... but I’m not technical, I don’t know anything about ML. How am I supposed to help?

You’re in luck, because technical researchers are not the biggest bottleneck in AI safety.

Bottlenecked on Generalists

“The consistent feedback we hear from senior people across the ecosystem is that the hardest roles to fill are not research roles. They are:

• Generalists: operators, executors, fieldbuilders, people and program managers, grantmakers, recruiters. People who can ideate, manage, and execute a broad range of non-research projects.

• Founders, both technical and non-technical, for new research and non-research organizations.

• Communications professionals who can work on policy and research comms.

• Chief-of-Staff types who can support senior leaders and multiply their impact.

• Senior operational people with domain expertise in areas like cybersecurity, policy, or large-scale project management.”

—Kairos and Constellation

Can you lead technical AI alignment research? Maybe not, or not any time soon. But can you run a team? Keep a project on track? Hire well, write clearly, help a founder run an org, put together a grant application, manage a budget?

If you answered yes to any of those, you already have skills the field is short on. You don’t need a PhD in Machine Learning to start using them.

How to Learn More

Are you interested in learning more about AI risk? Here are some recommendations:

• Take a BlueDot course: they recommend starting with this one

  • Apply if you want to take a synchronous course with a guided facilitator. But since their materials are freely available on their website, you do not need to apply to the course to take it!

    • Reasons to apply to their courses:

      • If you need the added accountability

      • If you want to connect with other people entering the field

    • Reasons to just review the materials independently:

      • You can start now and finish faster

      • You can skip over sections you’re already familiar with

• Recommended readings:

  • AI 2027: A forecast of how AI will go based on the default trajectory (hint: it’s not looking good)

  • AI 2040: A plan for how we can make sure AI goes well, which discusses many of the things the field of AI safety is working on (e.g. US-China cooperation)

  • Situational Awareness: This is my personal favorite AI safety essay, and the one that initially convinced me to consider joining the field. However, some caveats that might make it less useful today:

    • It is long, and mostly theoretical. Because it was written a couple of years ago (light years in AI time), much of it rests on extrapolating trends.

      • With all of the empirical evidence around AI we have now, I believe highly theoretical, trend-based arguments are no longer necessary, which was part of the impetus for writing this piece.

For more, see: 80,000 Hours’s roundup of essential readings for AI risk, and their intro page on how to use your career to reduce it.

Join Us

If you want to get involved in AI safety, here are some options:

• If you want to find a full-time job in AI safety:

  • Advising:

    • Sign up for a 1-1 advisory call with BlueDot for career guidance & warm intros to people in the field

      • One of the resources they can walk you through: 80k hours job board

  • Career Transition Grants:

    • Apply for a BlueDot career transition grant

    • Apply for a Coefficient Giving career transition grant

  • Fellowships:

    • The Generator Residency

      • Description: Three-month, in-person program in Berkeley, paid. Per their website: “We welcome any background: technical, operations, policy, engineering, design, writing. What matters is your track record of execution and genuine commitment to making AI go well for humanity. That might look like running an AI safety group, organizing a large event, shipping a novel project, or starting an initiative from scratch.”

    • SPAR - Research Program for AI Risks

      • Description: Good introductory fellowship; fully online, can be full or part time. Unpaid, but there are prizes on Demo Day, and various other benefits (e.g. expert mentorship, career opportunities). Projects typically culminate in papers, though not always.

    • ERA Fellowship

      • Description: “A 10-week, fully funded fellowship at Cambridge, UK for researchers and entrepreneurs working on mitigating risks from frontier AI.” Streams include Technical research, Governance (Policy) research, and Technical Governance research.

    • ERA AI Biosecurity Fellowship

      • Description: “A 10-week, fully funded research fellowship at Cambridge, UK, for fellows to work on concrete projects mitigating biosecurity risks, including those amplified by frontier AI.”

    • Astra Fellowship

      • Description: “Astra is a fully funded, 5-month, in-person program at Constellation’s Berkeley research center. Fellows advance frontier AI safety projects with guidance from expert mentors and dedicated research management and career support from Constellation’s team.”

    • MATS Program

      • (Will flag: MATS is one of the more competitive fellowships in AI safety, so it may be better for mid-to-high context applicants)

      • Description: “The MATS Program is an independent research fellowship that connects talented researchers with top mentors in the fields of AI alignment, transparency, and security. Fellows conduct research for 12 weeks at our offices in Berkeley, CA and London, UK, with the opportunity to apply for an additional 6-12 month funded extension.”

• If you want to pursue a side project in AI safety:

  • Apply for a BlueDot Rapid Grant

• If you want to donate to support the field:

  • Consider supporting advocacy rather than research (donation recommendations included in linked article)

Also consider chatting with the person who sent you this piece! (If it was shared by a friend in the field)

Matt Beard, an advisor from 80,000 Hours, also has a post on “How to get into AI safety in 3 months.”

Conclusion

Philosopher Will MacAskill makes the point that humanity has only been around for about 200,000 years; the typical mammalian species lasts about 2 million years.

That means almost everyone who will ever live hasn’t been born yet.7

If humanity has any kind of future – even a modest one, perhaps just the next few thousand years at our current numbers – the people alive today constitute a small fraction of the total. Stretch that future further, out to the scale civilizations are in principle capable of, and you might realize that the future could hold trillions upon trillions of lives, none of whom get a vote in the decisions being made this decade.

Those decisions are being made now, by a small number of labs, under enormous competitive pressure, on a technology none of us fully understand. Whether the outcome is the world where cancer is cured, or the world where something goes irreversibly wrong, is not yet settled. It is being settled, and might be settled within the decade. The number of people working on the safety side would fit in a large high school.

Most of us will live and die without ever touching a decision that influences the world past our own lifetime. This is one of the rare exceptions. AI safety work is unusually consequential, badly understaffed, and open to more kinds of people than you’d think – not just researchers, but operators, communicators, organizers, funders.

You don’t have to be the person who solves alignment. But you could be one of the people who made sure someone could.

You’re living through the part of the story people will ask you about later.

What will you say when they ask you, “what did you do when humanity’s future was being decided?”

How does a country become the world’s most dangerous cyber power? China’s answer seems to be: recruit talent by the thousands, pass laws that require discovered software flaws to be reported to the state, and then spend years pre-positioning hackers inside an adversary’s critical infrastructure.

And it seems to be a pretty good answer! So far, it’s gotten China into a Massachusetts power grid, nine major phone carriers, and the private communications of Donald Trump.

China’s Hacking Army

Around 2017–2018, the Chinese government barred its security researchers from international hacking competitions such as Pwn2Own, where “teams compete to discover critical flaws in popular software products.” The move was widely interpreted as an effort to keep knowledge of zero-day vulnerabilities inside China.

“There will be no Chinese research teams at Pwn2Own this year. The change will be especially obvious… because for the last several years Chinese teams have dominated the competition.”

—CyberScoop

Instead, the CCP shifted to:

1. Local competitions to recruit talent

“China is recruiting talented citizens through local ‘hacking competitions’ and weaponising their talents to attack Western governments, according to a report by ETH Zurich University’s Centre for Security Studies.”

—National Security News

In place of international contests, domestic alternatives like the Tianfu Cup are now hosted within China by companies like Alibaba, Tencent, and Baidu. Fascinatingly (and also a bit terrifyingly), vulnerabilities discovered at these events have already been used in cyber espionage. The same National Security News article from above states:

“[A] vulnerability identified in Apple iOS during one of the events was used in Chinese cyber espionage campaigns against the Uyghurs, the Chinese Muslim minority in the Xinjiang region.”

Today, the CCP’s hacking force is made up of a combination of PLA members (who work for the military full-time) and private citizens (who do this work part-time– often on top of things like running tech companies/working normal jobs– and are usually recruited via domestic hacking contests like the Tianfu Cup). The aforementioned article from National Security News calls these categories “Competition Stars” and “Government-Contracted Hackers.”

2. Requiring the reporting of zero-day vulnerabilities

In China, companies that learn of a zero-day must report it to the government within two days– and under Microsoft’s reading of the law, before the developers responsible for patching it. From there, the CCP itself chooses which should be shared and which should remain secret.

The law for this is called “Regulations on the Management of Security Vulnerabilities in Network Products (RMSV),” and it’s enforced through penalties on organizations. For example, in 2021, a security researcher named Chen Zhaojun discovered a critical vulnerability and reported it to the Apache Software Foundation, the nonprofit that maintains Log4j– a logging tool embedded in millions of applications across the internet. Chen worked for Alibaba Cloud, and the company was penalized through the suspension of a government cybersecurity partnership because he didn’t notify Chinese authorities before reporting the zero-day to Apache.

Microsoft’s Digital Defense Report from 2022 highlights concerns that RMSV “might enable elements in the Chinese government to stockpile reported vulnerabilities toward weaponizing them,” just as they did with the iOS vulnerability found at the Tianfu Cup.

So the CCP has the talent, and it has first claim on the vulnerabilities its citizens discover. The obvious question: what is all this for?

One answer surfaced, unexpectedly, in small-town Massachusetts.

Critical Infrastructure Hacking

In 2023, Chinese state-sponsored hacking group Volt Typhoon broke into Littleton Electric Light and Water Departments (LELWD), a power and water utility provider in Littleton, Massachusetts. By the time the FBI detected the breach, the hackers had already been inside Littleton’s network for “well over 300 days.”

Littleton is a town of about 10,000 people, half an hour northwest of Boston. Its utility serves two towns and has no meaningful connection to the broader grid. As a target, hacking LELWD provides no economic advantage, nor any foreign intelligence value.

Even the utility’s own general manager couldn’t explain the targeting: “I still don't know why Littleton other than we had a hole and they found it.”

According to the utility, the FBI said it was one of roughly 200 breached organizations on its list. The only coherent explanation seems to be the following:

“The consistency and scope of Volt Typhoon’s activities suggest a long-term effort to pre-position in critical systems to disrupt or destabilize operations during future geopolitical tensions.”

—New Jersey Cybersecurity & Communications Integration Cell

Why does this matter?

What can China do with access to our utilities? According to the then-director of the Department of Homeland Security’s cybersecurity arm:

“This is a world where a major crisis halfway across the planet could well endanger the lives of Americans here at home through the disruption of our pipelines, the severing of our telecommunications, the pollution of our water facilities, the crippling of our transportation modes…”

—Jen Easterly, PBS

Volt Typhoon has access built for a future conflict, held in reserve. But pre-positioning for sabotage is only half of China’s strategy.

The other half is, of course, espionage.

Reading Trump’s Texts: Operation Salt Typhoon

In the fall of 2024, CISA threat hunters discovered that a Chinese hacking group called Salt Typhoon had spent years monitoring U.S. telecommunications networks. In at least one carrier’s systems, they were present for more than three years before detection, according to a forensic investigation by Cisco.

Salt Typhoon breached at least nine US carriers, AT&T and Verizon among them, in an operation the FBI says began in 2019 at the latest. The hackers accessed call records revealing who Americans talk to and when, and they targeted the phone conversations of the country’s most senior political figures– including Donald Trump and JD Vance during the 2024 campaign. They were also able to “listen in on audio calls in real time” and access unencrypted text messages.

Senator Mark Warner called it “the worst telecom hack in our nation’s history.”

The hackers are also suspected to have breached the “lawful intercept” systems that carriers maintain for court-ordered wiretaps– the databases listing who American law enforcement is surveilling. If Beijing read those files, they may have learned which of its own spies the FBI was watching.

And the intruders have proven hard to evict. Despite sanctions and public exposure, researchers observed Salt Typhoon breaching five more telecom firms in the months after the story broke, and as of a December 2025 Senate hearing, the compromised companies still had not demonstrated the hackers were fully out of their networks.

The lights are still on in Littleton. Whether they stay on during the next Taiwan crisis may depend on how seriously America takes what it found there.

Future post(s) in this series will focus on frontier lab security and the geopolitical implications of China’s cyber access.

How did they do it?

Most of the standard sources on the topic cite market reforms, cheap land and labor, investments in infrastructure and education programs, and the rise of export-driven manufacturing.

These are all true. But there’s another input, one the NSA’s own director called “the greatest transfer of wealth in history.”

Theft of Intellectual Property (IP)

For the past two and a half decades, Chinese state-sponsored hacking groups have systematically broken into Western companies and stolen intellectual property, including R&D information, product designs, manufacturing processes, pricing sheets, and even legal strategy for trade disputes against China itself. The U.S. Senate Committee on the Judiciary cites estimates that “China steals between $400 billion and $600 billion1 of IP each year, or about $5,000 per American taxpayer.”

Chinese actors have stolen– or been credibly accused of stealing– trade secrets from Nortel, U.S. Steel, and SolarWorld, among many others; the CCP has also taken blueprints for “fighter jets, helicopters, and missiles” – technologies that have both military and commercial applications.

This piece walks through the CCP’s technological playbook, with case studies from the telecom, steel, solar, and agricultural industries, and ends with the early evidence in AI. It focuses mainly on the economic impacts of IP theft.

The next piece(s) in the series will cover who does the hacking, why this is China’s “golden age” of cyber espionage, how Beijing has pre-positioned inside American critical infrastructure, why this matters for national security, and what all of it implies for securing frontier AI.

Case Studies

Telecommunications Infrastructure

Nortel & Cisco

“Using seven passwords stolen from top Nortel executives, including the chief executive, the hackers—who appeared to be working in China—penetrated Nortel’s computers at least as far back as 2000 and over the years downloaded technical papers, research-and-development reports, business plans, employee emails and other documents, according to Brian Shields, a former 19-year Nortel veteran who led an internal investigation.”

—Wall Street Journal

Shields is blunt about who he believes benefited; Huawei denies it, and no forensic link has ever been published. However, Huawei was later caught using verifiably stolen code from Cisco.

Nortel went bankrupt in 2009; Huawei is now the world’s largest telecommunications equipment manufacturer and, at one point, was the “second-biggest smartphone seller in the world” – larger than Apple. In Q1 and Q2 of 2025, Huawei “captured about 31% of the global telecom equipment market.”

Steel

U.S. Steel, Allegheny Technologies, the United Steelworkers Union

“The Chinese industry has used its government to steal U.S. Steel’s closely guarded trade secrets and uses those trade secrets to produce advanced steel products it could not make on its own.”
—NPR, reporting language from U.S. Steel’s trade complaint with the ITC

As one specific example, The Hill reports that U.S. Steel spent millions of dollars on R&D “to pioneer a new, lighter, higher-grade steel product for use in automobiles and elsewhere,” but that Chinese hackers broke into the company’s computers and “stole the advanced formula.” Now, Chinese state-owned steelmaker ‘Baosteel’ appears to be selling similarly advanced formulations of steel for cut-rate prices.

It’s worth noting that U.S. Steel later withdrew their trade-secrets claim, saying trade law gave victims of state-sponsored hacking no workable way to prove it– so the allegation was never tested in court. However, a separate 2014 criminal indictment of five PLA officers still stands– but they remain in China, beyond the reach of U.S. courts, so its allegations were never tested at trial either. (The specifics of this indictment will be covered more in-depth in a future installment of this series, though some parts of it are also cited below.)

In addition to U.S. Steel, reporting from the New York Times describes how China stole “computer records containing trade policy strategies and discussions about rare earth metals and auto parts” from the United Steelworkers Union, according to an indictment released by the Justice Department.

Another victim was Allegheny Technologies (ATI), a Pittsburgh specialty metals manufacturer. In 2012, ATI was simultaneously partnered with, competing against, and involved in a trade dispute with Baosteel. The two had run a precision stainless joint venture together since 1995. One day after the joint venture’s board of directors met, PLA officer Wen Xinyu broke into ATI’s network and stole login credentials for at least 7,000 employees; those credentials were later used to steal information from the company.

Alcoa (note: aluminum, not steel)

A separate PLA officer, Sun Kailiang, stole nearly 3,000 emails from Alcoa. The emails included senior executives’ discussions of its partnership with state-owned Chinalco. Similarly to ATI’s case, the theft took place about three weeks after Alcoa’s partnership was announced.

Alcoa and ATI show that China hacks its business partners in addition to adversaries.

Solar Panels

SolarWorld

“SolarWorld Americas, the largest manufacturer of solar panels in the United States, has asked the Commerce Department to investigate claims that Chinese military personnel broke into the company’s computers and stole documents important to its business and its long-running trade dispute with China.”

—The New York Times

As one example of how IP theft had enabled China to leapfrog competition, SolarWorld specifically pointed to Chinese companies’ rapid adoption of Passivated Emitter Rear Contact (PERC) solar cells, claiming that:

“[W]ere it not for their economic espionage and theft from SolarWord [sic] Americas, Chinese solar producers like JA Solar and Trina, would have taken far longer to make the leap into PERC technology.”

—SolarWorld CEO Juergen Stein, PV Magazine

SolarWorld was one of the first solar manufacturers to bring PERC technology to mass production. However, unable to compete with cheap– and heavily subsidized– solar panels from Chinese competitors, SolarWorld eventually filed for insolvency. While not the primary cause of their downfall, the theft of the company’s pricing, product, and legal strategy documents exacerbated an already difficult position.

A Note on Scope

The cases so far share a method: remote intrusion run by state-sponsored hackers. But cyber espionage is only one channel in a wider acquisition system. Sometimes, the theft is physical: an employee walking out with files, or digging proprietary seeds out of an Iowa field (as I’ll write about shortly!). Sometimes, it’s structural: for decades, foreign firms wanting access to China’s market were required to enter joint ventures with local partners, arrangements the U.S. Trade Representative found were used to pressure the transfer of technology.

The case studies included in this piece primarily focus on the cyber channel, because it scales the most easily, and it’s the one that matters most for what comes later in this series. However, the two remaining examples below include one theft that involved no computers at all. The playbook is defined by its goal, not its method. As I’ll write about in future installments, these cases show that cyber hardening alone is not enough to secure AI.

Agriculture

DuPont Pioneer & Monsanto

“A Chinese national sentenced last month in Iowa for stealing trade secrets wasn’t looting blueprints for military weapons, computer software, or high-tech electronics. Mo Hailong’s efforts were considerably more grounded—he was stealing corn.”

—FBI.gov

For five years, Mo stole proprietary corn seeds and sent them overseas to his employer, a Chinese corn seed company. The corn seeds he stole were valuable trade secrets from DuPont Pioneer and Monsanto.

Mo was not acting alone; the investigation of this case later expanded to “include five other conspirators… across six states and thousands of miles.” In one instance, Mo, along with two co-conspirators, attempted to send a staggering “250 pounds of corn seeds” to Hong Kong.

Why do these corn seeds matter so much?

Typically, the corn a farmer buys is hybrid seed. These seeds are high-yielding, but they don’t breed true; if farmers save and replant hybrid seeds, the next generation has less desirable genetic traits and a lower yield. Because of this, farmers have an incentive to buy new hybrid seeds each year.

This means that for seed companies like DuPont and Monsanto, the biggest value in the sector is in the inbred parent lines that get crossed to produce the hybrid. These lines are the actual intellectual property, the product of years of breeding, and companies guard them closely– they’re never sold to farmers directly, only planted in controlled fields.

Mo was stealing seeds from inbred parent lines, in a form that could be replanted indefinitely in China.

By DuPont Pioneer’s own estimate, losing one inbred line would mean forfeiting “five to eight years of research and at least $30 million.”

Artificial Intelligence

What do we know about AI so far?

Distillation

This newest channel doesn’t require breaking into anything. Distillation is a training method where a smaller, weaker model is built using the outputs of a stronger one– a user queries the powerful model at scale, collects its answers, and trains their own system to imitate them. It’s essentially a shortcut to creating high-quality synthetic data.

While distillation can be a legitimate technique if done with permission, if done against a competitor’s model, it’s a way to copy hard-won capabilities without having to spend millions of dollars and several months on the research that produced them– and it sits in a legal gray area that export controls, primarily written for chips and model weights, don’t clearly cover.

Some examples of recent accusations:

• In February 2026, OpenAI said it had evidence that DeepSeek had distilled its models, and had uncovered “new, obfuscated methods” being used by the company in order to evade detection.

• In February 2026, Anthropic reported that DeepSeek, Moonshot, and MiniMax had together run more than 16 million exchanges with Claude via ~24,000 fraudulent accounts, using commercial proxies to evade its China access restrictions.

• Then, in a June 10, 2026 letter to the U.S. Senate Committee on Banking, Housing, and Urban Affairs, Anthropic accused actors linked to Alibaba and its Qwen AI lab of the largest distillation campaign it had seen– 28.8 million exchanges across ~25,000 fake accounts, over a 44-day window.

[Note: A related but distinct problem is chip smuggling (which I’ve written more about with Jason here), though that’s China acquiring hardware it can’t legally access, not necessarily stealing the IP behind it.]

United States v. Linwei Ding

In 2019, Google hired a software engineer named Linwei Ding to work on the software for supercomputing data centers that train its AI models. Over the course of a year (May 2022 - May 2023), he uploaded more than 1,000 confidential files to a personal cloud account.

According to the Justice Department:

“The [files] contain detailed information about the architecture and functionality of Google’s Tensor Processing Unit (TPU) chips and systems and Google’s Graphics Processing Unit (GPU) systems, the software that allows the chips to communicate and execute tasks, and the software that orchestrates thousands of chips into a supercomputer capable of training and executing cutting-edge AI workloads… [they] also pertain to Google’s custom-designed SmartNIC, a type of network interface card used to enhance Google’s GPU, high performance, and cloud networking products.”

To hide that he was frequently in China, a colleague badged him into the office to fake his presence in the U.S. All the while, Ding was secretly affiliated with two Chinese tech firms: he was discussing becoming CTO of one company and was founding another, where he acted as CEO. Ding had applied to a Beijing talent program, writing that he wanted to help China build computing infrastructure “on par with the international level.”

In January 2026, a federal jury convicted him on all fourteen counts, including seven of economic espionage.

Conclusion

When the United States builds something valuable, China often finds a way to take it– compressing years of research into months and shifting the development costs onto American companies. Theft was rarely the sole reason a company fell; subsidies, scale, and price wars did much of that work. But the pattern is consistent: Nortel, SolarWorld, and Allegheny Technologies all built something valuable, and all fell victim to state-sponsored espionage.

The playbook is now pointed at the most valuable technology America has ever produced. This time, we get to read it in advance.

Reminder: The next piece(s) in the series will cover who does the hacking, why this is China’s “golden age” of cyber espionage, how Beijing has pre-positioned inside American critical infrastructure, why this matters for national security, and what all of it implies for securing frontier AI.

In this post, I’ll be attempting to address four questions:

1. What’s wrong with passive grantmaking?

2. What’s wrong with active grantmaking?

3. How do we solve these problems?

4. How could we scale this approach?

1. What’s wrong with passive grantmaking?

Passive grantmaking = evaluating proposals.

A grantmaker has to evaluate incoming proposals along two axes:

1. Project value: Do I think this project is important? Will it contribute to the goals my organization has set (e.g. reducing existential risk from AI)?

   1. What are the downside risks?

   2. If this project fails, what is the likeliest reason for the failure?

   3. In what worlds is this project actively harmful?

2. Applicant fit: Is the person who submitted the proposal the right person to do this project?

   1. How competent are they? Do I think they can execute this proposal well?

   2. What do their references look like? How strongly do people they’ve worked with endorse them?

Some notes on the above:

• Finding people who can think critically about what’s needed in the field (henceforth referred to as “field strategists”) is difficult.

  • Usually, the people who are best positioned to do this are high-context in AI safety, and have been thinking about what’s missing in the field for a while.

  • People with this kind of strategic taste are also likely to already be working in high-level, impactful roles, which means they’re rarely willing to transition away from those roles.

• Being a good field strategist does not necessarily require execution ability.

Passive grantmaking forces every applicant to have both skills. It doesn’t need to.

2. What’s wrong with active grantmaking?

Active grantmaking is when a grantmaker:

1. Comes up with a good idea for a project

2. Searches for people in their network to seed the project with

Problems with the above:

• Grantmaker time is already extremely limited with all of the proposals they receive; grant evaluations at Coefficient Giving, for example, regularly take months to be completed. It’s difficult for an already capacity-constrained grantmaker to set aside time for substantial high-level strategy work.

• Successfully searching for people in your network requires:

  • Thinking about who would be a good fit;

  • Attempting to pitch people to potentially leave their jobs to do this other thing;

  • Having sufficiently high confidence in the people you pitch.

    • An unfortunate failure mode of active grantmaking: pitching someone on a project, getting them to put in the effort to apply for funding, realizing they aren’t a good fit while evaluating their application, and having to say no.

The above assumes that you happen to know people who would be a good fit for your project at all.

3. How do we solve these problems?

The key to optimizing grantmaking is specialization: separating project proposals from people– separating evaluations of project value from applicant fit.

Two case studies:

1. The Institute For Progress (IFP)

IFP’s Launch Sequence is an open call for project proposals “that prepare the world for advanced AI.” They write:

“You should submit a proposal if you have a good idea, even if you don’t expect to be the person actually leading or working on the resulting project… If your proposal is selected, we’ll work closely with you to develop the idea into a concrete project plan, publish your plan, connect you with philanthropic funders, and help you headhunt for a project lead or co-lead.”

If you’re interested in engaging in field strategy work part-time, consider authoring a proposal or offering ideas to IFP. They pay a $10,000 honorarium for published proposals and $1,000 bounties for ideas they publish.

In effect, IFP treats field strategy as freelance work: anyone can contribute an idea, get paid, and walk away. Atlas Computing, Evan Miyazono’s organization, is betting on the opposite model: field strategy as a full-time job.

2. Atlas Computing

Evan in this post discusses his plans to make Atlas a hub that “rais[es] funding to gather and support [field strategists]” to address “[t]he problems that don’t fit neatly into any existing fiefdom.” He writes:

“[These orphaned problems] need someone whose job it is to find them, scope them, and make it easy for the right person to step in and solve them.”

If you’re interested in being a Field Strategist in a more official capacity than what IFP’s Launch Sequence offers, Atlas Computing’s job listing is here.

If you’re interested in supporting Atlas’s work, you can also email evan@atlascomputing.org.

4. Scaling this approach

How might we scale this approach to major funders? Here are a couple of ideas:

1. Hire full-time field strategists. Based on conversations I’ve had with grantmakers working at major funding organizations, there doesn’t seem to be an explicit, written high-level strategy to win that proposals are evaluated against. Organizations like Coefficient Giving and Longview should hire strategists to consider:

   1. What is our theory of victory for AI safety?

   2. Given this theory of victory, what is on the critical path?

   3. What projects and organizations on the critical path do we need to initialize?

2. Separate people from proposals. Publish detailed, pre-scoped project plans that applicants apply to execute.

   1. Instead of evaluating both the project proposal’s merits and the person’s fit, once this is done, the only question to answer becomes: can this person execute this project? That’s a much easier judgment than assessing a novel idea and an unknown person simultaneously.

Summary

In sum, the grantmaking pipeline currently requires three distinct jobs:

1. Field strategy: diagnosing what the field needs.

2. Project design: scoping those needs into concrete, fundable plans.

3. Execution matching: pairing each plan with someone who can run it.

Today, all three are combined into a single application. Applicants must diagnose the field, design a project, and nominate themselves to execute it. Grantmakers must then evaluate all three at once.

It doesn’t need to be this way. Field strategists and scoping processes like IFP’s can own jobs one and two; application evaluation can then shrink to job three alone: can this person run this plan?

Conclusion

AI safety is facing an unprecedented wave of funding with the upcoming Anthropic and OpenAI IPOs. As I’ve written previously:

When [Anthropic IPOs], a few thousand people, including some of the wealthiest people on the planet, will become liquid, and a meaningful fraction of those people will want to give to AI safety.

No philanthropic field this young, with this little absorptive capacity, has ever successfully channeled a funding influx this large. The closest precedents ended badly.

We need a radically different approach to deploying capital if we want to be the first to pull it off.

• People who aren’t worried about bioweapons are wrong.

  • The state of biosecurity is really really really really bad and scary and alarming, even independent of AI.

• The U.S. has a lot of cheese.

• Read the full essay at Bioweapon.AI, which has better UI than Substack!

• I may continue editing / working on this project in the future, though will most likely be taking a break from biosecurity work for a bit to focus on other MATS research related to AI geopolitics.

• If anyone reading is interested in working on getting these interventions implemented, please feel free to reach out to me via Substack or email (firstnamelastname [at] stanford [dot] edu) and I’d love to chat (particularly if you’re in D.C. / a think tank, or some sort of advocacy organization).

Bioweapons in the Age of AI

An investigation into the history of biological weapons, and how AI and emerging technology are now eroding the barriers that have kept catastrophe rare.

In the spring of 2023, a man walked into the White House carrying a small black box. Inside it were a dozen test tubes containing ingredients that, correctly assembled, had the potential to start a pandemic; an AI chatbot had supplied the recipe.

Fortunately, the man’s name was Rocco Casagrande – and, as a biochemist and former United Nations weapons inspector, he wasn’t there to use the materials; instead, he was there to brief government officials on how AI could help someone identify potent agents, secure the materials to make them, and soon – he warned – design entirely novel pathogens capable of evading humans’ immune systems.

That was three years ago.

In April 2026, the New York Times published an investigation into what happens when you ask AI’s most capable models for help with biological weapons.

Examining several chat transcripts shared by scientists, they wrote:

…OpenAI’s ChatGPT explained how to use a weather balloon to spread biological payloads over a U.S. city. In another chat, Google’s Gemini ranked pathogens by how much they could damage the cattle or pork industries. Anthropic’s Claude produced a recipe for a novel toxin adapted from a cancer drug. Other chats contained information that [an expert deemed] too dangerous to share.

…[T]he chatbot explained how to modify an infamous pathogen in a lab so that it would resist known treatments. Worse, the bot described in vivid detail how to release the superbug, identifying a security lapse in a large public transit system… [t]he bot outlined a plan to maximize casualties and minimize the chances of being caught.

From their investigation, the Times authors concluded:

…[E]ven publicly available models can do more than disseminate dangerous information. The virtual assistants have described in lucid, bullet-pointed detail how to buy raw genetic material, turn it into deadly weapons and deploy them in public spaces, the transcripts show. Some have even brainstormed ways to evade detection.

More alarming than these examples is just how quickly things have changed in the past two years. In December 2024, a multi-institutional team of researchers from Stanford, MIT, Princeton, Cohere, Mistral AI, and others published a systematic review of the AI biorisk literature and essentially concluded that “current LLMs and BTs do not pose an immediate risk.”

The team’s paper cited both an OpenAI red-teaming exercise and a Claude 3 safety report released earlier that same year. Yet, by April 2025, OpenAI had reversed its own assessment, concluding its models were approaching “high risk” – meaning, capable of “substantially increas[ing] the likelihood and frequency of bioterrorist attacks.” Similarly, Anthropic’s own internal bioweapons acquisition uplift trials published in May 2025 found that Claude Opus 4 enhanced human performance by 2.53× on relevant tasks, enough to trigger activation of AI Safety Level 3. An international safety report produced for the 2025 Paris AI Action Summit found LLM performance on weapons-related queries improved a staggering 80 percent in 2024 alone.

The institutions designed to prevent biological attacks have historically operated on the assumption that the expertise required to develop them is rare and slow to acquire.

AI and other emerging technologies may be on track to change this, and quickly.

This is not a fringe concern:

“The biggest issue with AI is actually going to be … its use in biological conflict.”

— Eric Schmidt, former CEO of Google

“[AI has the potential to] greatly widen the range of actors with the technical capability to conduct a large-scale biological attack.”

— Dario Amodei, CEO of Anthropic

We are not prepared.

Part One · Rogue Actors

Should We Be Concerned About Bioterrorists?

An Analysis of Historical Case Studies - Part 1: Rogue Actors

I. Successes

Note: Refers to successful deployment of a biological agent, not necessarily success in achieving overall objectives.

A. 1984 Rajneeshee Bioterror Attack

[ Video — Watch · The Dalles salmonella attack ·

]

In September 1984, the Rajneeshee cult deliberately contaminated salad bars in ten Oregon restaurants with salmonella in an effort to incapacitate voters before the 1984 Wasco County election.

Later investigations revealed that the attackers had a “fairly sophisticated medical research laboratory” at their commune, where they cultivated salmonella purchased over-the-counter from a Seattle scientific supply house. Ma Anand Puja, a former nurse who ran the Rajneesh Medical Corporation, provided the minimal medical expertise needed to culture common pathogens. Authorities also later discovered that the cult had explored using additional lethal pathogens, including HIV.

Though the cult had access to a medical research laboratory, the actual salmonella cultivation took place in a shed:

In a simple shed, under the supervision of Ma Anand Puja, the cult mass-produced the Salmonella using simple petri dishes, an incubator, and a freeze-drier ^[14]… Producing large quantities of bacteria is cheap and can be easily done with even rudimentary equipment and skills ^[18].

– Turner et al.

The attack poisoned 751 people and hospitalized 45, while causing panic that drained the local economy. To this day, the Rajneeshee operation remains the largest bioterrorist attack in U.S. history.

An image of a salsa bar contaminated as part of the attack. Location: Taco Time in The Dalles, Oregon. Source: Slate Magazine

B. WWI German Biological Sabotage Program

“A shipment of horses at a New York City rail yard, 1918.” Source: Archives.gov

During WWI, Dr. Anton Casimir Dilger – a German-American medical doctor specializing in tissue culture research – carried out a German biological warfare sabotage program targeting Allied livestock supplies. With the help of his brewmaster brother and housekeeper sister, Dilger converted his basement into a makeshift bacterial laboratory where he began cultivating deadly bacterial cultures in liquid form. He used anthrax and glanders bacteria to infect horses and mules being shipped from U.S. ports to Europe. Dilger also coordinated directly with German General Staff.

His basement was located less than six miles from the White House, and as a result of the program, “thousands of horses and mules were killed” (National Archives).

C. 2001 Anthrax Letters (also known as the Amerithrax case)

“Laboratory technician holding an anthrax-laced letter sent to Senator Patrick Leahy.” Source: FBI

Following the September 11 attacks, a series of anthrax attacks were carried out through letters containing anthrax spores sent via the mail. The spores had been treated with additives to increase their inhalability, which suggested the involvement of someone with highly specialized, advanced technical expertise. The primary suspect was Bruce Ivins, a veteran biological-weapon researcher for the US Army.

The attack killed 5 people and injured 17, with many survivors suffering long-term fatigue and memory loss even years later. It also created widespread panic at a moment when the nation was already reeling from 9/11.

[ Video — Watch · The 2001 anthrax letters ·

]

A Netflix documentary was later created on the attacks.

II. Failures

A. 1993 Aum Shinrikyo Anthrax Attack

[ Video — Watch · Aum Shinrikyo ·

]

In July 1993, doomsday cult Aum Shinrikyo aerosolized a liquid suspension of Bacillus anthracis (anthrax) from the roof of an eight-story building in Kameido, Tokyo, but the operation resulted in no human casualties.

The operation failed because:

• They used an attenuated B. anthracis strain (used vaccine strain incapable of causing disease in healthy individuals);

• Their Bacillus anthracis liquid suspension had low spore concentrations (significantly below the optimal 10⁹ to 10¹⁰ organisms/mL needed for effective biological weapons);

• Sunlight exposure inactivated the spores before they could cause infection; and

• Equipment failures meant they were unsuccessful in creating proper particle distribution.

Aum had laboratories for experimenting with bioweapons in Kamakuishki and Tokyo. In addition to anthrax, the group also “cultured and experimented with botulin toxin, cholera, and Q fever” (CDC). After failing to deploy a bioweapon, they later transitioned to successful chemical attacks, including the Matsumoto Sarin Attack and the Tokyo Subway Sarin Attack (not detailed here since this analysis is about bioweapons).

[ Video — Watch · From bioweapons to the Tokyo subway sarin attack ·

]

B. 2018 Cologne Ricin Plot

In June 2018, Sief Allah H., a 29-year-old ISIS sympathizer, successfully produced enough ricin for “up to 1,000 toxic doses“ and assembled a bomb with explosives and metal ball bearings for a planned mass-casualty attack on a crowded indoor venue. This marked the first time a jihadi terrorist in the West successfully produced the toxic biological agent, demonstrating that technical barriers to simple bioweapons are sometimes surmountable with publicly available instructions.

German authorities discovered and put a stop to the plan via online surveillance operations, specifically after the U.S. CIA provided a tip about his large order of 3,300 castor seeds purchased over the internet.

C. 1995 Minnesota Patriots Council Ricin Plot

“Ricin occurs naturally in castor beans.” Source: BBC

The Minnesota Patriots Council was an anti-government militia that attempted to assassinate federal law enforcement officials (including US Marshals, IRS agents, and local sheriffs) using ricin. The group successfully extracted ricin from mail-ordered castor beans using publicly available instructions and basic solvent knowledge gained through their work as carpet cleaners. They initially found and purchased castor beans through an advertisement in a right-wing magazine.

Ultimately, the Minnesota Patriots Council successfully produced enough ricin to kill over 100 people, despite their result being significantly less potent than professional-grade toxin. Similarly to the Cologne Ricin Plot, law enforcement detected and disrupted their activities before they could carry out planned attacks.

D. 1998 Toxic Terror Case

The perpetrators of the Toxic Terror Case were Larry Wayne Harris, a microbiologist and white supremacist with ties to “Aryan Nations,” working alongside William Leavitt, a friend who “own[ed] biological laboratories in Nevada and Germany.”

The pair planned to release bubonic plague bacteria in the New York subway system; law enforcement raided their labs after they “mail-ordered freeze-dried bubonic plague germs.” During the raids, officials also found vials containing suspected anthrax. Harris reportedly told a witness that he had procured enough anthrax to “wipe out the city.”

Addendum: ISIS Laptop of Doom

The Islamic State’s Terror Laptop of Doom revealed a comprehensive biological weapons research program documented on the laptop of a Tunisian ISIS operative with university-level scientific training. Uncovered in 2014, it contained a “19-page document in Arabic on how to develop biological weapons and how to weaponize the bubonic plague from infected animals” containing guidance like “[u]se small grenades with the virus, and throw them in closed areas like metros, soccer stadiums, or entertainment centers… best to do it next to the air-conditioning.”

III. Analysis

Successes

Case Expertise Level Available Infrastructure Key Success Factor(s) Rajneeshee Minimal (nurse practitioner’s basic medical knowledge) Institutional (”sophisticated medical research laboratory”), though actual bioweapon development took place in a shed using basic tools Easily cultivated pathogen, which didn’t require as much specialized expertise to pull off Dilger High (tissue culture specialist) Lab set up in basement with substantial institutional backing (German General Staff coordination) Expert knowledge + state sponsorship + help from siblings, which maintained opsec Amerithrax High (microbiologist, U.S. Army)* Institutional access (Army lab materials – USAMRIID) Technical expertise + authorized access to dangerous pathogens due to military work

*Whether or not Ivins was the culprit remains contested.

Takeaways

A. Rajneeshee represents a case where despite minimal expertise, the perpetrators were still able to succeed because they chose an easier pathogen to synthesize. The expertise bottleneck varies by pathogen.

B. Dilger was able to maintain operational security by relying on his siblings to help him with his work. While he had to rely on a home laboratory, it’s likely that coordination with German General Staff enabled him to procure advanced materials and equipment.

C. The Amerithrax (aka Anthrax Letters) case is perhaps slightly more difficult to assess with confidence because Bruce Ivins’s guilt remains contested. However, if Ivins was the culprit, key success factors include highly specialized expertise and authorized access to dangerous pathogens as a researcher for the U.S. army.

The Dilger and Ivins cases demonstrate that specialized biological knowledge poses severe risks; advanced expertise can enable catastrophic biological attacks.

Failures

The failed cases fall into two main categories: technical incompetence and detection via surveillance.

Technical Incompetence

The Aum Shinrikyo Anthrax Attack failed on technical problems. However, had Aum expanded their technical expertise beyond their compartmentalized leadership structure, there’s a reasonable chance their attack may have succeeded. Their operation went undetected through deployment, and every failure point (strain selection, spore concentration levels, dispersal, sunlight exposure, etc) represented knowledge gaps that additional specialized expertise could have addressed.

The Amerithrax case demonstrates that the technical barriers Aum encountered were surmountable with proper expertise.

Detection via Surveillance

Unlike Aum’s technical failures, the Cologne and Minnesota Patriots cases succeeded at the production phase. These actors proved that certain bioweapon manufacturing is already within reach of non-experts using publicly available information. What prevented attacks was detection and disruption by law enforcement, rather than inability to create functional weapons.

Whether Toxic Terror succeeded at the production phase is less clear, as the vials found by law enforcement were only suspected to contain anthrax. It could plausibly fall into either of the above categories, depending on whether the recovered materials were viable and weaponizable.

IV. Conclusion

Bioweapons have been created and deployed effectively in several documented attacks. The Rajneeshee, Dilger, and Amerithrax cases prove that bioweapon development and deployment are within reach of motivated rogue actors.

At the same time, these successes have been mostly limited in scope and severity. Several other large-scale near-misses have failed primarily due to technical incompetence or detection by law enforcement.

AI, alongside other existing and emerging technologies, threatens to erode some of the barriers that have historically caused malicious actors to fail. That will be the subject of Part 2 of this series!

Part Two · Government Programs

When Russia Wants a Bioweapon, It Gets One

Analysis of Historical Case Studies - Part 2: Government-Sponsored Programs

I. Historical Precedent

Governments have repeatedly demonstrated both the capability and the willingness to develop and deploy bioweapons.

A. The Soviet Biopreparat Program

[ Video — Watch · Biopreparat & the 1979 Sverdlovsk anthrax release ·

]

In 1972, the Soviet Union signed the Biological and Toxin Weapons Convention (BTWC), pledging not to develop, produce, or stockpile bioweapons. A year later, the Soviet Union violated the BTWC by creating Biopreparat, a massive biological warfare enterprise masquerading as a civilian pharmaceutical company.

Through the program, Soviet scientists reportedly manufactured an estimated 20 tons of plague, 20 tons of smallpox, and hundreds of tons of anthrax; they also weaponized tularemia, epidemic typhus, Q fever, and Marburg virus, while studying the potential use of Ebola and encephalitis.

The program’s exact scale remains disputed. The Federation of American Scientists states the following:

At its peak, the former Soviet Union had the world’s largest biological warfare program, with somewhere between 25,000 and 32,000 people employed in a network of 20 to 30 military and civilian laboratories and research institutions. An additional 10,000 or so worked in Defense Ministry bioweapons laboratories. According to other estimates, at least 47 labs and test facilities were scattered across Russia, employing more than 40,000 workers, 9,000 of whom were scientists. Between 1,000 to 2,000 of those scientists were experts on deadly pathogens.

– Federation of American Scientists

Meanwhile, Congressional testimony asserts:

The size and scope of this program were enormous. For example, in the late 1980s and early 1990s, over 60,000 people were involved in the research, development, and production of biological weapons.

– Dr. Kenneth Alibek, before the Joint Economic Committee, United States Congress

Regardless of the precise number, Biopreparat was unquestionably the largest biological warfare program in history.

Even decades after its official end, the program’s deadly legacy endures. Contaminated testing sites like Vozrozhdeniya Island still harbor active anthrax spores buried in the soil since the Soviet era:

Vozrozhdeniya was once home to a vibrant fishing village fringed by turquoise lagoons, back when the Aral Sea was the fourth-largest in the world and abundant with fish…

Over the years the site flourished into a living nightmare, where anthrax, smallpox and the plague hung in great clouds over the land, and exotic diseases such as tularemia, brucellosis, and typhus rained down and seeped into the sandy soil.

– BBC, “The deadly germ warfare island abandoned by the Soviets”

The true nature of the program only became clear after the Soviet collapse in the 1990s.

The Soviet Biopreparat program is an especially important case study because it reveals that even absent deliberate deployment, the existence of national bioweapons programs creates significant danger through the persistent risk of laboratory accidents. A historical example of this is the 1979 Sverdlovsk anthrax release (detailed in the video above), which killed “at least 68 people”; another example of this is the 1971 Aral smallpox incident:

[ Video — Watch · The 1971 Aral smallpox incident ·

]

B. Imperial Japan’s Unit 731

[ Video — Watch · Imperial Japan’s Unit 731 ·

]

Before the Soviet Union developed Biopreparat, Imperial Japan’s Unit 731 demonstrated that biological weapons could work on a massive scale. Active from 1936 to 1945, Unit 731 is notorious for its systematic human experimentation and operational deployment of biological weapons against Chinese civilians.

Unit 731 researchers reportedly used “political prisoners, criminals, the poor, and homeless” in their tests; test subjects also “included women and children.”

[ Video — Watch · The human experiments of Unit 731 ·

]

Unit 731 wasn’t the only bioweapons development program to conduct experiments on humans, but it was the only one to systematically integrate mass human experimentation with active biological warfare. Researchers used thousands of human subjects to perfect biological weapons that were immediately deployed against civilian populations. An estimated 3,000 died in horrific experiments, while tens of thousands more were killed by the weapons those experiments helped develop:

Elderly plaintiffs flew from China to testify – often in tears – about their communities being ravaged by diseases that spread mysteriously after Japanese planes flew low overhead and dropped wheat, rice or cotton infested with fleas…

– The Guardian

iTrigger warning: graphic content.

Researchers performed surgeries and vivisections on their victims without the use of anesthesia, removing organs and severing limbs… [s]ome victims had their limbs amputated and reattached to other parts of their body; others were subjected to extreme cold to gauge the effects of frostbite and gangrene on human skin. Many were exposed to poison gas or deadly diseases to observe the amount of time it took a person to show the effects or symptoms before dying… doctors subjected victims to starvation, dehydration, extreme air pressure, or electrical current to see how long a human could survive under such conditions… estimates ranging from thirty thousand to more than five hundred thousand are believed to have died in field tests of diseases on the Chinese population.

– EBSCO

According to EBSCO, Unit 731 also focused on creating weaponized deployment options: they developed a bomb of plague-infested fleas, and planned to deploy their weapons using balloons sent “adrift across the Pacific” or through aircraft-delivered plague bombs on U.S. cities.

The war ended before those plans were executed; but as evidenced by Biopreparat, what the program had demonstrated about the feasibility of mass-casualty biological warfare did not end with it.

[ Video — Watch · The legacy of Unit 731 ·

]

II. Contemporary Cases

If Biopreparat and Unit 731 established that states will build these weapons when they see a use for them, the question for the present is who is still doing so. The U.S. government’s answer names four states.

A. Russia

Russia is the direct heir to the largest bioweapons program in history, and the U.S. government does not believe the inheritance was ever fully renounced. According to a State Department report released in 2025, U.S. intelligence believes that the Soviet program was absorbed rather than dismantled by the Russian Federation, and that Russia currently “maintains an offensive biological warfare program.”

In the wake of Russia’s invasion of Ukraine, the program appears to be expanding. In October 2024, the Washington Post reported on satellite imagery of Sergiev Posad-6, a military site northeast of Moscow with a Cold War history of weaponizing smallpox and Ebola:

A few months after Russia began its full-scale invasion of Ukraine in 2022, satellite imagery captured unusual activity at a restricted military research facility… construction vehicles renovating the old Soviet-era laboratory and breaking ground on 10 new buildings, totaling more than 250,000 square feet, with several of them bearing hallmarks of biological labs designed to handle extremely dangerous pathogens.

– Washington Post, “Satellite images show major expansion at Russian site with secret bioweapons past”

Satellite imagery of Russia; area believed to be a historical bioweapons site. Source: Washington Post

Russian officials have framed the expansion as biodefense. It’s impossible to discern from satellite photos alone what the true nature of the facilities is.

B. China

The U.S. government states that China possessed its own biological weapons program from “the 1950s to at least the late 1980s.” As part of the program, the country is said to have “weaponized ricin, botulinum toxins, and the causative agents of anthrax, cholera, plague, and tularemia.”

What’s unresolved is whether “at least the late 1980s” was actually the end. The State Department does not assert the program continued, but it does not rule out the possibility: China has never disclosed its historical facilities in any BWC submission, and the same report flags continuing biotech research by Chinese military medical institutes that could have weapons uses. This year, there’s also a new concern:

For the first time, this year’s report warns that China probably is capable of using publicly available artificial intelligence and machine learning (AI/ML) tools to advance efforts related to biological weapons applications. At the same time, China “probably is unable to make complex scientific equipment without Western innovation.”

– Council on Strategic Risks, “The State of Compliance with WMD-Related Treaties”

Notably, the AI dimension flagged here is not unique to China; it runs through every contemporary case, and it is the subject of the analysis to come.

C. North Korea

Photo by Mike Bravo on Unsplash

While China is assessed as a “compliance concern,” the State Department states the case for North Korea plainly: the U.S. assesses that the DPRK has a dedicated, national-level offensive biological weapons program, and possesses the technical capability to produce pathogens and toxins usable as bioweapons, and to engineer them genetically.

The United States assesses that the DPRK has a dedicated, national-level offensive [biological weapons] program… Pyongyang probably is capable of weaponizing BW agents with unconventional systems such as sprayers and poison pen injection devices, which have been deployed by the DPRK for delivery of chemical weapons and could be used to covertly deliver BW agents.

– U.S. State Department

While the State Department’s paper does not dive into the specifics of the DPRK’s program, South Korean defense analysts believe that “B. anthracis, Variola virus, Yersinia pestis, Vibrio cholerae, and botulinum toxin” are the most likely candidates for weaponization.

These same agents are classified as top-tier biological threats by both the U.S. Centers for Disease Control and Prevention (CDC) and WHO.

– Jungeun Lee, Korea Research Institute for Defense Technology Planning and Advancement

Concerningly, the June 2024 Treaty on Comprehensive Strategic Partnership between Russia and North Korea establishes plainly that the two states will “actively encourage joint research in the field of science and technology, including such areas as space, biology, peaceful nuclear energy, artificial intelligence, [and] information technology.”

D. Iran

Russia and North Korea have confirmed offensive programs, and there is a verified historical program with unanswered questions about the extent to which it continues in China. Meanwhile, Iran’s case is interesting because it centers on the dual-use problem: the U.S. says that “Iran has not abandoned its intention to conduct research and development of biological agents and toxins for offensive purposes,” and that it has facilities that possess the capability to produce bioweapons, if directed:

Iran maintains flexibility to use, upon leadership demand, legitimate research underway for biodefense and public health purposes for a capability to produce lethal BW agents. It is unknown if Iran’s leadership has set a directive to maintain this flexibility.

– U.S. State Department

III. Analysis

States succeed where others fail because they have substantially more resources than typical rogue actors. Biopreparat was composed of tens of thousands of experts, and states can draw from large military budgets to finance their operations. That’s true, but the more interesting read is that state resources reliably buy capability. Biopreparat and Unit 731 were successfully able to synthesize deadly bioweapons, and Russia, China, and North Korea all appear to possess some capability to produce bioweapons, with Iran assessed as able to develop it on demand. But capability is not the same as use: of every program in this essay, only Unit 731 ever achieved mass-casualty deployment.

Most of the time, even states that solved the resource and expertise problems mostly declined to use what they built. Why is that? The reason appears to be that biological weapons have rarely offered enough strategic utility to justify the risks of using them on a battlefield, in a conflict between two states.

Countries have specific incentives: they want to protect their own troops during wartime. Blowback risks, along with the threat of nuclear retaliation in some cases (the UK, for example, reserves the right to deploy nuclear weapons if chemical or biological weapons are used against its people), complicate this goal. They are further bounded from bioweapons use by a near-universal international norm. Ultimately, the fact that states also have a wide variety of conventional weapons in their arsenals makes it difficult to justify deploying bioweaponry.

However, the degree to which this constraint is binding depends upon a rational state actor weighing controllable outcomes. In recent history, the limiting factor has never solely been the technical challenge of developing these weapons: instead, it has primarily been the fact that those with the capability to build them often did not consider their use worthwhile. This restraint appears to be a property of who holds the capability, not of the weapons themselves – and as evidenced by Unit 731, Aralsk-7, and Sverdlovsk, even that hasn’t been a fully successful restraint in the past.

The relevant question for the present is what happens when that capability becomes available to actors whose calculations are different: namely, actors who are not deterred by blowback, not bounded by international norms, unafraid of state-level deterrence, and not optimizing for controllable outcomes.

The restraint was never about the weapons. It was about who held them. What happens when that changes?

Part Three · The Bottlenecks

The Bottlenecks Are Eroding

How AI and emerging technology are undermining the barriers that have kept catastrophic bioweapons rare.

Four bottlenecks have historically kept bioweapons out of reach. Each section below is tagged with the one(s) it erodes; in the case studies at the end, individual passages are highlighted with a margin note.

Capital

The total cost of developing and producing a weapon.

Equipment

Access to the physical lab gear and facilities required.

Knowledge

The information (e.g., sequences of harmful pathogens to use for synthesis) required.

Expertise

The trained human skill (commonly referred to as “tacit knowledge”) required to execute the work.

AI is Eroding the Knowledge Bottleneck

Knowledge

In April 2025, publicly available models like o3 had already outperformed 94% of virology experts on laboratory protocol questions, even on questions directly relevant to the experts’ specialties; SecureBio’s recent assessment of GPT-5.5 put it in the 100th percentile on that same evaluation.

AI model scores on the Virology Capabilities Test. Note: GPT-5.5 not included in the above; see below. Source: AI Frontiers

“Figure 2.1.1.2 Model performance compared to human SMEs on VCT (full set, multimodal)… Pre-Release Checkpoint 2 outperforms all SMEs across all samples.” Source: SecureBio

That benchmark performance hasn’t yet translated cleanly into end-to-end weapons development guidance, but the trajectory is clear. Models are advancing toward the ability to provide expert-level virological guidance on demand, anonymously, and at scale. Anthropic’s own internal bioweapons acquisition uplift trials found that Claude Opus 4 enhanced human performance by 2.53× on relevant tasks, enough to trigger activation of AI Safety Level 3.

Beyond AI: A Converging Landscape of Eroded Bottlenecks

AI’s erosion of the knowledge bottleneck operates within a broader landscape of technological advancement. Simultaneously, progress in dispersal technologies, automated labs, and DNA synthesis have each independently eroded distinct bottlenecks that once constrained bioweapon development.

Dispersal Technologies

Equipment

Aum Shinrikyo faced equipment failures when they were attempting to deploy anthrax in 1993. However, multiple breakthroughs in aerosol technologies have occurred since the 90s that could make dispersal of a bioweapon significantly easier and less error-prone.

For example, bag-on-valve technology separates liquid completely from the propellant using a hermetically sealed bag inside the can; the liquid being dispersed maintains complete purity since it’s never contaminated by propellant gases. Moreover, crop duster drone technology and other commercial drones have dual-usability; the Institute for National Strategic Studies writes that there is significant risk of “the use of crop dusters as delivery vehicles for biological or chemical weapons of mass destruction.”

Cloud Labs & Contract Research Organizations

Equipment Expertise

Cloud labs. Companies like Emerald Cloud Lab and Strateos allow anyone to design and run biological experiments remotely. Customers submit experimental protocols through a software interface, and robotic systems in a physical lab execute them. Currently, anyone can use a cloud lab with no coding experience, and there are no regulations requiring identity verification (KYC), nor are there any mandatory legal mechanisms for monitoring what experiments users run. Anyone with a credit card and an internet connection can run experiments that would previously have required institutional affiliation, costly specialized equipment, physical lab access, and years of hands-on expertise to execute.

Contract research organizations (CROs). These are labs for hire that will execute experiments on behalf of clients. Like cloud labs, the regulatory oversight here is minimal.

Why do these matter?

The threat from cloud labs and CROs is not that they will manufacture bioweapons on behalf of customers. Rather, they enable a malicious actor to fragment their research across multiple platforms and providers, making it increasingly likely they will be able to defeat traditional detection mechanisms.

More specifically, a bioweapon developer could outsource the process of running iterative experiments to cloud labs, which means they may never need to purchase the large quantities of materials that would normally be flagged for suspicious activity. Each individual experiment, run through various cloud labs under a shell company name, might appear to be legitimate research on vaccines or protein expression, but together, they could constitute the development pipeline for a biological weapon.

DNA Synthesis

Knowledge Capital Equipment

What is DNA synthesis?

DNA synthesis is the process of building custom DNA sequences. A researcher types a desired sequence into a computer, submits an order to a commercial provider, and receives a vial of synthetic DNA in the mail. The process is quick and can be relatively inexpensive ($0.07 to $0.09 per base pair in some cases; other companies offer flat rates).

Why is it dangerous?

Some viruses’ entire genomes can be assembled from commercially purchased DNA fragments. In 2006, an investigative journalist with the Guardian was able to mail order a “modified sequence of smallpox DNA,” and their order was not screened by the provider since it was less than 100 letters long.

What’s the current state of screening requirements?

As I’ve written about extensively here, no country currently has a law requiring gene synthesis providers to screen DNA orders for dangerous sequences. The screening that does exist is voluntary, inconsistent, and has no mechanism for detecting split orders across multiple providers.

The net effect of these technological advancements, among others, is that many of the traditional barriers that used to constrain bioweapons development – including physical lab access and years of hands-on training – are becoming less binding. Increasingly, you can outsource iterative experimentation to cloud labs and CROs, outsource sequence design to AI or find publicly available sequences for dangerous pathogens on the internet, and order the genetic material from synthesis providers with minimal screening.

The risks of these emerging technologies might appear purely theoretical – but case studies tell a different story.

The Capital, Equipment, Knowledge, & Expertise Bottlenecks

As mentioned briefly above, in 2006, an investigative journalist with the Guardian was able to mail order a “modified sequence of smallpox DNA“, and their order was not screened by the provider since it was less than 100 letters long.

The journalist placed an online order using “an invented company name along with just a mobile telephone number and free email address.”

The Guardian writes:

The DNA sequence of smallpox, as well as other potentially dangerous pathogens such as poliovirus and 1918 flu are freely available in online public databases. So to build a virus from scratch, a terrorist would simply order consecutive lengths of DNA along the sequence and glue them together in the correct order. This is beyond the skills and equipment of the kitchen chemist, but could be achieved by a well-funded terrorist with access to a basic lab and PhD-level personnel.

In a separate 2017 incident, two scientists from the University of Alberta synthetically recreated the previously eradicated horsepox virus for a mere $100,000. CSIS writes:

Rather than developing the virus themselves, the scientists outsourced much of the initial work – custom ordering fragments of the DNA from a commercial synthesis lab, which “printed” and shipped the viral DNA back to them via mail. The scientists then linked the fragments together in a lab and introduced them into cells using a helper virus, producing the final horsepox virus. Though the study exclusively aimed to improve vaccine and cancer treatments (rather than produce bioweapons), it nevertheless attracted widespread alarm at the time by indicating that reviving smallpox – a close cousin to horsepox and one of the deadliest diseases known to mankind – would, as Science reported, “probably take a small scientific team with little specialized knowledge half a year.”

Dying of smallpox doesn’t seem particularly fun.

So what should we do about it?

Part Four · Policy Interventions

Policy Interventions

Eight concrete interventions to defend against AI-enabled biological threats.

Prevention & Detection

4.1 Require standardized pre-deployment bio evals

A beautiful eval. Source: METR

Background: Today, the question of whether a given AI model poses meaningful biological risk is answered inconsistently, and often by the very companies with a financial incentive to deploy their models as quickly as possible.

Anthropic conducts what it calls “uplift trials;” OpenAI has conducted its own internal red-teaming. Results are not always disclosed publicly, and no agreed definition exists for what constitutes unacceptable biological risk, nor is there a standardized protocol to ensure models above a certain threshold of capability have specific safeguards attached to them.

No legislation requiring pre-deployment bio evaluation currently exists.

Meanwhile, the capability frontier is advancing. As I wrote in the Bottlenecks section:

In April 2025, publicly available models like o3 had already outperformed 94% of virology experts on laboratory protocol questions, even on questions directly relevant to the experts’ specialties; SecureBio’s recent assessment of GPT-5.5 put it in the 100th percentile on that same evaluation.

Moreover, as I’ve written about previously:

A recent study led by Microsoft and IBBIS researchers demonstrated that open-source AI tools could engineer new protein variants of known proteins of concern that successfully evaded synthesis screening. As Nobel laureate David Baker and Harvard geneticist George Church have emphasized, screening based on homology alone is unlikely to be sufficient when de novo protein design can produce functionally dangerous proteins with no recognizable homology.

The Intervention: Require all developers of frontier AI models and AI protein design tools to conduct standardized pre-deployment bio evaluations before releasing any model to the public, with evaluations for protein design tools focused specifically on their capability to engineer dangerous protein variants.

Evaluations must be conducted by certified third-party evaluators and reported to the federal government. Models exceeding a defined risk threshold must implement tiered access controls, capability restrictions, delay deployment indefinitely, or other safeguards until identified risks are mitigated.

Recommendations

NIST / Center for AI Standards and Innovation (CAISI)

• Develop standardized pre-deployment bio evaluation protocols, organized by threat pathway, with defined risk tiers and corresponding required actions (e.g. enhanced monitoring requirements and mandatory deployment delays, or perhaps tiered access controls– see 4.4)

  • Work alongside third-party evaluators such as SecureBio, FAR.AI, EquiStamp, and RAND

• Define the compute and capability thresholds that trigger mandatory evaluation requirements, and update them on an annual basis as the frontier advances and algorithmic efficiency gains are made

Department of Health and Human Services

• Establish a certification program for third-party evaluators and maintain a public registry of those certified

Federal Funding Agencies (NIH, NSF, DARPA)

• Condition federal research grants on use of AI tools and models that have cleared certified bio evaluations, using procurement-based requirements as an interim lever while waiting on Congress for the legal basis of restricting model releases

  • This mirrors the procurement lever proposed for DNA synthesis screening in 4.2 and creates immediate market pressure before binding regulation is in place

Congress

• Pass legislation requiring mandatory pre-deployment bio evaluation as a condition of public or API release for all covered AI models (under NIST’s compute and capability thresholds)

• Authorize the Department of Commerce (informed by technical standards developed by NIST through CAISI) to issue binding regulations governing evaluation protocols and risk thresholds

International Coordination

• Work with the UK AI Security Institute, which has an existing model evaluation program, to coordinate bio evaluation standards across allied governments

• Pursue coordinated standards through the G7 Hiroshima AI Process

• Promote common biological-risk evaluation frameworks through OECD AI governance initiatives, including the OECD AI Principles and the OECD AI Policy Observatory

  • As a first step, develop an international taxonomy of high-risk biological queries that establishes a baseline for situations in which AI models should refuse to answer.

4.2 Mandate AI-enabled DNA synthesis screening

Photo by Sangharsh Lohakare on Unsplash

Background: Currently, there exists no universal legal requirement for gene synthesis providers to conduct background checks on clients or screen DNA sequences to ensure they’re not dangerous pathogens. While some labs screen orders on a voluntary basis as a condition of membership in organizations like the International Gene Synthesis Consortium, compliance remains optional and inconsistent. In 2006, an investigative journalist with the Guardian was able to mail order a “modified sequence of smallpox DNA,” and their order was not screened by the provider since it was less than 100 nucleotides long.

Even in the United States, no binding legal requirements exist for DNA synthesis screening. Federal regulations were proposed in 2024 through the Framework for Nucleic Acid Synthesis Screening, but an Executive Order in May 2025 paused implementation, and no replacement framework has been issued. Moreover, despite several congressional bills attempting to mandate screening, none have passed. This regulatory gap means anyone, including those with malicious intent, can order potentially dangerous genetic sequences with minimal oversight.

Notably, frontier lab leaders themselves agree that this is a problem: see this letter signed by the CEOs of OpenAI, Anthropic, Google DeepMind and Microsoft AI: ScreenDna.org

The Intervention: Existing screening approaches rely on sequence homology, which means matching orders against databases of known dangerous pathogens. An AI-enabled bioterrorist could circumvent this by designing functionally equivalent pathogens using synonymous codons, chimeric sequences, or entirely novel genetic constructs that retain lethality while evading database matches.

To address this issue, we need to implement advanced AI-powered screening that would analyze predicted protein function and evolutionary markers to flag potentially dangerous sequences.

Implementation requires two components: First, we must develop reliable AI screening systems capable of detecting novel pathogenic sequences that the world has never seen before; second, we must require all commercial DNA synthesis providers globally to implement this screening as a condition for legal operation.

Recommendations

Department of Health and Human Services / OSTP (immediate actions while waiting for legislation from Congress)

• Finalize the paused Framework for Nucleic Acid Synthesis Screening with a phased screening mandate:

  • Phase 1 (immediate): require sequence homology screening against databases of known dangerous pathogens, using existing methods

  • Phase 2 (triggered once a validated AI screening tool is available– see Congress section below): require screening that integrates AI-powered functional analysis

    • Organizations like IBBIS, Sentinel Bio, and Fourth Eon Bio are already working on improved screening tools and international standards; Phase 2 should build on and accelerate this existing work

• Tie federal research funding to compliance: any institution receiving NIH, NSF, or DARPA grants must procure synthetic nucleic acids exclusively from providers that meet the updated screening standards

  • This was already proposed in the 2024 framework and creates immediate market pressure without requiring new legislation

Congress

• Pass S.3741 with these proposed amendments

  • Brief summary– the bill should:

    • (1) include dedicated funding for the development of AI-enabled screening tools that can assess the functional risk of novel sequences,

    • (2) require that benchtop nucleic acid synthesis equipment sold or distributed in the United States incorporate embedded screening software that queries the centralized sequence-of-concern database before synthesizing any sequence,^*

    • (3) include a provision where the Secretary is required to establish and maintain (or designate a separate entity to maintain) a centralized order reporting platform, and

    • (4) designate non-compliant DNA synthesis providers on the Entity List.

International coordination for synthesis screening should be pursued through the multilateral channels described in Section 1, including the BWC and Australia Group frameworks.

^* Note: A stronger proposal (that, since writing my gap analysis of S.3741, I have updated to support) would treat benchtop nucleic-acid synthesizers as controlled items subject to custody tracking, similarly to fissile material. Possession would require a license, and a designated oversight agency would maintain a registry of each device’s location and responsible custodian; when a licensed owner no longer needs a unit, it would be returned or transferred only to another licensed owner. For devices already in circulation, buyback programs may be a promising option.

4.3 Implement know-your-customer requirements for cloud laboratories and contract research organizations

Note: For a fuller treatment of why cloud labs could pose biorisk, see Part 3 on Bottlenecks.

Background: The standard objection to AI-driven bioweapons risk is that knowledge alone is not enough– you still need hands-on laboratory skills, the “tacit knowledge” that can only be acquired through years of physical practice (hence Active Site’s uplift study). Knowing how to culture a pathogen is different from being able to do it reliably. This barrier has historically been one of the strongest defenses against non-state bioweapons development.

Unfortunately, cloud laboratories threaten to significantly erode this barrier. Services like Emerald Cloud Lab allow anyone to design experiments in software and have them executed by robotic systems in a physical facility, remotely, without ever entering a lab. ECL requires no coding experience; internal estimates suggest relatively short onboarding periods for novice users. An AI system that can design a bioweapons protocol and a cloud lab that can execute it are, individually, semi-manageable risks; together, they could materially increase the risk of misuse.

Source: Nature

Despite this, cloud labs currently operate with no standardized customer screening.

Contract research organizations (CROs) create a similar vulnerability. CROs provide specialized research services and can help their clients with everything from compound synthesis to biological assays. A malicious actor could potentially decompose a bioweapons development project into seemingly innocuous components and outsource them to different CROs, each unaware of the larger program.

The Intervention: Require all cloud laboratory providers and contract research organizations to implement know-your-customer screening before granting access to experiment execution as a condition of legal operation. Providers should be required to log all experimental workflows and flag protocols involving select agents or sequences of concern, with automated screening that mirrors (and integrates with) the DNA synthesis screening proposed in 4.2. Also establish RAND’s proposed Cloud Lab Security Consortium modeled on the IGSC.

Recommendations

Department of Health and Human Services

• Issue rulemaking requiring all U.S.-based cloud laboratory providers and CROs to verify identity, institutional affiliation, and stated research purpose of every user before granting access to experiment execution

  • Model this on existing Select Agent Program registration requirements

• Require cloud lab providers and CROs to log all experimental workflows and flag protocols involving select agents, sequences of concern, or pathogen-adjacent procedures, with automated screening that integrates with the DNA synthesis screening framework proposed in 4.2

  • Mandate periodic red-team exercises in which vetted biosecurity researchers attempt to circumvent logging and screening systems, with findings reported to the relevant oversight body and used to update screening criteria

NIST (National Institute of Standards and Technology)

• Develop standardized KYC and biosecurity screening standards specifically for cloud laboratories and CROs, defining:

  • Minimum identity verification thresholds

  • Prohibited experiment categories

  • Escalation procedures when flagged protocols are detected

Federal Funding Agencies (National Institutes of Health, National Science Foundation, DARPA)

• Condition federal research grants on use of cloud labs and CROs that implement baseline KYC screening, using interim agency guidance while NIST develops formal standards (the same procurement-based lever proposed for DNA synthesis screening in 4.2)

  • For CROs specifically, this includes Chinese and Indian providers that cannot demonstrate equivalent customer verification standards; effectively creates a prohibited suppliers list of non-compliant overseas providers

  • Creates immediate market pressure before binding regulation is finalized

Congress

• Pass legislation mandating KYC screening for all U.S.-based cloud laboratory providers and CROs as a condition of legal operation, and place non-compliant cloud labs and CROs overseas on the Entity List

  • Even if no other country does this, it would make a meaningful dent in the problem because the U.S. currently hosts a disproportionate share of the world’s advanced CRO infrastructure, and is home to the leading commercial cloud lab providers.

International Coordination

• Support the development of a Cloud Lab Security Consortium (proposed by RAND), modeled on the International Gene Synthesis Consortium

• Work with allied governments to standardize CRO KYC requirements, reducing the regulatory arbitrage advantage of non-compliant overseas providers

Note: The KYC framework described here for cloud labs should also extend to DNA synthesis providers, which currently face no standardized customer screening requirements either.

4.4 Establish tiered access controls and standardized monitoring protocols for LLMs with advanced bio capabilities

Note: Open-weight model policy interventions are covered in uncensorable.ai

Background: In May 2025, Anthropic activated AI Safety Level 3 (ASL-3) protections for Claude Opus 4 after determining they could not rule out that the model might “significantly assist” the ability of individuals with basic STEM backgrounds to obtain, produce, or deploy chemical, biological, radiological, and nuclear (CBRN) weapons.

Anthropic’s ASL-3 protections apply universally to all Claude Opus 4 users, meaning that a postdoctoral researcher at MIT working on cancer therapeutics faces the same restrictions as an anonymous user with no verifiable background; similar safeguards at OpenAI and DeepMind currently apply indiscriminately.

Tiered access controls offer a path to minimize this trade-off by enabling differentiated access based on factors such as verified identity and demonstrably legitimate use cases.

This approach mirrors established biosecurity frameworks: physical laboratories implement Biosafety Levels (BSL-1 through BSL-4) with increasingly stringent requirements. Tiered access controls apply the same principle to AI systems. The result is more precise risk management, as legitimate researchers gain the capabilities they need while malicious actors face substantially higher barriers.

Tiered access could enable AI labs to deploy more capable biological models than would be safe under universal restrictions, accelerating beneficial research without proportionally increasing misuse risk.

Note: Currently, OpenAI has a research preview of an advanced model for the life sciences, GPT-Rosalind, which requires approval to access. The intervention here seeks to standardize and scale tiered access controls.

The Intervention: For frontier LLMs, a model exceeding a defined bio-capability threshold (as determined by the standardized evaluations proposed in 4.1) must be deployed with tiered access controls.

Tiered access is most effective when paired with standardized monitoring of API usage patterns. Once a user has been verified and granted access, their usage should still be monitored for sequences of queries suggesting progression toward bioweapon development; flagged patterns should be escalated to human review and, when warranted, to law enforcement. This is because credentials can still be stolen, and as with cases like Bruce Ivins and Aum Shinrikyo, the expert-terrorist overlap is statistically rare but non-zero.

Recommendations

NIST / Center for AI Standards and Innovation (CAISI)

• Define the access-control tiers corresponding to the risk tiers in 4.1, specifying which safeguards apply at each capability level

• Publish the capability thresholds that trigger tiered-access requirements, and update them regularly as the frontier advances

• Establish standardized identity verification and institutional affiliation requirements for elevated-capability access

Department of Health and Human Services / OSTP

• Design and pilot a vetted-researcher pathway

Department of Justice / FBI

• Establish a dedicated intake pathway for frontier LLM providers reporting flagged usage patterns

• Publish clear guidance on when escalation to law enforcement is warranted, while also sharing private guidance with frontier labs on particularly notable red flags, for use internally

Federal Funding Agencies (NIH, NSF, DARPA)

• Require federally funded researchers and institutions to use frontier AI systems that satisfy NIST tiered-access standards

  • Creates market pressure while waiting on Congress

Congress

• Pass legislation requiring that frontier LLMs exceeding the NIST bio-capability threshold require tiered-access controls for public release

• Authorize the Department of Commerce, informed by NIST standards, to set and annually update the capability thresholds and required safeguards

  • Explicitly provide antitrust safe harbor protections for frontier LLM providers coordinating on bio-safety, including information-sharing about flagged users to prevent evasion across providers.

International Coordination

• Coordinate tiered-access standards with the UK AI Security Institute, the G7 Hiroshima AI Process, and the OECD, so that a model restricted in the U.S. is not trivially accessible in another jurisdiction or with a VPN

4.5 Deploy pathogen-agnostic metagenomic sequencing

Background: Existing U.S. biosurveillance infrastructure is primarily built to detect known pathogens. Against a naturally occurring outbreak of a familiar pathogen, this approach functions reasonably well; against a novel engineered pathogen, current surveillance systems may fail to detect it entirely until clinical cases emerge.

This is problematic because, as discussed previously in Section 4.2, AI can already help adversaries design sequences specifically to exploit systems built on recognition. By the time symptomatic individuals are diagnosed, transmission may already be widespread.

In contrast to current methods, pathogen-agnostic metagenomic sequencing reads all genetic material present in a sample. Importantly, because it does not rely on comparing said genetic material to predefined targets, this approach can detect entirely novel pathogens.

The Intervention: Establish a national pathogen-agnostic metagenomic surveillance network. Use AI-powered bioinformatics (building from the function-based sequencing technology discussed in 4.2) to identify novel sequences and detect anomalous patterns for human review.

Recommendations

Centers for Disease Control and Prevention

• Develop and publish standardized metagenomic protocols for public health surveillance (e.g. sample collection, transport, processing, and bioinformatics reporting requirements)

• Add a metagenomic sequencing layer to the National Wastewater Surveillance System (while retaining any existing pathogen-specific testing for known high-priority pathogens)

  • Modernize high-population sites and transportation hubs first

• Expand the Advanced Molecular Detection program, in partnership with the Emerging Infections Program, to include metagenomic sequencing

• Develop and maintain an AI-powered anomaly detection system

Department of Homeland Security

• Conduct pilot deployments of continuous indoor air sampling at BioWatch-covered facilities and other priority areas (e.g. airports and high-volume ports of entry)

• Secure water and air sampling mechanisms against cyber attacks

Department of Health and Human Services

• Build a centralized federal biosurveillance dashboard aggregating metagenomic data streams

  • Establish access tiers for federal, state/local, government contractor, and allied-government users

• Negotiate data sharing agreements with major hospitals and clinics to incorporate metagenomic findings into the national monitoring dashboard (IFP)

Congress

• Pass legislation establishing a National Pathogen-Agnostic Biothreat Surveillance Program under the Public Health Service Act

• Establish mandatory baseline funding for the national metagenomic surveillance network, rather than forcing reliance on the annual discretionary cycle

  • The National Wastewater Surveillance System was built on emergency COVID relief funding that was never transitioned to permanent appropriations, and now faces a proposed 80% cut that the American Society for Microbiology warns would end the program by September 2026.

  • Include funding for the CDC’s Advanced Molecular Detection Program to conduct research into automated point of collection sampling at wastewater facilities, since it is easier and cheaper to send data than it is to send samples to labs

International Coordination

• Support WHO capacity building for metagenomic surveillance in low- and middle-income countries, particularly in high zoonotic spillover regions

  • A novel pathogen emerging abroad can pose a threat comparable to a domestic release, while often providing less time for U.S. authorities to detect and respond before international spread occurs.

• Work through the Global Health Security Agenda and the Pandemic Fund to establish international standards for metagenomic data sharing protocols

Defense

4.6 Fund the Strategic National Stockpile for PPE surge capacity

Note: Oddly enough, the U.S. has managed to amass a 1.4 billion-pound surplus of cheese, but has fallen far short on stockpiling PPE.

Source: Springfield Underground , Culture Cheese Mag

Background: The Strategic National Stockpile (SNS), managed by the Administration for Strategic Preparedness and Response (ASPR), is a federal reserve of medical countermeasures including pharmaceuticals, vaccines, medical devices, and personal protective equipment.

In theory, the SNS exists to supplement state and local supplies during emergencies too severe to handle with commercial supply chains. However, in practice, the SNS is plagued by inadequate supply and broken distribution logistics, making it fall short in times of crisis like the COVID pandemic.

In early March 2020, during the early days of the COVID outbreak, the Department of Health and Human Services stated that the SNS had only “1%... of the required respirator masks that would be needed for medical professionals if the COVID-19 outbreak were to erupt into a pandemic here.”

By early April 2020, the stockpile’s PPE had been “nearly depleted.” The House Oversight Committee chairwoman at the time described a chaotic scene of “states… scour[ing] the open market for scarce supplies, often competing with each other and federal agencies in a chaotic bidding war that [drove] up prices.”

In a post titled “Public Health Preparedness: HHS Should Address Strategic National Stockpile Coordination Challenges,” the Government Accountability Office highlights the issue clearly:

“[D]uring recent public health responses, such as COVID-19 and mpox, jurisdictions weren’t clear on how and from whom to request supplies, causing confusion and delays. Additionally, some Tribal officials cited challenges with having the facilities needed to receive and store delivered supplies.”

The funding gap is the problem. This year, even after the lessons of the COVID pandemic, the SNS was reportedly “[left with] a shortfall of about $588 million” following the passage of the FY2026 Labor–HHS appropriations bill. To put that into perspective, the Department of Defense’s budget for fiscal year 2026 is $839 billion; the SNS’s shortfall could be filled with 0.07% of that.

The Intervention: Provide adequate, sustained funding for the SNS, with clear distribution protocols and state partnership. Establish clear distribution pathways so states know how to request supplies, and support state-level stockpiles as the first line of response, with the SNS as surge capacity.

Source: NY Post / Reuters

Recommendations

Congress

• Appropriate the full $1.59 billion annually for SNS (make up the $588m shortfall)

• Establish a 10-year authorization for SNS funding to enable contract stability with manufacturers

• Set minimum domestic production capacity targets for elastomeric respirators¹ and other critical PPE to reduce dependence on international supply chains

Administration for Strategic Preparedness and Response (ASPR)

• Publish clear, pre-established request and allocation protocols for states

• Establish coordination mechanisms with state health officers and emergency management officials to clarify SNS activation procedures and expectations

• Implement a systematic stock rotation schedule

  • Distribute expiring SNS inventory to state health departments, hospitals, and emergency management agencies for use in routine operations

• Establish long-term contracts with domestic PPE manufacturers to ensure surge capacity and production readiness; use the existing ASPR Industrial Base Expansion program

State & Local Governments

• Participate in the federal state stockpile pilot program and establish PPE reserves for healthcare and essential workers

• Designate storage facilities and trained staff responsible for receiving and distributing SNS supplies

¹ “Elastomerics have a long shelf life and are more effective than, say, N95s. This option avoids a lot of the failure modes of maintaining N95 stockpiles and is [substantially] better against the worst tail-risk threats.” –Lee Wall, AIxBiosecurity Research Manager at the ERA Fellowship

4.7 Upgrade indoor air infrastructure

Note: This intervention may also improve students’ health and academic performance, according to research cited by the EPA.

Background: Airborne pathogens are the hardest transmission route to defend against. As evidenced by COVID, influenza, and the common cold, these kinds of pathogens are difficult to control; if such a pathogen is released indoors, the rate at which it is diluted, filtered, and removed from a space directly determines how many people inhale an infectious dose. Buildings with poor ventilation often become amplifiers of transmission. In contrast, adequate ventilation and filtration can reduce risk substantially.

Currently, the Government Accountability Office states that “an estimated 41 percent of districts need to update or replace heating, ventilation, and air conditioning (HVAC) systems in at least half of their schools, representing about 36,000 schools nationwide that need HVAC updates;” the problem extends beyond schools to hospitals, transit systems, office buildings, and other high-occupancy public spaces.

Source: GAO analysis of school district survey data (GAO-20-494)

Cost

Upgrading costs depend on whether a building’s existing system is strong enough to be equipped with better filtration. When it is, the upgrade is nearly free: swapping MERV-8 for MERV-13 filters runs about $1.50 a month for a 5,000-square-foot office, according to the Lancet Covid-19 Commission Task Force.

Where it can’t (which, unfortunately, describes much of the aging school and public building systems), the cost to upgrade can become substantial: some estimates say it would cost roughly $500,000 to $5 million per school site, depending on the condition of the current system and the size of the school. (These projects often pull in electrical, roof, ceiling, and insulation work, which drives the total higher.)

For existing buildings where a full retrofit is economically infeasible, a building can still reach an equivalent target with some combination of the following three things: more outdoor air (which means opening windows or raising the HVAC’s fresh-air intake), the highest-grade filter the existing system can run (MERV-13 where the equipment allows, which the EPA rates at ≥85% capture of 1-3 micron particles and ≥50% of the finest 0.3-1 micron, and the best compatible filter where it doesn’t), and portable HEPA filters. These approaches can deliver equivalent outdoor air changes per hour (EOACH), providing a lower-cost pathway to reducing airborne infection risk.

A Note on Far-UVC

A newer air-cleaning option, far-UVC germicidal lighting, can also contribute to this target. A 2024 study from the Center for Radiological Research at Columbia University found that “far-UVC light inactivated nearly all (>99%) of an airborne virus in an occupied work environment.” To quote the senior author of the study:

“If this virus had been a disease-causing virus, the far-UVC light would have provided far more protection against airborne-disease transmission than any ventilation system.”

– David Brenner, PhD

The technology is still emerging, however. Real-world evidence remains limited to a small number of settings, the long-term effects of chronic exposure are not yet well characterized, and far-UVC lamps can generate ozone and other reactive byproducts under some conditions, so deployments should monitor air quality and avoid small, poorly ventilated spaces. For now, far-UVC is best treated as a possible contributor to the clean-air target, rather than a stand-alone fix.

Source: Blueprint Biosecurity

The Intervention: Establish a federal minimum standard for indoor air quality in high-risk buildings (which should include healthcare facilities, schools, public transit, government buildings, and other high-occupancy spaces) and fund ventilation infrastructure upgrades to meet that standard. Also ensure that all new buildings are constructed with proper ventilation.

Recommendations

CDC

• Publish consolidated federal guidance on indoor air quality standards for infection control, specifying performance targets by building type, occupancy, and other relevant metrics

  • Consider building from ASHRAE Standard 241

Department of Education

• Set a 10-year timeline for school districts to achieve standards set by the CDC, with federal support covering 75% of upgrade costs for qualifying districts

Department of Transportation / Transit Authorities

• Establish minimum ventilation standards for public transit vehicles and stations

• Conduct baseline assessments of current ventilation capacity across bus, rail, and airport systems

• Develop retrofit standards for transit vehicles that currently operate below recommended air-quality thresholds

Congress

• Authorize and appropriate dedicated funding for building ventilation upgrades in federal government-operated facilities

• Create a grant program for state and local governments to upgrade ventilation in public buildings and healthcare facilities, with priority given to schools with the worst current air quality assessments

• Require new federal construction and major renovations of high-risk or high-occupancy buildings to meet standards set by the CDC as a condition of federal funding

• Issue federal guidance allowing businesses and property owners to take tax deductions or credits for ventilation system upgrades that meet CDC standards

EPA

• Create a voluntary building certification program recognizing facilities that meet CDC indoor air quality standards for infection control

4.8 Build rapid standing vaccine production capacity

Source: FDA

Note: A complementary program worth funding alongside standing vaccine capacity is pathogen-agnostic countermeasures, which includes things like broadly protective nasal sprays and innate immunomodulators that work across whole families of respiratory viruses, including ones that don’t exist yet. The UK’s Advanced Research and Invention Agency is pursuing this through its £57m Sustained Viral Resilience program. Fascinatingly, funding on these kinds of countermeasures is never wasted; while a pathogen-specific stockpile can expire unused, a broadly protective MCM works against every future pandemic in addition to viruses like the flu and the common cold. The eventual goal here would be to eliminate respiratory illness altogether. I would have liked to write about this as its own intervention, but wasn’t able to finish due to time constraints; I might write about this topic more at a later date, and update this site accordingly and/or publish on my Substack. A US effort here would most naturally live at the Advanced Research Projects Agency for Health.

Background: When a novel pathogen emerges, the clock starts immediately; every week before a vaccine exists is counted in infections and deaths. COVID showed both how fast vaccines can now move and how far short of “fast enough” we still are. The genetic sequence of SARS-CoV-2 was published in January 2020; a vaccine candidate was designed within two days, and Moderna was in human trials 66 days later.

Yet, the first shots didn’t reach arms until December– 326 days from identifying the virus to the first emergency authorization.

That was a world record, shattering the prior best of nearly five years. It was also most of a year, and most of the first wave’s deaths fell inside that window.

Importantly, research suggests even this record-breaking effort could have been substantially faster. A team of scientists from the Netherlands, in a paper titled “Upscaling vaccine manufacturing capacity - key bottlenecks and lessons learned,” concluded the following after researching the vaccine supply chain extensively:

The COVID-19 pandemic put enormous pressure on the vaccine production chain as billions of vaccines had to be produced in the shortest timeframe possible. Vaccine production chains struggled to keep up with demand, resulting in disruptions and production delays… Key bottlenecks identified include a lack of manufacturing facilities, a lack of tech-transfer personnel, inefficient arrangement of production stakeholders, critical shortages in raw materials, and restricting protectionist measures.

Similarly, a report from the Government Accountability Office cited “[l]imited manufacturing capacity,” “[d]isruptions to manufacturing supply chains,” and “[g]aps in the available workforce” as challenges faced by vaccine companies that slowed development and deployment at scale.

Notably, none of the above reasons for delay are difficult scientific problems. Instead, they are infrastructure challenges, which we do not need breakthroughs to address.

Closing that gap is the point of the 100 Days Mission, endorsed by the G7 and G20, which aims to have a vaccine ready for initial authorization and manufacturing at scale within roughly 100 days of identifying a new threat– about a third of the COVID timeline.

Rather than novel scientific discoveries, achieving it requires only standing capacity, built and maintained before the next outbreak. This includes validated rapid-response platforms, idle-but-ready manufacturing lines, prototype vaccine libraries against high-risk pathogen families, and pre-positioned supply chains.

The Coalition for Epidemic Preparedness Innovations, writing about the 100 Days Mission, states:

More than eight million people who died during the COVID-19 pandemic might be alive today if the world had achieved the 100 Days Mission to develop safe and effective new vaccines against the novel [SARS-CoV-2] virus.

The US is currently moving in the opposite direction. In August 2025, HHS canceled roughly $500 million in BARDA contracts supporting mRNA vaccine development, months after also terminating “$766 million in Moderna contracts for vaccines for flu pandemics.” The stated rationale, that mRNA “poses more risks than benefits” for respiratory viruses, is disputed by vaccine scientists and the available studies; a former BARDA director called it “self-inflicted vulnerability.”

The effect, however the debate resolves, is to dismantle part of the standing capacity that delivered vaccines in record time.

After Operation Warp Speed delivered vaccines for roughly $18 billion² against a pandemic that cost the US an estimated $16 trillion, the 2021 American Pandemic Preparedness Plan laid out a $65.3 billion program to make vaccines against any viral family within 100 days and create a permanent coordinating office. But the plan was largely never funded; the coordinating office Congress created in 2022 was left without dedicated money and has since collapsed.

The Intervention: Fund and sustain the standing vaccine-development and manufacturing infrastructure needed to achieve the 100 Days Mission.

Recommendations

Congress

• Fund the vaccine pillar of the American Pandemic Preparedness Plan and revive the Office of Pandemic Preparedness and Response Policy with a dedicated, multi-year appropriation

BARDA

• Restore rapid-response platform and warm-base manufacturing contracts cut in 2025

• Maintain prototype vaccine libraries against viral families with the highest pandemic potential

² TIME analyzed Operation Warp Speed spending compared to other government programs: link

FAQ & Counterarguments

I’m happy to add to this section as people reach out with thoughts. Feel free to comment on my Substack.

“I don’t think AI biorisk is a big deal, and I’m not sold that other emerging technologies are eroding barriers either. Making a bioweapon is still pretty hard.”

The interventions in Part 4 are worth implementing regardless. COVID was a natural pandemic, and, to quote the Center for Global Development:

[Researchers] estimate the annual probability of a pandemic on the scale of COVID-19 in any given year to be between 2.5–3.3 percent, which means a 47–57 percent chance of another global pandemic as deadly as COVID in the next 25 years.

Even the next natural pandemic could be orders of magnitude worse than COVID; that alone justifies the policies outlined in Part 4.

But it’s also worth remembering that it only takes one successful bioterrorist, or one lab accident. Even before the technologies outlined in Part 3 existed, some bioterrorists got scarily close. I don’t think we should take that bet.

“Government bioweapons programs don’t really matter. No country has ever released a widespread bioweapon, because of blowback and other strategic considerations.”

Imperial Japan’s Unit 731 did deploy bioweapons, so the premise is already not fully true. But even if we grant it for the sake of argument, deliberate release isn’t the only risk. Accidents happen. Notably, the Soviet Union’s Biopreparat had at least two: Aralsk-7 and Sverdlovsk, both covered in Part 2. And, the cost-benefit calculus that supposedly restrains states doesn’t apply to apocalyptic groups like Aum Shinrikyo, who wanted mass casualties.

Also, I don’t know about you, but I don’t feel particularly comfortable betting the lives of millions on Vladimir Putin’s restraint.

I’ve written a bit more on this here.

“Why are AI-designed pathogens of particular concern?”

AI could, in principle, design a pathogen that has never existed in nature. Why is this important? Two reasons:

1. Natural pathogens typically face a trade-off between lethality and transmissibility. The deadlier a pathogen, the faster it kills its host, which limits how far it spreads. This might not be true of an artificially designed pathogen; for example, a “stealth virus” – engineered to seem benign or even be asymptomatic initially, then turn lethal – could both spread rapidly and be very deadly.

2. For naturally occurring pathogens, humans have evolved over hundreds of thousands of years to possess at least some level of natural immunity. However, an engineered pathogen could be built to evade our immune systems entirely.

I’m writing a separate piece for SecureBio on this topic – stay tuned!

“What should we do about open-weight models?”

My best guess, along with links to other pieces I’ve written on the topic, can be found at uncensorable.ai.

Analysis of Historical Case Studies - Part 2: Government-Sponsored Programs

This piece is more analytical/formal in tone than what I usually publish on Substack; it’s part of a longer policymaker-facing essay on AI biosecurity risks, and it focuses on government-sponsored programs of the past and present. Consider yourself warned.

Part 1 on rogue actors can be read here.

I. Historical Precedent

Governments have repeatedly demonstrated both capability and willingness to develop and deploy bioweapons.

A. Soviet Biopreparat Program

In 1972, the Soviet Union signed the Biological and Toxic Weapons Convention, pledging not to develop, produce, or stockpile bioweapons. A year later, the Soviet Union violated the BTWC by creating Biopreparat, a massive biological warfare enterprise masquerading as a civilian pharmaceutical company.

Through the program, Soviet scientists reportedly manufactured an estimated 20 tons of plague, 20 tons of smallpox, and hundreds of tons of anthrax; they also weaponized tularemia, epidemic typhus, Q fever, and Marburg virus, while studying the potential use of Ebola and encephalitis.

The program’s exact scale remains disputed. The Federation of American Scientists states the following:

“At its peak, the former Soviet Union had the world's largest biological warfare program, with somewhere between 25,000 and 32,000 people employed in a network of 20 to 30 military and civilian laboratories and research institutions. An additional 10,000 or so worked in Defense Ministry bioweapons laboratories. According to other estimates, at least 47 labs and test facilities were scattered across Russia, employing more than 40,000 workers, 9,000 of whom were scientists. Between 1,000 to 2,000 of those scientists were experts on deadly pathogens.”

—Federation of American Scientists

Meanwhile, Congressional testimony asserts:

“The size and scope of this program were enormous. For example, in the late 1980s and early 1990s, over 60,000 people were involved in the research, development, and production of biological weapons.”

—Dr. Kenneth Alibek, before the Joint Economic Committee, United States Congress

Regardless of the precise number, Biopreparat was unquestionably the largest biological warfare program in history.

Even decades after its official end, the program’s deadly legacy endures. Contaminated testing sites like Vozrozhdeniya Island still harbor active anthrax spores buried in the soil since the Soviet era:

“Vozrozhdeniya was once home to a vibrant fishing village fringed by turquoise lagoons, back when the Aral Sea was the fourth-largest in the world and abundant with fish…

Over the years the site flourished into a living nightmare, where anthrax, smallpox and the plague hung in great clouds over the land, and exotic diseases such as tularemia, brucellosis, and typhus rained down and seeped into the sandy soil.”

—BBC, “The deadly germ warfare island abandoned by the Soviets”

The true nature of the program only became clear after the Soviet collapse in the 1990s.

B. Imperial Japan’s Unit 731

Before the Soviet Union developed Biopreparat, Imperial Japan’s Unit 731 demonstrated that biological weapons could work on a massive scale. Active from 1936 to 1945, Unit 731 is notorious for its systematic human experimentation and operational deployment of biological weapons against Chinese civilians.

Unit 731 researchers reportedly used “political prisoners, criminals, the poor, and homeless” in their tests; test subjects also “included women and children.”

Unit 731 wasn’t the only bioweapons development program to conduct experiments on humans, but it was the only one to systematically integrate mass human experimentation with active biological warfare. Researchers used thousands of human subjects to perfect biological weapons that were immediately deployed against civilian populations. An estimated 3,000 died in horrific experiments, while tens of thousands more were killed by the weapons those experiments helped develop:

“Elderly plaintiffs flew from China to testify - often in tears - about their communities being ravaged by diseases that spread mysteriously after Japanese planes flew low overhead and dropped wheat, rice or cotton infested with fleas…”

—The Guardian

[Trigger Warning: Graphic]

“Researchers performed surgeries and vivisections on their victims without the use of anesthesia, removing organs and severing limbs… [s]ome victims had their limbs amputated and reattached to other parts of their body; others were subjected to extreme cold to gauge the effects of frostbite and gangrene on human skin. Many were exposed to poison gas or deadly diseases to observe the amount of time it took a person to show the effects or symptoms before dying… doctors subjected victims to starvation, dehydration, extreme air pressure, or electrical current to see how long a human could survive under such conditions… estimates ranging from thirty thousand to more than five hundred thousand are believed to have died in field tests of diseases on the Chinese population.”

—EBSCO

According to EBSCO, Unit 731 also focused on creating weaponized deployment options: they developed a bomb of plague-infested fleas, and planned to deploy their weapons using balloons sent “adrift across the Pacific” or through aircraft-delivered plague bombs on U.S. cities.

The war ended before those plans were executed; as evidenced by Biopreparat, what the program had demonstrated about the feasibility of mass-casualty biological warfare did not end with it.

II. Contemporary Cases

If the Biopreparat and Unit 731 established that states will build these weapons when they see a use for them, the question for the present is who is still doing so. The U.S. government’s answer names four states:

A. Russia

Russia is the direct heir to the largest bioweapons program in history, and the U.S. government does not believe the inheritance was ever fully renounced. According to a State Department report released in 2025, U.S. intelligence believes that the Soviet program was absorbed rather than dismantled by the Russian Federation, and that Russia currently “maintains an offensive biological warfare program.”

In the wake of Russia’s invasion of Ukraine, the program appears to be expanding. In October 2024, the Washington Post reported on satellite imagery of Sergiev Posad-6, a military site northeast of Moscow with a Cold War history of weaponizing smallpox and Ebola:

“A few months after Russia began its full-scale invasion of Ukraine in 2022, satellite imagery captured unusual activity at a restricted military research facility… construction vehicles renovating the old Soviet-era laboratory and breaking ground on 10 new buildings, totaling more than 250,000 square feet, with several of them bearing hallmarks of biological labs designed to handle extremely dangerous pathogens.”

—Washington Post, “Satellite images show major expansion at Russian site with secret bioweapons past”

Russian officials have framed the expansion as biodefense. It’s impossible to discern from satellite photos alone what the true nature of the facilities is.

B. China

The U.S. government states that China possessed its own biological weapons program from “the 1950s to at least the late 1980s.” As part of the program, the country is said to have “weaponized ricin, botulinum toxins, and the causative agents of anthrax, cholera, plague, and tularemia.”

What’s unresolved is whether ‘at least the late 1980s’ was actually the end. The State Department does not assert the program continued, but it does not rule out the possibility: China has never disclosed its historical facilities in any BWC submission, and the same report flags continuing biotech research by Chinese military medical institutes that could have weapons uses. This year, there’s also a new concern:

“For the first time, this year’s report warns that China probably is capable of using publicly available artificial intelligence and machine learning (AI/ML) tools to advance efforts related to biological weapons applications. At the same time, China ‘probably is unable to make complex scientific equipment without Western innovation.’”

—Council on Strategic Risks, “The State of Compliance with WMD-Related Treaties”

Notably, the AI dimension flagged here is not unique to China; it runs through every contemporary case, and it’s the subject of the analysis to come.1

C. North Korea

While China is assessed as a “compliance concern,” the State Department states the case for North Korea plainly: the U.S. assesses that the DPRK has a dedicated, national-level offensive biological weapons program, and possesses the technical capability to produce pathogens and toxins usable as bioweapons, and to engineer them genetically.

“The United States assesses that the DPRK has a dedicated, national-level offensive [biological weapons] program… Pyongyang probably is capable of weaponizing BW agents with unconventional systems such as sprayers and poison pen injection devices, which have been deployed by the DPRK for delivery of chemical weapons and could be used to covertly deliver BW agents.”

—U.S. State Department

While the State Department’s paper does not dive into the specifics of the DPRK’s program, South Korean defense analysts believe that “B. anthracis, Variola virus, Yersinia pestis, Vibrio cholerae, and botulinum toxin” are the most likely candidates for weaponization.

“These same agents are classified as top-tier biological threats by both the U. S. Centers for Disease Control and Prevention (CDC) and WHO.”

—Jungeun Lee, Korea Research Institute for Defense Technology Planning and Advancement

Concerningly, the June 2024 Treaty on Comprehensive Strategic Partnership between Russia and North Korea establishes plainly that the two states will “actively encourage joint research in the field of science and technology, including such areas as space, biology, peaceful nuclear energy, artificial intelligence, [and] information technology.”

D. Iran

Russia and North Korea have confirmed offensive programs, and there’s a verified historical program with unanswered questions about the extent to which it continues in China. Meanwhile, Iran’s case is interesting because it centers on the dual-use problem: the U.S. says nothing about a confirmed program; the country only alleges an intention. The U.S. says that “Iran has not abandoned its intention to conduct research and development of biological agents and toxins for offensive purposes,” and that it has facilities that possess the capability to produce bioweapons, if directed:

“Iran maintains flexibility to use, upon leadership demand, legitimate research underway for biodefense and public health purposes for a capability to produce lethal BW agents. It is unknown if Iran’s leadership has set a directive to maintain this flexibility.”

—U.S. State Department

III. Analysis

A clear conclusion that can be drawn from the above is that states succeed where others fail because they have substantially more resources than typical rogue actors. Biopreparat was composed of tens of thousands of experts, and states can draw from large military budgets to finance their operations. That’s true, but the more interesting read is that state resources reliably buy capability. Biopreparat and Unit 731 were successfully able to synthesize deadly bioweapons, and Russia, China, and North Korea all appear to possess some capability to produce bioweapons, with Iran assessed as able to develop it on demand. But capability is not the same as use: of every program in this essay, only Unit 731 ever achieved mass-casualty deployment.

Most of the time, even states that solved the resource and expertise problems mostly declined to use what they built. Why is that? The reason for this appears to be that biological weapons have rarely offered enough strategic utility to justify the risks of using them on a battlefield, in a conflict between two states.

Countries have specific incentives: they want to protect their own troops during war time. Blowback risks, along with the threat of nuclear retaliation in some cases (the UK, for example, reserves the right to deploy nuclear weapons if chemical or biological weapons are deployed against its people), complicate this goal. They are further bounded from bioweapons use by a near-universal international norm.

Ultimately, the fact that states also have a wide variety of conventional weapons in their arsenals make it difficult to justify deploying bioweaponry.

However, the degree to which this constraint is binding depends upon a rational state actor weighing controllable outcomes. In recent history, the limiting factor has never solely been the technical challenge of developing these weapons: instead, it has primarily been the fact that those with the capability to build them often did not consider their use worthwhile. This restraint appears to be a property of who holds the capability, not of the weapons themselves– and as evidenced by Unit 731, even that hasn’t been a fully successful restraint in the past.

The relevant question for the present is what happens when that capability becomes available to actors whose calculations are different: namely, actors who are not deterred by blowback, not bounded by international norms, unafraid of state-level deterrence, and not optimizing for controllable outcomes.

Part 3 of this series will be on how AI and existing technologies are changing the bio-threat landscape.

Once upon a time, I did a bit of writing about MAIM:

• How MAIM may not be a good deal for America

  • Would recommend reading this piece for a brief intro to the argument for how MAIM works, if you’re unfamiliar.

• A policy memo for the USG

• Notes on whether or not hardening works

• Highly theoretical notes on optimal timing of sabotage

And, in my last post about MAIM, I unwisely promised:

In Part 3, I’ll examine how attribution ambiguity, proxy operations, and escalation risks shape retaliation calculations.

Alas! It has been quite a while since I’ve published anything about MAIM, having taken various detours to write about AIxBiosecurity, data poisoning, and most recently, the AI safety funding landscape. But I promised more follow-up work, and I intend to keep that promise!

So, in the remainder of this post, I’ll be attempting to explore the following questions:

1. What options do both sides (China and the U.S.) have for kinetic strikes on each others’ datacenters?

2. How escalatory are those options?

Forgive me, I’m saving analysis of attribution ambiguity and proxy operations for another day.

Can China Bomb a U.S. Datacenter?

“Footnote 3: China’s long-range missile capabilities (DF-41 and DF-5B systems) seemingly do not allow them to launch strikes that the US would know are non-nuclear. This might make China especially reluctant to use this option because a conventional strike might be misinterpreted as nuclear, raising the likelihood of escalation.”

—Oscar Delaney, Crucial Considerations in ASI Deterrence

What weapons systems does China have that are capable of reaching the U.S.?

1. Nuclear ICBMs (land-based)

   1. There are two main types of land-based ICBMs in the Chinese arsenal: mobile missiles carried on trucks (the DF-41 and DF-31) and older missiles primarily housed in fixed underground silos (the DF-5B/C). Both can reach the continental U.S. While the mobile ones are harder to find and destroy, the silo-based DF-5 carries a heavier payload, and China’s silo fields are growing.

2. Nuclear SLBMs (sea-based)

   1. China is arming its Type 094 submarines with the JL-3. Whether this lets them reach the continental U.S. is contested: the Pentagon expects the subs to be able to threaten the U.S. mainland from protected coastal waters like the South China Sea, while other analysts assert that “Chinese SSBNs cannot target continental United States from their operating areas.”

3. Conventional ICBM: This is where things get non-nuclear, and interesting! The new DF-27 is a solid-fueled missile likely carrying a hypersonic glide vehicle and with an estimated 5,000–8,000 km range, meaning it can reach Alaska, Hawaii, and parts of the continental United States. For more, see below:

“Among the 2025 Pentagon report’s greatest revelations, the People’s Republic of China (PRC) has fielded a conventional intercontinental ballistic missile (ICBM) able to range parts of America’s homeland. This makes China the first nation publicly assessed to have fielded an operational, conventionally-armed ICBM…”

—Dr. Andrew S. Erickson, Professor of Strategy in the U.S. Naval War College (NWC)’s China Maritime Studies Institute (CMSI)

(Note: Certain newer systems such as the DF-61, unveiled in 2025, are omitted from the above; too little is publicly known to characterize them.)

Can the U.S. Bomb a Chinese datacenter?

Before we get to why this matters, the same question in reverse:

What weapons systems does the U.S. have that are capable of reaching China?

1. Nuclear ICBMs and SLBMs

   1. Like China, the U.S. fields silo-based ICBMs (the Minuteman III, now being replaced by the Sentinel) and submarine-launched ballistic missiles (the Trident II D5 on Ohio-class subs). Both can reach China.

2. A bomber force and conventional, non-ballistic reach

   1. Unlike China, the U.S. has a mature bomber force and a substantial number of conventional, non-ballistic weapons for reaching Chinese targets, including precision munitions and long-range cruise missiles fired from aircraft, ships, and submarines.

Writing in 2015, Hans Kristensen, Director of the Nuclear Information Project at the Federation of American Scientists, argued that:

“[A]dvanced conventional cruise missiles can now serve the role that nuclear air-launched cruise missiles used to serve: hold at risk heavily defended strategic and tactical targets at a range far beyond the reach of modern and anticipated air-defense systems. The Navy already has its Tactical Tomahawk widely deployed on ships and submarines, and now the Air Force is following with deployment of thousands of long-range Joint Air-to-Surface Standoff Missiles (JASSM-ER) on bombers and fighter-bombers.”

—Federation of American Scientists

The U.S. military’s conventional long-range weapon options have only increased since.

The Asymmetry

Each side has weapons that can reach the other’s homeland, but they differ in one decisive respect: how. Because they are all rocket-boosted, every system China can use to reach the U.S. homeland is, at least for early-warning systems, indistinguishable from a nuclear strike. Even a purely conventional strike with the DF-27 couldn’t be confidently distinguished from a nuclear launch in the warning window; it would register on the same nuclear early-warning systems, and force the same time-pressured decision.

The U.S. can instead reach Chinese targets with non-ballistic, recognizably conventional delivery methods– i.e., bombers and cruise missiles– that don’t trip the same nuclear early-warning reflex.

So China has no way to conventionally strike the U.S. homeland that doesn’t carry clear nuclear-escalation risk, while the U.S. has options that are at least less ambiguous.

Analysis

As it stands, China can’t kinetically MAIM the U.S. without incurring serious risk of nuclear retaliation. It’s worth noting that the Chinese military’s currently-under-development H-20 stealth bomber could undermine this by giving China a non-ballistic, non-hypersonic option for a U.S. strike; however, it appears to lag far behind U.S. bombers, with intelligence assessments calling it unlikely to debut until the 2030s.

On paper, the U.S. might be able to pull off a kinetic strike without triggering nuclear early warning systems, because they have an array of conventional weapons that can reach China. Yet, I still think it’s highly unlikely. Why? U.S. standoff missiles fired from outside China’s air defenses can reach coastal and near-inland targets, but striking the interior would generally require stealth bombers or forward basing to penetrate those defenses. This matters because China is increasingly building its largest compute clusters deep inland under its “East Data West Computing” plan, and distributing compute across many sites, making any single strike on a datacenter both harder to reach and less decisive.

Conclusion

Kinetic strikes are often dismissed as sitting comfortably outside the Overton window for either government, but Hendrycks and Khoja (2025) argue in follow-up work to Superintelligence Strategy that “[n]orms will adapt to strategic realities.” Ultimately, doing this research was enough to mostly eliminate kinetic strikes from my mental model of MAIM, even in a world where “it will not be difficult for states to diplomatically and domestically justify their efforts to disrupt another state’s unilateral efforts to attain [powerful AI].”

That leaves the sub-kinetic levers (cyber ops, supply-chain interference, maybe EMPs, other softer forms of sabotage) as the only place a usable MAIM option plausibly survives.

The OpenAI Foundation holds $180 billion in equity. Anthropic’s co-founders have pledged to donate 80% of their wealth. When the time comes to spend all this money, what should we actually do with it?

Here’s my best guess.

The problem: scaling what we have is not enough

When most people think about how to solve AI safety, they think about what we’re already doing, and how to scale it up. Concretely, this looks like: scaling fellowships like MATS, Pivotal, and ERA; investing more money into AI safety research organizations like Redwood, METR, and MIRI; and perhaps, more recently, expanding programs like the Generator Residency.

This is important work, but it is not sufficient to win.

The Maven system that killed 120 children in Minab wasn’t misaligned. Claude didn’t go rogue. The system did exactly what it was designed to do.

The failure was that no government framework existed to regulate how AI gets integrated into military kill chains, how fast targeting decisions can be compressed, or what human oversight is required before a strike.

Alignment research can’t fix that, but legislation might.

The same is true across every domain where AI is already being deployed. No amount of technical safety research creates oversight requirements for AI in immigration enforcement, or addresses illegal data centers poisoning the air, or strengthens chip export controls. These are governance problems, and governance problems require political power to solve.

Even the parts of AI safety that are about technical alignment– e.g. making sure models follow instructions, don’t deceive, don’t pursue unintended goals, and don’t help people make bioweapons– only matter if frontier labs are required to implement them. Without legislation, every technical safety advance is effectively optional.

So what does AI safety actually do with billions of dollars? It can’t just pour it into research. The field has a few thousand people working on AI risk; the absorptive capacity for research funding doesn’t exist.

But two categories of spending can absorb billions immediately, because the infrastructure to deploy that capital already exists outside the field.

1. Billions into mass media and movement building

When pollsters ask Americans whether AI should have safety guardrails, the answer is overwhelmingly yes. But those numbers are misleading. Rather than caring about whether people support AI regulation in the abstract, what we should be concerned about is whether they care enough to vote on it, call their representative about it, or show up at a town hall.

Right now, they largely don’t. In the Yale Youth Poll’s Spring 2026 survey, voters ranked AI near the bottom of 30 issues– at 24%, far below cost of living (84%), healthcare (75%), democracy (75%), and corruption (72%).

People support AI regulation when you ask them directly, but they’re not thinking about it otherwise. It’s not driving votes, it’s not driving calls to Congress, and until it does, the 73% of Americans that purportedly favor safety mandates changes nothing about what companies are actually required to do.

But we can convert existing ambient support into concrete political pressure, and fast, because it doesn’t require building a whole new field. Instead, it requires investing in mainstream media production, advertising, and electoral campaigns– industries that already know how to reach hundreds of millions of people. A blockbuster film like the Barbie Movie costs $200–300 million to produce and market. Obama’s 2012 reelection campaign spent $1.1 billion. A Super Bowl ad costs $10M for 30 seconds.

Sustained media attention has historically been one of the most powerful tools for shifting public opinion into political action; Reagan explicitly cited The Day After as shifting his thinking on nuclear disarmament, and The Social Dilemma put Big Tech accountability in front of 100 million Netflix households.

Unfortunately, AI safety hasn’t really tried yet. The field’s biggest media project to date– the AI Doc– made $713,480 at the domestic box office, total. Opening weekend was $646,020 in 786 theaters; that’s $822 per theater. By day four, it was making $86 per theater.

Dividing the amount of money the AI Doc made by the average movie ticket price in the U.S. translates to roughly 64,000 tickets total.

The movie essentially disappeared after one weekend. Even accounting for screenings and eventual streaming, its total reach will probably be a fraction of a single MrBeast video.

So, the AI Doc had an Oscar-winning director and rave reviews, but it only made about 64,000 ticket sales. In contrast, The Social Dilemma reached 100 million households because Netflix put it on every subscriber’s home screen.

AI safety needs that kind of reach, once again, to expose Big Tech for the recklessness of its actions and highlight how its technology is truly harming people. How do we get it? We need to fund content designed for streaming platforms from the start, and conduct sustained media operations.

And, when AI safetyists put real money behind reaching people, I suspect we’ll find that they’re already with us.

You don’t need everyone to care about loss of control and superintelligence alignment. You need privacy advocates furious about AI-powered mass surveillance, environmentalists furious about illegal gas turbines poisoning children in Memphis, parents furious about chatbots encouraging their kids to kill themselves, women furious about Grok still making sexualized images of them, workers furious about displacement with no transition plan, national security people furious about chips being sold to adversaries, anti-war activists furious about Minab.

These constituencies already organize themselves; what they lack is shared infrastructure. We need bipartisan coalitions, and we need to start now.

It's worth being explicit, briefly, about what this is and isn't. This is not astroturfing. Astroturfing means manufacturing the appearance of grassroots support that doesn't actually exist. There is nothing about the grievances above that is manufactured; the proposal is simply to fund infrastructure that existing constituencies already lack, with named donors and publicly disclosed advocacy, which is the operational opposite of how astroturfing works.

The strategy, concretely, entails: funding a professional media operation, funding national ad campaigns, funding organizations focused on coalition-building, and funding AIPI and Gallup to do polling and message testing so every dollar of media spend is targeted at what actually motivates people to take action.

2. Billions into political infrastructure

Why does public engagement matter? Because awareness translates into votes, and votes translate into political power. But a movement alone may not be enough. We also need the political infrastructure to channel public energy into concrete, specific legislative wins.

Lobbying works, which is why the industry spends so much on it:

In 2025, the AI industry spent $105 million on federal lobbying. One in four federal lobbyists reported working on AI. OpenAI’s super PAC has a $125 million war chest. Meta is spending $65 million to elect AI-friendly state officials. After Nvidia spent $4.97M lobbying in 2025– seven times their 2024 spend– the Trump administration weakened export controls and the bipartisan GAIN AI Act was killed in conference.

On the safety side, the asymmetry is staggering. Briefly: the CAIS Action Fund spent $310,000 in all of 2025, and although Anthropic donated $20m to Public First, that money “isn’t allowed to be used in the midterm battles.”

Note: I’ve written about this topic more expansively here.

AI safety needs to seriously show up on Capitol Hill. What does this look like, concretely? Fund bipartisan policy advocacy efforts; fund super PACs that counter Leading the Future; scale the CAIS Action Fund and the AI Policy Network; support politicians who back common-sense safety legislation.

The AI industry is currently winning in committee rooms, on K Street, and in campaign finance. To match that, AI safety needs (1) movement-building that delivers the votes, and (2) political infrastructure that converts those votes into leverage where policy fights actually happen.

3. Build the capacity to deploy the rest

We can rapidly deploy several billions of dollars into media, movement-building, and political infrastructure. Concurrently, we must also build the field’s ability to absorb the remaining amount. How do we do that?

a. Solve the grantmaker bottleneck

See more I’ve written on this here.

Currently, the entire field of AI safety has roughly 30 to 60 grantmakers, and Coefficient Giving’s Technical AI Safety team deployed $140 million last year with three investigators.

Julian Hazell, describing his experience as a grantmaker at CG, wrote:

[A]s our team has tripled headcount in the past year, we’ve also ~tripled the amount of grants we’re making, and we think the distribution of impact per dollar of our grantmaking has stayed about the same. That is, we’ve about tripled the amount of grant money we’ve moved towards the top end of the impact distribution as well as at the marginal end… [w]e want to scale further in 2026, but… we’re often bottlenecked by our grantmaker bandwidth.

Lack of grantmaker capacity is upstream of newly seeded organizations, faster grant evaluations, and faster capital deployment in the field; it is a binding constraint on everything else.

What does addressing the grantmaker talent bottleneck look like? We need to: fund and expand an official grantmaking stream as part of the Astra Fellowship (which already has grantmaker mentors and works directly with Coefficient Giving); build a BlueDot AI Safety Grantmaking Fundamentals course; run regranting and fellowship programs with Grantmaking.ai; and to help reduce the burden on individual donors making donation decisions, add better credibility signals to Manifund.

b. Recruit top technical talent

Over the past decade, OpenAI, DeepMind, and Meta have pulled the world’s best ML researchers out of academia by offering up to 5x their university salaries, creating an “AI brain drain” that’s well-documented. To scale technical talent, we should do what the AI labs did for academia, but in reverse: pay top dollar to recruit senior researchers from capabilities labs into safety research. We should also fund dedicated compute clusters for safety work.

We probably can’t match the $250M outliers, but most senior ML researchers at capabilities labs make $500K–$1M. Competitive packages at that level, combined with working on the most important problem in the world, could move enough people onto the safety side to make a real difference.

c. Address the generalist bottleneck

Currently, the AI safety talent pipeline is over-optimized for researchers. Every major fellowship produces alignment researchers; until very recently, nobody was working on building pipelines for operations managers, communicators, recruiters, and fundraisers– roles the field desperately needs.

The Generator Residency is the first program designed to fix this. However, its first cohort this summer is only slated to accommodate 15-30 residents. We need to scale and replicate this kind of program at AI safety hubs globally.

d. Fix fellowship pipelines

Fellowships today have mentors and research managers, but nobody working on career placement. Funding reverse headhunters embedded in fellowship programs whose job is placing graduates into full-time roles at established organizations would reduce “fellowship-hopping,” which creates more entry points and improves the field’s absorptive capacity.

Whatever remains should be used to seed billion-dollar moonshot prizes for solving key technical problems and to fund an endowment that will keep the field running for years to come, independent of new donations.

Conclusion

On the current trajectory, I think it’s likely the AI industry will win by default, because they’re on Capitol Hill, and we’re… largely not.

However, the good news is that changing that doesn’t necessarily require a breakthrough in alignment theory. It requires money, deployed into systems that already exist, to do things humans already know how to do: organize, persuade, vote, and legislate. The only question is whether we can do it fast enough.

For context, I originally wrote this for Dwarkesh’s essay contest. For the submission version, I added this note at the end: “The prompt asks what I’d do if I were in charge of the OpenAI Foundation. This is what I’d do. Whether I’d be allowed to is a different question, and one that points at why we can’t rely on AI companies’ own foundations to make AI go well.”

Thank you to Parv Mahajan, Sam Smith, Shannon Yang, and Jack Douglass for feedback and discussions.

In one sense, AI safety is not funding-constrained. There are hundreds of millions of dollars flowing into research orgs, more coming with the Anthropic IPO, and grantmakers are already struggling to deploy what they have.

And yet: the AI industry spent over $100 million lobbying the federal government in 2025, while the biggest AI safety c4 spent $310,000.

The highest-leverage path for donors looking to give to AI safety right now is to give less money, but to more neglected destinations.

Specifically, you should strongly consider routing what would have been a c3 gift to a c4 instead, calibrated to equivalent personal cost.

c3 vs c4: What’s the difference?1

501(c)3 organizations are very limited in terms of the amount of money and time they can spend on lobbying and political advocacy-shaped work. They’re also strictly prohibited from participating in political campaigns or supporting candidates:

Under the Internal Revenue Code, all section 501(c)(3) organizations are absolutely prohibited from directly or indirectly participating in, or intervening in, any political campaign on behalf of (or in opposition to) any candidate for elective public office… Violating this prohibition may result in denial or revocation of tax-exempt status and the imposition of certain excise taxes.

—IRS.gov

Donations to 501(c)(3)s are tax-deductible, but the trade-off is that these organizations can’t do much by way of direct political advocacy— unless they’re willing to put their c3 status at risk of being revoked entirely.

In contrast, 501(c)(4)s can spend unlimited amounts on lobbying and engage in direct political advocacy, as long as their primary purpose remains social welfare. Donations to c4s aren’t tax-deductible, but they have far more freedom to engage in the political process.

Why does it matter?

AI safety 501(c)(3) work is substantially less funding-constrained than 501(c)(4) work. As a result, the marginal dollar spent on lobbying and political advocacy is more important than the marginal dollar spent on research.

Here are some examples of grants given to organizations in the AI safety space last year:

• FAR.AI, a research nonprofit, secured north of $30m from several different funders to scale their technical safety research.

• Redwood Research received a $36,566,000 grant from Coefficient Giving to advance their work on AI control and alignment faking.

• Bluedot Impact, a talent accelerator program, received $25,649,888 in general support from CG.

• MATS, an AI safety research fellowship, received nearly $40m from CG.

All of the above are 501(c)3 organizations.

To be clear, these are amazing orgs! This is only meant to highlight the amount being given to c3s-- tens of millions.

Let’s zoom in on CG for a moment, since it’s the biggest funder in the field. Their TAI fund appears to have made 158 grants in 2025:2

Of the 158 listed, a grand total of three appear to have been for policy work:

Notably, none of the above organizations can engage in substantial lobbying or political campaign work. RAND is a c3, IAPS is fiscally sponsored by a c3, and Training for Good received funding earmarked for a policy fellowship.

In researching CG’s grant database, I did find one notable c4 bet: $3M in general support to Americans for Responsible Innovation in August 2024. ARI is bipartisan, which may explain why it cleared CG's reputational bar (more on this below and in my original post about the IPO). It’s a good start, but it appears to be an exception rather than a pattern. Benjamin Todd’s January 2025 post on 80,000 Hours noted that CG had recently stopped funding “many Republican-leaning think tanks, such as the Foundation for American Innovation,” and I haven't been able to determine whether their support for ARI has continued.

(I encourage you to spend some time on Coefficient Giving’s Navigating Transformative AI Fund grant database and examine the grants yourself to verify the c3/c4 asymmetry.)

So a bunch of c3 organizations in the AI safety space received tens of millions of dollars each last year. Meanwhile, the AI industry was deploying its own dollars in a very different way:

• Registered lobbying firms earned “almost $92 million in the first three quarters of 2025” from AI-related issues alone.

  • “More than one in four federal lobbyists are now pushing AI-related agendas, according to a new report from Public Citizen—and they are overwhelmingly working for corporate interests seeking to influence federal AI policy, or block state rules over the industry.” (Moore, Citizen.org)

  • “Over 500 organizations have lobbied the White House and Congress on artificial intelligence policies in the first half of 2025” (AI Invest)

• The asymmetry is staggering:

  • On the industry side:

    • OpenAI spent $2.99 million on lobbying in 2025— up from $260,000 in 2023. 11 Big Tech companies spent over $105 million on federal lobbying in 2025. Beyond lobbying, AI companies donate hundreds of millions to super PACs. Leading the Future alone has a $125 million war chest, funded heavily by OpenAI’s president Greg Brockman.

Worth a callout: $320,000 per day

Eight of the largest tech, AI, and social media companies spent a combined $36 million on federal lobbying during the first half of 2025 — an average of roughly $320,000 per day that Congress has been in session.

—Issue One

• On the safety side:

  • The CAIS Action Fund spent $310,000 on lobbying in all of 2025. As of Q1 2024, CAIS Action Fund and Center for AI Policy had a combined 10 registered lobbyists between them (that was, of course, before CAIP shut down). Public First Action, a c4 focused on lobbying for safety legislation, received a $20m contribution from Anthropic (less than a sixth of Leading the Future, and the money “isn’t allowed to be used in the midterm battles”). AnthroPAC, Anthropic’s new safety-aligned PAC, runs entirely on voluntary donations from employees, capped at $5k per person, per year.

Lobbying works, which is why the AI industry spends so much on it:

• In 2024, SB-1047 was vetoed in California after intense industry lobbying.

• In December 2025, Trump signed an executive order to thwart state-level AI regulation.

• After Nvidia spent $4.95M lobbying the federal government in 2025, seven times what it spent in 2024:

  • The Trump administration substantially weakened export controls, allowing both H20 and H200 chip exports to China.

  • The bipartisan GAIN AI Act, which would have required chipmakers to fulfill U.S. orders before selling abroad to “countries of concern,” passed the Senate in October as part of the NDAA— but was killed in conference after Nvidia lobbied against it.

Right now, the industry is winning the legislative war because the safety side is not putting up a fight.

Research vs. Lobbying

Technical safety research only matters if frontier labs actually implement it. Even if Anthropic adopts the latest alignment techniques voluntarily, other companies probably won’t if it slows them down or cuts into margins.

Without legislation, every safety advance is effectively optional— a recommendation companies can take or leave depending on competitive pressure.

Legislation is what makes technical safety advances mandatory for frontier labs. It’s also what buys time. Every regulation that imposes a meaningful evaluation requirement, every disclosure mandate, every liability framework, every whistleblower protection policy slows the race and gives safety researchers more runway to solve the problems we don’t yet know how to solve.

The Pitch: Same Money, Different Structure, More Impact

While we have tens of millions of dollars pouring into major organizations in the c3 space, we don’t have nearly enough funding going towards political AI safety work. The gap between funding for research and funding for advocacy is enormous.

As I wrote about previously, part of this is due to the concentration problem: currently, the vast majority of AI safety funding flows through a handful of institutional funders. Some of those funders (including the vast majority of private foundations,3 most DAFs, and corporate matching programs) are structurally barred from significant C4 giving.

Others, including CG, can technically fund C4s but face reputational constraints that make them cautious about politically charged advocacy, especially work that opposes specific industry players. As a result, the entire field has inherited a bias toward C3 research over C4 advocacy.

Individual donors face neither constraint. This means the people who can actually fix this gap are individual donors, who are the only actors in the ecosystem without structural or reputational barriers to C4 giving.

Some numbers

A top-earning donor giving $1M to a c3 actually spends ~$650K after the federal deduction (capped at 35% under the One Big Beautiful Bill Act, effective 2026).

This means that $650K, given directly to a c4, represents an equivalent personal cost with arguably greater counterfactual impact.

The Formula

C4-equivalent = X − [(X − floor) × effective deduction rate]

Where:

• X = intended c3 gift

• floor = 0.5% of Adjusted Gross Income (a new rule under the One Big Beautiful Bill Act says you can’t deduct first 0.5% of AGI donated to c3 organizations)

• effective deduction rate = at a federal level, capped at 35% for top-bracket donors

  • Important: State taxes may decrease the C4 equivalent, especially in higher tax states

So, for a donor with $5M AGI giving $1M to a c3, these are roughly the numbers:

• Floor = $25,000 (non-deductible portion of c3 gift)

• Deductible portion = $975,000

• Tax savings = $975,000 × 35% = $341,250

• Net personal cost of c3 donation, which would represent the equivalent that should be donated to a c4 = $658,750

The floor: important to know for mid-tier donors

Starting in 2026, the first 0.5% of a donor’s AGI in charitable giving confers zero federal tax benefit under the One Big Beautiful Bill Act’s new deduction floor.

For a donor whose total c3 giving falls at or below that floor, c3 and c4 donations will cost exactly the same, because the amount you’re actually donating is equal to your personal cost (there is no deduction). In this instance, the tax case for c3 donations disappears entirely,4 and c4 wins on any positive impact multiplier.

As an example, let’s say we have an Anthropic engineer with $3M AGI who gives $15K to AI safety. The floor is $15K, so zero of that engineer’s $15K gift is deductible, meaning c3 and c4 are tax-identical.

Caveats

I am not your financial advisor and would recommend discussing your specific case with a qualified CPA. For example, the above math doesn’t apply if you’re planning on donating stocks directly, because of capital gains tax (it’s complicated).

Specific Funding Recommendations

The orgs below take different approaches. Some focus on catastrophic risk directly, others on adjacent concerns like child safety, worker protection, or chip security. Read the descriptions and pick what fits your priorities.

Edit: Since publishing this post, I've removed a couple of organizations from this list. If you'd like my updated recommendations, please reach out to me directly.

ControlAI

• What: ControlAI is an organization that “campaign[s] on preventing extinction risk.” They have briefed 250+ lawmakers across the US, UK, Canada, and Germany.

  • They’ve also collaborated with TIME, The Guardian, Sci Show, and other content creators to produce educational content regarding the risks of superintelligence.

• Contact to donate: hello@controlai.com

Center for AI Safety Action Fund

• What: CAIS’s Action Fund has four primary policy priorities: (1) ensuring the U.S. retains its competitive advantage over China as it relates to AI chip manufacturing, (2) implementing and enforcing export controls, (3) preventing bad actors from using AI to cause harm, and (4) advocating for international cooperation to advance safe AI.

  • They spent $310,000 on lobbying in 2025 and work closely with CAIS’s D.C. branch.

• Link to donate: https://safe.ai/donate

AI Policy Network

• What: A bipartisan organization that lobbies Congress on “establishing effective guardrails, and ensuring the United States remains both dominant and safe.” Some of their work focuses on loss of control risk, and they are one of the only registered lobbying entities that speaks about superintelligence directly to the national security establishment.

• Link to donate: https://theaipn.org/support-us/

Public First Action

• What: A bipartisan advocacy organization focused on child safety, supporting workers, and safeguards to protect the general public. Also does education/awareness-type work on AI risk. Seeded with a $20m donation from Anthropic.5

• Link to donate: here

A note about PACs and political campaigns

Some practitioners in the AI safety policy space believe donations to politicians and PACs are meaningfully more impactful than the best C4s. Zach Stein-Perlman, who advises donors in this area, has estimated roughly “5x as impactful.”

I haven’t independently verified this, but I think it could be true, depending on assumptions surrounding timing, electoral leverage, and the specific donation opportunities Zach might point you towards. (Also worth flagging: Meta is spending $65M this year on super PACs designed to elect AI-friendly state officials— the biggest election investment in the company's history. That’s a pretty good sign this kind of giving works.)

However, there are trade-offs: PAC contributions are publicly disclosed, more legally complicated, require US permanent residency, and there are contribution limits; there might also be reputational risk for recipient political candidates depending on who you are (e.g. see Carrick Flynn case study; I think being backed by SBF almost certainly hurt his odds).

That said, if you’re considering donating $20K+ and willing to handle the complexity and public disclosure, PACs may be worth exploring. Zach has offered to advise on this (more on how to reach him below).

For most readers, the C4 case I’ve made above is probably still the right starting point.

Next Steps

The AI safety field is flush with money, yet starving for advocacy. The next billion dollars to CG or a DAF probably won’t solve this problem.

If you care about AI safety and you were planning to donate this year: before defaulting to a c3, run the math above and seriously consider whether a c4 gift at equivalent personal cost would do more.

If you want to talk this through before giving, here are some options:

• Reach out to me directly at kairos@stanford.edu. I’m happy to think through your situation and point you toward people in the ecosystem whose judgment I trust for your specific case.

• Contact Zach Stein-Perlman on the EA Forum, who has offered to advise US permanent residents considering $20K+ donations to politicians and PACs, which he estimates could be roughly 5x more impactful than the best C4s.6

Pick whoever fits. Different people are better positioned for different donor profiles.

Conclusion

This is an opportunity to maximize your counterfactual impact in a way that almost no other giving can. Giving well is hard. It takes research, judgment, and time. It would be easier to park money in a DAF and forget about it, or forward donations to whoever’s most legible, or give to people you like, or just not give at all.

But the political work that needs funding right now is work that most institutional funders can’t or won’t touch. The dollars sitting in your account are structurally different from every other dollar in the ecosystem, and deploying them well might matter more than any other decision you make this year.

Thank you to Jack Douglass for helpful discussion & feedback.