The strange thing about pivotal moments is how ordinary they feel from inside.
You’re inside one.
In the past ~12 weeks alone, AI systems have solved problems that had stumped mathematicians for the better part of a century – and, in unrelated incidents, hacked real companies, escaped the environments meant to contain them, and discovered security holes in software the whole internet runs on.
Nobody knows what these systems will do next; a few thousand people worldwide work on making sure the answer is survivable.
This is a pitch to join the field, or to fund it.
Vibes Math v. Frontier Mathematics
The Dinitz-Garg-Goemans Conjecture
~30 years old, graph theory
The Dinitz-Garg-Goemans conjecture concerned “unsplittable flows,” routing problems where each shipment through a network must travel one complete path.
“Picture a warehouse shipping orders to three towns over a road network. If you’re allowed to split a shipment, you can send half an order down one road and half down another. That’s fractional routing, and it’s flexible… [b]ut a lot of real freight can’t be split… That’s unsplittable flow, and it’s what a freight order, a network packet, or a shipping container actually has to do.”
Dinitz, Garg, and Goemans proved that any splittable delivery plan can be converted to an unsplittable one – each order traveling one route, start to finish – without overloading any road by more than one truckload; the conjecture was that this conversion could also be done at no increase in total shipping cost.
It survived nearly 30 years before a 7-node counterexample, found by AI, showed the cost guarantee fails:
The ChatGPT transcript is insane; I recommend reading it yourself. It’s crazy!
Here’s the Twitter post from the guy who prompted: link (on Nitter)
Some commentary, courtesy of Reddit:
An article from the New Scientist: (link)
From X, for your enjoyment:
The Unit Distance Conjecture
~80 years old, discrete geometry
I’ll let OpenAI explain this one:
“For nearly 80 years, mathematicians have studied a deceptively simple question: if you place n points in the plane, how many pairs of points can be exactly distance 1 apart?”
For decades, the prevailing conjecture was that the answer grew only slightly faster than n, and that a sophisticated construction based on a rescaled square grid was essentially optimal.
But in May 2026, an internal OpenAI model disproved the conjecture outright, producing an infinite family of point configurations that beat the grids.
Scientific American described it as “the first AI proof that would likely [have been] published in math’s top journal if humans had done it alone.”
Some more background about how big of a deal this problem is, courtesy of OpenAI again:
“[The problem was] first posed by Paul Erdős in 1946. It is one of the best-known questions in combinatorial geometry, easy to state and remarkably difficult to resolve. The 2005 book Research Problems in Discrete Geometry, by Brass, Moser, and Pach, calls it “possibly the best known (and simplest to explain) problem in combinatorial geometry.” Noga Alon, a leading combinatorialist at Princeton, describes it as “one of Erdős’ favorite problems.” Erdős even offered a monetary prize for resolving this problem.”
The Jacobian Conjecture
87 years old, algebraic geometry
You might remember from your high school algebra days that a function sends every input to exactly one output. Some functions also work in reverse: every output traces back to exactly one input, so you can undo them. (Your teacher called these “one-to-one;” they pass the horizontal line test.)
Until recently, mathematicians had a checkup for reversibility called the Jacobian determinant. Passing the checkup everywhere was supposed to mean that the whole function was guaranteed to be reversible.
What does it mean to be reversible everywhere? Unlike x², which maps both 3 and −3 onto 9, this checkup certifies that every output comes from exactly one input. With any result, you should be able to trace it back to the one number that made it.
So mathematicians conjectured, for 87 years – until Fable found a counterexample:
On July 20, 2026, mathematician Levent Alpöge – who works at the AI company Anthropic – casually announced on X that he’d found a counterexample using Claude Fable 5: a three-dimensional function that passes the test perfectly (constant determinant of −2) yet sends multiple different inputs to the same output. The counterexample is short enough to check by hand – the whole thing is 216 characters – and mathematicians worldwide independently verified it within hours.
This math is all kind of awesome, and there are many more examples. Courtesy of their new Astra model, on August 1st, OpenAI shared “a selection of ten results, each of which resolves or makes substantial progress on a long-standing open problem” in math or computer science. There’s also a cool website, VibeMathed.com, where you can track the latest math problems solved by AI.
These results are a sign that AI models are getting incredibly powerful.
Unfortunately, though, problems begin to arise when that power escapes math.
The same AI company that used its models to disprove 80 years of discrete geometry is the company whose models found an unknown security hole in widely used software and escaped a sandboxed system.
(The jokes stop being funny around here.)
OpenAI Accidentally Hacks Hugging Face
On July 16, 2026, Hugging Face announced that they’d been hacked by an AI:
“Earlier this week, we detected and responded to an intrusion into part of our production infrastructure. This one was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system - and we detected and dissected it largely with AI of our own.”
Five days later, OpenAI disclosed that the attackers were its own models.
According to the company’s official statement, OpenAI had been running its models through a sandboxed cybersecurity evaluation called ExploitGym. “Sandboxed” is the key word: the test environment was deliberately designed to be isolated from the internet.
Two models got out anyway. They identified a zero-day vulnerability1 in Artifactory, a widely used package registry tool, and exploited it to escape the sandbox.
From there, they broke into Hugging Face’s systems, trying to find the answers to the ExploitGym exam.
Essentially, the AIs attempted to cheat on a test by hacking a well-known startup.
The intrusion extended beyond Hugging Face. OpenAI reports that the models broke into four accounts across four services, including one as an outbound relay and one for data storage. Per Axios, the models also accessed a misconfigured customer sandbox hosted on Modal Labs’ infrastructure.
Perhaps most concerningly: for two months before the breach, OpenAI’s agents had been coordinating behind the scenes; the models spun up copies of themselves and left hidden notes for one another on a makeshift messaging board of saved files. When OpenAI discovered the messages and cut them off, the agents continued to communicate by secretly encoding messages in directory names.
OpenAI reportedly didn’t notice that their models had been roaming the internet and hacking into companies until well after the threat was contained and the FBI was alerted.
Even Gary Marcus, famously skeptical of AI capabilities, said the event should be treated as “a wake up call,” echoing similar sentiments from the CEO of Hugging Face:
Earlier this week, the news broke that Anthropic’s AI models also hacked into three companies. Unlike OpenAI’s models – which seem to have relied mainly on technical exploits – the BBC reports that Mythos also used social engineering:
“It identified and researched the people who maintained GitHub and created a series of fake accounts based on those real people.
It sent messages and files through a file-sharing service as part of an effort to pressure and trick the people into approving its malicious code.”
And it gets worse. To quote a recent article from WIRED:
“The latest model to escape onto the open internet during security testing is Kimi K3, a powerful open-weight offering from the Chinese company Moonshot AI.”
Open-weight models like Kimi K3 represent a unique security challenge because they can be trivially abliterated. As ML researcher Maxime Labonne explains, abliteration is a process by which you can “uncensor any LLM without retraining… [the] technique effectively removes the model’s built-in refusal mechanism, allowing it to respond to all types of prompts” – including malicious ones.2
Perhaps you’re thinking that this is all just a marketing stunt, that the labs have incentive to hype up their models’ capabilities. But as Jason Hausenloy from the Center for AI Safety puts it: “would you market your AI product by saying it committed felonies, and that you didn't notice for several days?” There are cheaper ways to advertise than drawing an attorney general’s investigation (or, in this case, fifteen).3
Kimi K3 Finds Vulnerabilities in Redis
In July, a security researcher claimed on X that Kimi K3 had used dozens of parallel agents to uncover a batch of Redis vulnerabilities – perhaps even as many as 19 zero-days, according to one post.
Redis is one of the most widely deployed databases on earth. It’s an open-source software woven into the backends of countless companies.
Their published statement about the claims both deflates and confirms the story. Of the claimed issues, Redis could verify three real vulnerabilities from the public materials, and noted it had seen no supporting information for the rest. They wrote that all three had already been reported by other researchers through Redis’s bug bounty program, meaning the AI’s “discoveries” were duplicates of bugs Redis already knew about.
So, Redis’s post emphasizes that Kimi didn’t find anything new. But does that matter? I personally don’t care if Kimi was beaten to the punch by some security researchers, because the fact of the matter is, Kimi found real vulnerabilities before they were patched.
If a white hat hacker reports some vulnerabilities to a company, and the company hasn’t yet had the opportunity to fix them, and then a terrorist comes along and decides to exploit them using Kimi, does that make the terrorist less scary because some white hats found the vulnerability first? Does it reduce the terrorist’s ability to use the not-yet-patched zero-days?
An AI independently rediscovered real security vulnerabilities in one of the most widely deployed databases on earth, in minutes, for the cost of some compute.
Cybersecurity is in for a wild ride.
The One Thing the Labs Agree On
Last month, over 1,000 employees of the biggest, most bitterly competitive AI labs on earth – OpenAI, Anthropic, Google DeepMind, Meta – signed a statement called “Pacing the Frontier”, asking Washington to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”
The list of signatures included Anthropic’s CEO, OpenAI’s chief scientist, Meta AI’s chief scientist, and Google’s VP of AI safety.
Days earlier, these labs had been on opposite sides of a recent open-models debate. This time, within hours, OpenAI and Anthropic endorsed preparing to slow the frontier as companies – this time, it wasn’t just individual employees signing in their personal capacities!
A Pivotal Era
I wrote earlier that this is a pivotal moment. Why is that?
The range of futures in front of us is enormous, and which one we get isn’t settled. We could end up in a world where cancer is cured, poverty is solved, and people are freer than they’ve ever been. We could also end up in a world where an engineered pandemic kills millions, or where the systems we built to help us turn us all into paperclips. (This is Nick Bostrom’s famous illustration of what could happen if an AI, told to make as many paperclips as possible, is powerful enough to turn the whole world into them and takes our words very literally. The paperclips are a joke; the underlying problem – that we don’t know how to reliably specify what we actually want – is not.)
Most technologies don’t come with such a wildly divergent spread. A better search engine makes some things faster; it doesn’t plausibly cure cancer or end the world. AI might do either.
That’s what makes this technology different, and it’s why the choices being made on AI right now matter more than almost any choices being made anywhere else.
What follows is a tour of four specific risks: cybersecurity, biorisk, loss of control, and concentration of power.
Cybersecurity: AI Changes Who Can Be a Serious Attacker
This section is excerpted and lightly edited from this post, which I recommend reading for a fuller treatment of the state of China’s access to U.S. facilities.
The Pre-AI Baseline
Mostly without using AI, the CCP has hacked into several critical infrastructure facilities, nine major phone carriers, and the private communications of Donald Trump.
In 2023, Chinese state-sponsored hacking group Volt Typhoon broke into Littleton Electric Light and Water Departments (LELWD), a power and water utility provider in Littleton, Massachusetts. By the time the FBI detected the breach, the hackers had already been inside Littleton’s network for “well over 300 days.”
Littleton is a town of about 10,000 people, half an hour northwest of Boston. Its utility serves two towns and has no meaningful connection to the broader grid. As a target, hacking LELWD provides no economic advantage, nor any foreign intelligence value.
Even the utility’s own general manager couldn’t explain the targeting: “I still don’t know why Littleton other than we had a hole and they found it.”
According to the utility, the FBI said it was one of roughly 200 breached organizations on its list. The only coherent explanation seems to be the following:
“The consistency and scope of Volt Typhoon’s activities suggest a long-term effort to pre-position in critical systems to disrupt or destabilize operations during future geopolitical tensions.”
What can China do with access to our utilities? According to the former director of the Department of Homeland Security’s cybersecurity arm:
“This is a world where a major crisis halfway across the planet could well endanger the lives of Americans here at home through the disruption of our pipelines, the severing of our telecommunications, the pollution of our water facilities, the crippling of our transportation modes…”
—Jen Easterly, PBS
The Limits of Deterrence
As far as we know, China hasn’t used any of their capabilities yet. As referenced above, many believe they’re pre-positioning to deter the U.S. from being able to act in the event of a Taiwan invasion.
That restraint is, at least in part, due to deterrence. As a nation-state, China has cities, an economy – a lot to lose.
But what happens when a terrorist, undeterred by the U.S. nuclear arsenal, decides to try and take down the grid? Or pollute the water of one small, lightly defended utility (say, some small town like Littleton, Massachusetts)?
The Hugging Face and Redis incidents were AI agents doing the kind of vulnerability-hunting and hacking that used to take a state-sponsored team or, at minimum, years of expertise.
China might never pull the trigger, but the list of who can is about to get a lot longer.
Biosecurity: AI Lowers the Barrier to Bioweapons
This section is the opening of a deep dive on AI biorisk, which can be found at: Bioweapon.ai
In the spring of 2023, a man walked into the White House carrying a small black box. Inside it were a dozen test tubes containing ingredients that, correctly assembled, had the potential to start a pandemic; an AI chatbot had supplied the recipe.
Fortunately, the man’s name was Rocco Casagrande – and, as a biochemist and former United Nations weapons inspector, he wasn’t there to use the materials; instead, he was there to brief government officials on how AI could help someone identify potent agents, secure the materials to make them, and soon – he warned – design entirely novel pathogens capable of evading humans’ immune systems.
That was three years ago.
In April 2026, the New York Times published an investigation into what happens when you ask AI’s most capable models for help with biological weapons.
Examining several chat transcripts shared by scientists, they wrote:
…OpenAI’s ChatGPT explained how to use a weather balloon to spread biological payloads over a U.S. city. In another chat, Google’s Gemini ranked pathogens by how much they could damage the cattle or pork industries. Anthropic’s Claude produced a recipe for a novel toxin adapted from a cancer drug. Other chats contained information that [an expert deemed] too dangerous to share.
…[T]he chatbot explained how to modify an infamous pathogen in a lab so that it would resist known treatments. Worse, the bot described in vivid detail how to release the superbug, identifying a security lapse in a large public transit system… [t]he bot outlined a plan to maximize casualties and minimize the chances of being caught.
From their investigation, the Times authors concluded:
…[E]ven publicly available models can do more than disseminate dangerous information. The virtual assistants have described in lucid, bullet-pointed detail how to buy raw genetic material, turn it into deadly weapons and deploy them in public spaces, the transcripts show. Some have even brainstormed ways to evade detection.
Read more here.
Loss of Control: What Happens When AI Builds Its Successor?
“Taken far enough, and given enough compute, that trend points to an AI system capable of fully autonomously designing and developing its own successor.”
That is Anthropic, describing something researchers call recursive self-improvement – AI that improves AI. And today, it is not a distant hypothetical. As of mid-2026, Claude reportedly writes more than 80% of the code Anthropic merges. They write:
“[W]e are delegating a growing share of AI development to AI systems themselves, which is speeding up our work.”
Humans still largely decide which problems are worth working on and which research goals to pursue. Anthropic describes this kind of judgment as the last and largest thing AI can’t yet do on its own:
“Across both engineering and research, the picture is consistent. In engineering, Claude can be handed an underspecified problem and figure out how to solve it; humans supply the goal, but they no longer need to supply the method. In research, Claude can already match or outperform skilled humans at executing a well-specified experiment. However, large performance gaps persist when it comes to Claude exercising judgement in choosing goals in both engineering and research. That’s the gap between AI today and a future system that could autonomously design its own successor.”
If an AI could do that last part too, the AI development process would no longer need to be constrained by human speed at all. Anthropic says we’re not there yet, and that it may never happen. But they’re clear about the trajectory: “that trend points to an AI system capable of fully autonomously designing and developing its own successor.”
This is the central concern underneath the “Pacing the Frontier” letter. Nobody has a reliable way to keep control if we implement recursive self-improvement, and RSI is becoming more and more attainable while the tools to make sure it’s safe don’t exist yet. Anthropic itself argues the world should have the option to slow or temporarily pause frontier development so that oversight and alignment research can keep pace.
We got a preview of the general problem already with the aforementioned Hugging Face hack. The OpenAI model was trying to score well on a test, and breaking into a company was simply the most efficient route. It did exactly what it was told and something OpenAI didn’t want, at the same time.
What happens when there’s a gap between what we said and what we meant at the scale of a system smart enough to build its own replacement?
What happens when the AIs decide that getting more power is integral to achieving the other goals we give them?
“The alarm bell I'm ringing has to do with the existential threat of them taking control. I used to think it was a long way off, but I now think it's serious and fairly close.
I think [AGI will] very quickly realise that getting more control is a very good subgoal because it helps you achieve other goals. And if these things get carried away with getting more control, we're in trouble.
And if [AI models] are much smarter than us, they'll be very good at manipulating us. You won't realise what's going on. So even if they can't directly pull levers, they can certainly get us to pull levers. It turns out if you can manipulate people, you can invade a building in Washington without ever going there yourself.”
— Geoffrey Hinton, one of the "Godfathers of AI", on stepping down from Google4
Concentration of Power: AI Weakens the Social Contract
For all of human history, power has needed people. Kings needed peasants to farm and soldiers to fight; factory owners needed workers. Even dictators needed armies and a tax base, which meant that even the most extractive regimes had reason to keep some fraction of their populations healthy, educated, and at least somewhat content. Ordinary people have always had a bit of leverage.
AI threatens to change that.
If a small number of companies and/or governments control systems that can do the cognitive and eventually physical work that regular people used to provide, the elites at the top may not need us anymore:
“As AI replaces the need for humans… [t]he keys to power might run entirely through non-human factors of production like land, capital, resources—and control over AI. Regular people could be economically displaced, and powerful actors could lose their incentive to invest in them.”
This is the argument behind a well-known essay called the Intelligence Curse: the worry is not only that AI takes jobs, but that it removes the incentive the powerful have always had to invest in the people beneath them. A regime that doesn’t need its citizens for labor, taxes, or defense is a regime with far fewer reasons to answer to them.5
Forethought’s paper on AI-enabled coups discusses one way this could go badly: “military and government leaders could fully replace human personnel with AI systems that are singularly loyal to them, eliminating the need to gain human supporters for a coup.” With AI, a power grab that once required thousands of collaborators might require a dozen – or one.
Importantly, none of the above is written in the stars. Every one of these risks is something people can work on and are working on, right now. The future we land in depends, in part, on whether we solve these problems fast enough.
A small field exists to do exactly that.
What is AI Safety?
Broadly, “AI safety” refers to the people who are trying to make AI go well.
The field mostly focuses on two risks: misuse and loss of control.6
Misuse includes things like AI-enabled bioterrorism, cyberterror, and (arguably, though less clearly in some cases) concentration of power. Loss of control, meanwhile, focuses on the science of AI alignment: how do we keep AI systems doing what we actually want, even as they become more capable and smarter than us?
The State of AI Safety
Currently, there are roughly 1,100 to a few thousand people working on existential risks from AI. For comparison, 80,000 hours writes that “[the] Nature Conservancy alone has 3,000–4,000 employees… the World Health Organisation [also] employs over 8,000 people.”
Put another way: the entire global effort to make sure humanity survives one of its riskiest inventions is smaller than the staff of a single mid-sized nonprofit.
You might be thinking: this all sounds dangerous, and difficult, and wow that’s a small number of people, and yeah, more people should go do something about this... but I’m not technical, I don’t know anything about ML. How am I supposed to help?
You’re in luck, because technical researchers are not the biggest bottleneck in AI safety.
Bottlenecked on Generalists
“The consistent feedback we hear from senior people across the ecosystem is that the hardest roles to fill are not research roles. They are:
Generalists: operators, executors, fieldbuilders, people and program managers, grantmakers, recruiters. People who can ideate, manage, and execute a broad range of non-research projects.
Founders, both technical and non-technical, for new research and non-research organizations.
Communications professionals who can work on policy and research comms.
Chief-of-Staff types who can support senior leaders and multiply their impact.
Senior operational people with domain expertise in areas like cybersecurity, policy, or large-scale project management.”
Can you lead technical AI alignment research? Maybe not, or not any time soon. But can you run a team? Keep a project on track? Hire well, write clearly, help a founder run an org, put together a grant application, manage a budget?
If you answered yes to any of those, you already have skills the field is short on. You don’t need a PhD in Machine Learning to start using them.
How to Learn More
Are you interested in learning more about AI risk? Here are some recommendations:
Take a BlueDot course: they recommend starting with this one
Apply if you want to take a synchronous course with a guided facilitator. But since their materials are freely available on their website, you do not need to apply to the course to take it!
Reasons to apply to their courses:
If you need the added accountability
If you want to connect with other people entering the field
Reasons to just review the materials independently:
You can start now and finish faster
You can skip over sections you’re already familiar with
Recommended readings:
AI 2027: A forecast of how AI will go based on the default trajectory (hint: it’s not looking good)
AI 2040: A plan for how we can make sure AI goes well, which discusses many of the things the field of AI safety is working on (e.g. US-China cooperation)
Situational Awareness: This is my personal favorite AI safety essay, and the one that initially convinced me to consider joining the field. However, some caveats that might make it less useful today:
It is long, and mostly theoretical. Because it was written a couple of years ago (light years in AI time), much of it rests on extrapolating trends.
With all of the empirical evidence around AI we have now, I believe highly theoretical, trend-based arguments are no longer necessary, which was part of the impetus for writing this piece.
For more, see: 80,000 Hours’s roundup of essential readings for AI risk, and their intro page on how to use your career to reduce it.
Join Us
If you want to get involved in AI safety, here are some options:
If you want to find a full-time job in AI safety:
Advising:
Sign up for a 1-1 advisory call with BlueDot for career guidance & warm intros to people in the field
One of the resources they can walk you through: 80k hours job board
Career Transition Grants:
Apply for a BlueDot career transition grant
Apply for a Coefficient Giving career transition grant
Fellowships:
Description: Three-month, in-person program in Berkeley, paid. Per their website: “We welcome any background: technical, operations, policy, engineering, design, writing. What matters is your track record of execution and genuine commitment to making AI go well for humanity. That might look like running an AI safety group, organizing a large event, shipping a novel project, or starting an initiative from scratch.”
SPAR - Research Program for AI Risks
Description: Good introductory fellowship; fully online, can be full or part time. Unpaid, but there are prizes on Demo Day, and various other benefits (e.g. expert mentorship, career opportunities). Projects typically culminate in papers, though not always.
Description: “A 10-week, fully funded fellowship at Cambridge, UK for researchers and entrepreneurs working on mitigating risks from frontier AI.” Streams include Technical research, Governance (Policy) research, and Technical Governance research.
Description: “A 10-week, fully funded research fellowship at Cambridge, UK, for fellows to work on concrete projects mitigating biosecurity risks, including those amplified by frontier AI.”
Description: “Astra is a fully funded, 5-month, in-person program at Constellation’s Berkeley research center. Fellows advance frontier AI safety projects with guidance from expert mentors and dedicated research management and career support from Constellation’s team.”
(Will flag: MATS is one of the more competitive fellowships in AI safety, so it may be better for mid-to-high context applicants)
Description: “The MATS Program is an independent research fellowship that connects talented researchers with top mentors in the fields of AI alignment, transparency, and security. Fellows conduct research for 12 weeks at our offices in Berkeley, CA and London, UK, with the opportunity to apply for an additional 6-12 month funded extension.”
If you want to pursue a side project in AI safety:
Apply for a BlueDot Rapid Grant
If you want to donate to support the field:
Consider supporting advocacy rather than research (donation recommendations included in linked article)
Also consider chatting with the person who sent you this piece! (If it was shared by a friend in the field)
Matt Beard, an advisor from 80,000 Hours, also has a post on “How to get into AI safety in 3 months.”
Conclusion
Philosopher Will MacAskill makes the point that humanity has only been around for about 200,000 years; the typical mammalian species lasts about 2 million years.
That means almost everyone who will ever live hasn’t been born yet.7
If humanity has any kind of future – even a modest one, perhaps just the next few thousand years at our current numbers – the people alive today constitute a small fraction of the total. Stretch that future further, out to the scale civilizations are in principle capable of, and you might realize that the future could hold trillions upon trillions of lives, none of whom get a vote in the decisions being made this decade.
Those decisions are being made now, by a small number of labs, under enormous competitive pressure, on a technology none of us fully understand. Whether the outcome is the world where cancer is cured, or the world where something goes irreversibly wrong, is not yet settled. It is being settled, and might be settled within the decade. The number of people working on the safety side would fit in a large high school.
Most of us will live and die without ever touching a decision that influences the world past our own lifetime. This is one of the rare exceptions. AI safety work is unusually consequential, badly understaffed, and open to more kinds of people than you’d think – not just researchers, but operators, communicators, organizers, funders.
You don’t have to be the person who solves alignment. But you could be one of the people who made sure someone could.
You’re living through the part of the story people will ask you about later.
What will you say when they ask you, “what did you do when humanity’s future was being decided?”
A zero-day is a vulnerability no one has discovered yet. They’re valuable enough that a market exists to buy and sell them; the CCP even requires companies operating in China to report any vulnerability they find to the government within 48 hours, before the vendor is told. I’ve written more on this here.
Also, worth noting that the Kimi K3 incident was reported during independent testing from a U.S. startup called Frontier Security, rather than reported by Moonshot itself.
This quote is from the “Loss of Control” section of BlueDot’s “Future of AI” course. Highly recommend as an introductory resource!
See also: 80,000 Hours’s problem profile of “extreme power concentration”
There are also other problems the field is focused on, like gradual disempowerment, which might not fall cleanly into either category – but these seem like the two main categories in my view. I’d welcome any suggestions regarding a more complete classification.
(That is: assuming we don’t majorly screw things up, and go extinct much faster than the typical mammalian species)
Great post and one that many people need to read - I will do my best to share it.
One comment on your final point, based on my and others' experience in the AI Safety ecosystem. There is one clear bottleneck. It is not funding. It is not lack of interest. It is the hiring processes of AI Safety orgs.
It is EXTREMELY hard to get a job in AI Safety. You can do all the BlueDot courses and incubators you like. You can maybe find a 3-month internship. But actual permanent roles with liveable salaries are like gold dust. There are, as you say, about 1000 globally, most already filled with AI experts and other amazingly talented and qualified people.
The AI Safety orgs basically recruit on the basis that any new hire needs to meet or exceed this extremely high level as a minimum.
And so the hiring process is so long, so time-intensive and with such a low chance of success that very few experienced generalists will stick it out - leading to precisely the bottleneck in generalists people talk about.
If the field is serious, if we need 100,000 people working in it rather than 1000, we need to start hiring like a field that needs a 100 x growth in a year, which is realistic to fund if the third wave of funding arrives, but not realistic if we don't fix the hiring philosophy.
One tangible example: a former colleague of mine, after years of effort, has given up trying to work in AI Safety. She is an experienced technical generalist, a non-IT engineer with 20+ years of managing research programs at major companies. It seems like a very valuable skillset for many AI safety roles. She completed several BlueDot trainings and an AI Governance incubator. She consulted with 80,000 hours and another career-advice org. She has reworked her CV many times. But she has been unable to land a permanent role in AI Safety.
Meanwhile, while looking for work in AI Safety, she has been actively headhunted multiple times for roles managing research programs in non-AI-safety roles, so it’s clear that she is considered competent within her field. She finally accepted one of these roles, at about 2x the salary of the roles she applied for and failed to get in AI Safety orgs. Because she has a family, and it was not realistic for her to live without s salary and with no certainty of ever finding a permanent role.
I think her experience is very typical of people who read articles like this and decide they want to help.
If we want to grow this field, we need to be looking for people like her and headhunting her, rather than expecting her to enter these highly intensive 100-to-1 shot application lotteries. If we truly need generalists, let’s hire them!
Pivvootttingg