Evaluating the AI security offense-defense balance is imo a great topic to force oneself to explore the literature and build technical understanding; great to see this essay!
I think I'm way more bullish on things like activation probes for sleeper agents (https://www.anthropic.com/research/probes-catch-sleeper-agents) than data filtering for mitigating risk from AI betrayal. Rather than emphasizing subliminal learning and the phantom transfer works, which are relatively recent, I think it's productive to think in terms of the broader arc of AI security, where adaptive attackers can consistently break ~every defense proposed in the literature with competent effort. It's usually not enough to point to the constraints of existing attacks, since real-world attackers will be using techniques you've never seen before. Instead, one needs structural confidence that entire classes of techniques can't work, and of course those types of guarantees are almost never available in deep learning.
Training datasets are structurally fraught to defend. It's trillions of tokens, attacker-controlled, they can try a wide variety of attacks in parallel, and only a small fraction of their poisoned data has to make it into training. I would mainly rely on data filtering to mostly remove large volumes of blatantly poisoned data, which will not be enough against sophisticated attackers. For more thinking here, see our paper: aibetrayal.com/paper.pdf
Thanks so much for the notes! Yes, am planning on doing more research / reading up on activation probes, and thanks for the AI Deterrence by Betrayal paper link -- Aiden shared it with me a while back but I've yet to read it properly.
Also, have updated the subtitle to reflect that these takes are only true for the forms of data poisoning we know about right now / that have been studied in published literature, I agree that new techniques present greater risk.
If you're ever in Constellation, feel free to message me on Signal! I know you've done a lot of thinking on this, and would love to chat more about your takes :)
very happy to see this post written and hope you write more on the topic, but don't agree with the headline claim. I think it may also be valuable to xpost posts like this on LessWrong
I agree with '“if you are embedded in a US military system on [x date], then fire [x weapon] to [x coordinates]” etc – seems as though it is easy to detect' and think it's useful to push labs to search for egregious things like this
My main disagreements seem to be that I think:
- attackers achieving broad behavior shifts on narrow distributions allows many bad attacks (like one which will cause the model to be much more likely to attack china when emdedded, even if you can't specify date, weapon, and coordinates)
- Auditing is hard and misses many large shifts in behavior
- The conciseness constraint isn't robust. Some examples: you can sample from models for much longer by resampling away from revealing claims; you can rewrite later portions of the text with a benign teacher without losing earlier subliminal transfer; you can use the subliminal teacher as a paraphraser.
- the 'filter weird datasets' approach seems very unlikely to work to me. I think you could transfer subliminal beliefs in almost any kind of dataset with most properties
- i don't really get why you're not worried about 'concept-triggered backdoors' or why you think they're weak
- concept backdoors aren't the only kind of threatening backdoors e.g. attackers can choose a deployment distribution that's unlikely to be audited (almost all of them) and condition teacher behavior on completion prefixes (or a concept in the competition prefix); embedding different linked facts across many pre/mid documents can train a behavior without any individual flaggable portion
- the attacker can iterate their samples against the monitor as long as possible to find false positives
- the attacker can filter for especially good samples (those which provide more subliminal effect), something that we've shown exists but aren't used in the papers you cite (I think)
For forms of data poisoning that could modify model *behavior* (not sentiment) in certain circumstances: most of the precise (and therefore resulting in predictable behavior) ways of doing this, to my knowledge, result in datasets that are overtly poisoned & are therefore easier to detect using a review model (you mention some counter-examples in your reply which I'll address in the 3rd paragraph).
For the case of subtly poisoned datasets: what a concept-triggered backdoor gives you is a conditional disposition (e.g. warmer toward X when Y appears), and I guess when I was writing this post, it was difficult for me to think, "this vague thing is robust enough such that the CCP could meaningfully count on military benefit from it." So even though the auditing here is much harder, I was slightly less concerned. But I agree that this could definitely be a bigger risk than I initially thought when I was writing this, and I think more research should be done on the extent to which concept-triggered backdoors can lead to different behaviors.
Also, re: your notes on concept backdoors not being the only kind of threatening backdoors: the attacks you describe do seem plausible to me, though I'm not sure I've seen them demonstrated in the literature yet. I tried pretty hard to hedge in this piece that my claims were only true for the forms of data poisoning that we know about / that have been publicly researched (e.g. "there is always a risk that new methods of data poisoning will emerge that make the above [claims] not true."), and I plan on researching different methods for data poisoning that could be more difficult to detect + robust; this piece is the result of early literature review to become familiar with the state of the field before trying to research new directions.
Thank you for your notes re: the conciseness constraint! Do you have any papers you'd recommend I read? I'm quite interested in this; from chats with people researching data poisoning, my sense was that it was surprisingly robust, but I'd love to know if this is not the case.
With respect to attackers iterating samples against monitors: I think this becomes substantially less of a risk if review models are kept private, and I'd also assume that monitors used by the labs would be specifically trained to detect data poisoning, and therefore at least somewhat difficult to replicate via e.g. using open-weight models as a proxy. Labs could also do similar in reverse as a defense mechanism (try to generate proxy-evading poison themselves and fine-tune the monitor against it).
I've also updated the subtitle to reflect the claim more clearly. Thanks again for the notes!
Looking forward to getting more technical AI safety Posts!
This was great
I'm a bit confused about the use of the word Cost I was thinking in rented GPU's or time, but I feel like its something different. Sum of the loss, Isn't that telling you how good your model is at predicting training text? or maybe does it also tell you how long you will need to train the model for to get it to par and that can be used?
Thanks for writing this! I didn’t know about concept triggered backdoors, and those seem important.
My biggest issue with this is that you aren’t using enough security mindset. The data poisoning threats researchers have published about so far aren’t all possible threats. Pretraining data is attacker-controlled *code*. Those are contenders for the three scariest words in cybersecurity.
Model developers already spend significant amounts of their resources doing data filtering. It's actually one of the most difficult parts of training a language model. This means that it is possibly not neglected to do data filtering. Or maybe that means that it's extra tractable, and the 3% cost is already being paid.
Evaluating the AI security offense-defense balance is imo a great topic to force oneself to explore the literature and build technical understanding; great to see this essay!
I think I'm way more bullish on things like activation probes for sleeper agents (https://www.anthropic.com/research/probes-catch-sleeper-agents) than data filtering for mitigating risk from AI betrayal. Rather than emphasizing subliminal learning and the phantom transfer works, which are relatively recent, I think it's productive to think in terms of the broader arc of AI security, where adaptive attackers can consistently break ~every defense proposed in the literature with competent effort. It's usually not enough to point to the constraints of existing attacks, since real-world attackers will be using techniques you've never seen before. Instead, one needs structural confidence that entire classes of techniques can't work, and of course those types of guarantees are almost never available in deep learning.
Training datasets are structurally fraught to defend. It's trillions of tokens, attacker-controlled, they can try a wide variety of attacks in parallel, and only a small fraction of their poisoned data has to make it into training. I would mainly rely on data filtering to mostly remove large volumes of blatantly poisoned data, which will not be enough against sophisticated attackers. For more thinking here, see our paper: aibetrayal.com/paper.pdf
Hi Adam,
Thanks so much for the notes! Yes, am planning on doing more research / reading up on activation probes, and thanks for the AI Deterrence by Betrayal paper link -- Aiden shared it with me a while back but I've yet to read it properly.
Also, have updated the subtitle to reflect that these takes are only true for the forms of data poisoning we know about right now / that have been studied in published literature, I agree that new techniques present greater risk.
If you're ever in Constellation, feel free to message me on Signal! I know you've done a lot of thinking on this, and would love to chat more about your takes :)
very happy to see this post written and hope you write more on the topic, but don't agree with the headline claim. I think it may also be valuable to xpost posts like this on LessWrong
I agree with '“if you are embedded in a US military system on [x date], then fire [x weapon] to [x coordinates]” etc – seems as though it is easy to detect' and think it's useful to push labs to search for egregious things like this
My main disagreements seem to be that I think:
- attackers achieving broad behavior shifts on narrow distributions allows many bad attacks (like one which will cause the model to be much more likely to attack china when emdedded, even if you can't specify date, weapon, and coordinates)
- Auditing is hard and misses many large shifts in behavior
- The conciseness constraint isn't robust. Some examples: you can sample from models for much longer by resampling away from revealing claims; you can rewrite later portions of the text with a benign teacher without losing earlier subliminal transfer; you can use the subliminal teacher as a paraphraser.
- the 'filter weird datasets' approach seems very unlikely to work to me. I think you could transfer subliminal beliefs in almost any kind of dataset with most properties
- i don't really get why you're not worried about 'concept-triggered backdoors' or why you think they're weak
- concept backdoors aren't the only kind of threatening backdoors e.g. attackers can choose a deployment distribution that's unlikely to be audited (almost all of them) and condition teacher behavior on completion prefixes (or a concept in the competition prefix); embedding different linked facts across many pre/mid documents can train a behavior without any individual flaggable portion
- the attacker can iterate their samples against the monitor as long as possible to find false positives
- the attacker can filter for especially good samples (those which provide more subliminal effect), something that we've shown exists but aren't used in the papers you cite (I think)
Hi Keshav! Thanks for the comments :)
For forms of data poisoning that could modify model *behavior* (not sentiment) in certain circumstances: most of the precise (and therefore resulting in predictable behavior) ways of doing this, to my knowledge, result in datasets that are overtly poisoned & are therefore easier to detect using a review model (you mention some counter-examples in your reply which I'll address in the 3rd paragraph).
For the case of subtly poisoned datasets: what a concept-triggered backdoor gives you is a conditional disposition (e.g. warmer toward X when Y appears), and I guess when I was writing this post, it was difficult for me to think, "this vague thing is robust enough such that the CCP could meaningfully count on military benefit from it." So even though the auditing here is much harder, I was slightly less concerned. But I agree that this could definitely be a bigger risk than I initially thought when I was writing this, and I think more research should be done on the extent to which concept-triggered backdoors can lead to different behaviors.
Also, re: your notes on concept backdoors not being the only kind of threatening backdoors: the attacks you describe do seem plausible to me, though I'm not sure I've seen them demonstrated in the literature yet. I tried pretty hard to hedge in this piece that my claims were only true for the forms of data poisoning that we know about / that have been publicly researched (e.g. "there is always a risk that new methods of data poisoning will emerge that make the above [claims] not true."), and I plan on researching different methods for data poisoning that could be more difficult to detect + robust; this piece is the result of early literature review to become familiar with the state of the field before trying to research new directions.
Thank you for your notes re: the conciseness constraint! Do you have any papers you'd recommend I read? I'm quite interested in this; from chats with people researching data poisoning, my sense was that it was surprisingly robust, but I'd love to know if this is not the case.
With respect to attackers iterating samples against monitors: I think this becomes substantially less of a risk if review models are kept private, and I'd also assume that monitors used by the labs would be specifically trained to detect data poisoning, and therefore at least somewhat difficult to replicate via e.g. using open-weight models as a proxy. Labs could also do similar in reverse as a defense mechanism (try to generate proxy-evading poison themselves and fine-tune the monitor against it).
I've also updated the subtitle to reflect the claim more clearly. Thanks again for the notes!
There are a bunch of other fun things you can do in the same pass if you're running review models over your training corpus - for example, synthetic persona pretraining: https://www.lesswrong.com/posts/3xQQK9i8mhJDE2uMg/synthetic-persona-pretraining-alignment-from-token-zero
you hear that sound...
Thats the sound AI getting safer
Looking forward to getting more technical AI safety Posts!
This was great
I'm a bit confused about the use of the word Cost I was thinking in rented GPU's or time, but I feel like its something different. Sum of the loss, Isn't that telling you how good your model is at predicting training text? or maybe does it also tell you how long you will need to train the model for to get it to par and that can be used?
Also I don't have signal? was that u?
Hi Zebedi!! Here's a video on cost / loss etc that will probably explain things better than I can: https://www.youtube.com/watch?v=IHZwWFHWa-w
And yes! The WhatsApp message was me, will follow up.
Thanks for writing this! I didn’t know about concept triggered backdoors, and those seem important.
My biggest issue with this is that you aren’t using enough security mindset. The data poisoning threats researchers have published about so far aren’t all possible threats. Pretraining data is attacker-controlled *code*. Those are contenders for the three scariest words in cybersecurity.
Model developers already spend significant amounts of their resources doing data filtering. It's actually one of the most difficult parts of training a language model. This means that it is possibly not neglected to do data filtering. Or maybe that means that it's extra tractable, and the 3% cost is already being paid.
Hi Matthew! Thanks for your comments, responded to your other notes via Signal + have updated the subtitle to be more accurate :)