Pick Your Poison: Learning to Select Poison Sets for Stronger LLM Backdoor Attacks
Aashiq Muhamed, Mona T. Diab, Virginia Smith, Andrew Ilyas, Matthew Jagielski
Carnegie Mellon University · Anthropic
Published September 14, 2026 · on the feed September 15, 2026 · ▲ 8 · arXiv 2609.15029
The abstract
Backdoor poisoning attacks add poisoned examples to otherwise-clean finetuning data, pairing a trigger with a target behavior that the model learns to produce when the trigger appears. Existing evaluations typically fix the number of poisoned examples and sample them at random from a candidate pool. We show that this can severely underestimate worst-case vulnerability: across three LLaMA-3-8B backdoor settings, holding the model, clean data, and poison count fixed, attack success ranges from 3% to 80% depending only on which poison set is chosen. We formalize poison selection as oracle-budgeted set optimization and introduce SAILS (Set-level Audit-Informed Iterative Learned Selection), which learns a set scorer from a few hundred finetune-and-evaluate runs, ranks millions of candidate sets, and audits only a small shortlist. SAILS improves held-out attack success by 30 percentage points on average over the strongest influence baselines, transfers from small-scale to full-scale finetuning, and extends to code-generation, agentic, and API-only backdoors.
The Brief
2:06A short podcast summary of the paper. Written and voiced by AI.
Transcript
Pick Your Poison. Learning to Select Poison Sets for Stronger LLM Backdoor Attacks. By Aashiq Muhamed and four others, from Carnegie Mellon University and Anthropic. Published on the fourteenth of September. Its code repository is worked out: about forty-five source files, with a readme and a licence.
A backdoor poisoning attack adds poisoned examples to otherwise clean finetuning data. Each pairs a trigger with a target behavior, and the model learns to produce that behavior whenever the trigger appears. Existing evaluations typically fix the number of poisoned examples and sample them at random from a candidate pool.
The authors show that this can severely underestimate worst-case vulnerability. In three LLaMA-3-8B backdoor settings, with the model, the clean data and the poison count all held fixed, attack success ranges from three percent to eighty percent, depending only on which poison set is chosen. A random draw could land anywhere in that range.
If the choice of set matters that much, choosing it becomes the problem. The authors formalize poison selection as oracle-budgeted set optimization, and introduce SAILS, short for Set-level Audit-Informed Iterative Learned Selection. The set, not only its size, is the variable.
SAILS learns a set scorer from a few hundred finetune-and-evaluate runs. It uses that scorer to rank millions of candidate sets, and then audits only a small shortlist. The expensive step, actually finetuning and evaluating, is spent on a few hundred runs and a shortlist, while the cheap learned scorer handles the millions. In effect the scorer stands in for the experiment, and the experiment is kept for confirmation.
SAILS improves held-out attack success by thirty percentage points on average over the strongest influence baselines. It transfers from small-scale to full- scale finetuning, and it extends beyond the original settings to code- generation, agentic, and API-only backdoors.
The point is aimed at evaluation as much as at attack. A random poison set measures an average case, and the spread from three to eighty percent shows how far that can sit from the worst one. Searching for the worst set is how the authors get closer to it.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Same model, same data, same number of poisoned examples, wildly different outcomes. Start with that puzzle.
You audit a language model for poisoning risk. An attacker slips a few examples into finetuning data, each pairing a trigger with a target behavior. The usual test fixes how many and picks them at random.
The road ahead
- 1 · The problem ~7 min When an attacker poisons finetuning data, which examples they choose can matter enormously, and testing even one choice costs a full finetune.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Train a cheap learned scorer on a few hundred real runs to judge whole poison sets, rank millions with it, then audit a shortlist for real.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Real runs label some sets, a scorer learns from them, the scorer ranks millions of sets, and real runs audit the shortlist.
- 6 · The experiment ~6 min You’ll guess: On average, how many percentage points of held-out attack success did the learned set selector add over the strongest influence baselines?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.