CaptchaArena: A Large-Scale, Fine-Grained Dataset for Training Computer-Use Agents on Interactive CAPTCHAs
Zhenhao Zhang, Zhaoyu Fan, Haohan Ying, Jingwen Hu, Hancen Fan, Junhao Zhou, Zitian Chen, Linchao Zhu
Columbia University · Zhejiang University · University of Rochester · University of Illinois at Urbana-Champaign
Published September 25, 2026 · on the feed September 30, 2026 · ▲ 5 · arXiv 2609.31957
The abstract
Interactive CAPTCHAs remain challenging for computer-use agents, while existing datasets face trade-offs among type coverage, interaction fidelity, and trajectory supervision. To address these gaps, we present CaptchaArena, the first large-scale, fine-grained training dataset for interactive CAPTCHA solving. It contains 50K puzzles across 20 CAPTCHA types and 5 interaction modes, with every solution verified through execution. CaptchaArena provides 50K screenshot-action trajectories, including 46K with step-by-step reasoning annotations. It also includes fine-grained pixel-mask annotations for irregular targets. Using CaptchaArena, we train CaptchaAgent, a single 9B policy for all 20 CAPTCHA types, with supervised fine-tuning followed by reinforcement learning. The environment verifier directly provides the RL reward. Supervised fine-tuning reaches 70.5 Pass@1, and reinforcement learning further improves it to 71.7, while also improving performance on two external benchmarks. These results demonstrate the value of large-scale, fine-grained computer-use supervision for training interactive CAPTCHA agents. We release CaptchaArena and CaptchaAgent at https://github.com/X0X0X00/CaptchaArena.
The Brief
2:06A short podcast summary of the paper. Written and voiced by AI.
Transcript
CaptchaArena. A Large-Scale, Fine-Grained Dataset for Training Computer-Use Agents on Interactive CAPTCHAs. By Zhenhao Zhang and seven others, from Columbia University, Zhejiang University, and two other institutions. Published on the twenty-fifth of September. Its code repository is public and real: about thirty source files.
Interactive CAPTCHAs are still hard for computer-use agents, and the datasets available for training them each give something up. Some lack coverage of CAPTCHA types, some lack fidelity in the interaction, and some lack supervision of the trajectory. CaptchaArena is presented as the first large-scale, fine-grained training dataset for solving interactive CAPTCHAs. It is meant to close all three gaps at once: many types, interaction as it really happens, and supervision at every step.
It holds fifty thousand puzzles, spread across twenty CAPTCHA types and five interaction modes. Alongside them come fifty thousand trajectories pairing screenshots with actions, and forty-six thousand of those carry step-by-step reasoning annotations. For targets with irregular shapes, there are fine-grained annotations at the level of pixel masks.
Every solution in the set was verified by executing it. That same environment verifier is what supplies the reward when the authors move to reinforcement learning, so the check that certifies the data is also the one grading the agent during training. A puzzle counts as solved only if the solution actually works when carried out, which is a stricter standard than a label someone wrote down.
The intended users are those training computer-use agents. The authors demonstrate with CaptchaAgent, a single nine-billion-parameter policy that handles all twenty CAPTCHA types, trained first with supervised fine-tuning and then with reinforcement learning. Fine-tuning reaches seventy point five pass at one. Reinforcement learning lifts that to seventy-one point seven, and also improves performance on two external benchmarks.
Both the dataset and the agent are released. The argument drawn from them is about supervision rather than architecture: for agents that must solve interactive CAPTCHAs, large-scale and fine-grained computer-use data is what the authors credit for the result.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a small puzzle that stops a capable agent dead.
You build a computer-use agent that works from screenshots, mouse clicks and key presses. It handles long tasks, but one interactive CAPTCHA in the middle stops the whole workflow.
The road ahead
- 1 · The problem ~7 min Agents that use a computer by screenshots and clicks get stuck on interactive CAPTCHAs, and the training data on offer is thin in one way or another.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Make each puzzle a browser task whose solution is run and accepted by the site's verifier, record that run with reasoning, and reuse the verifier as reward.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Verify each solution by running it, record the run, annotate it with reasoning, fine-tune the policy on it, then refine with verifier-rewarded reinforcement learning.
- 6 · The experiment ~6 min You’ll guess: After supervised fine-tuning, what did adding reinforcement learning do to CaptchaAgent's Pass@1?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.