HuatuoGPT-3: RL-Only Domain Adaptation from Base Models
Junying Chen, Xinyuan Xie, Ziniu Li, Wenyuan Gu, Jianquan Li, Xiang Wan, Guangjun Yu, Ruoyu Sun and 2 others
The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data · Shenzhen Loop Area Institute · National Health Data Institute, Shenzhen
Published October 5, 2026 · on the feed October 7, 2026 · ▲ 43 · arXiv 2610.05966
The abstract
Domain adaptation aims to turn a general-purpose large language model (LLM) into an expert for a target domain. While the dominant SFT+RL pipeline offers a convenient cold start, it may reduce exploration diversity and introduces additional complexity through multi-stage optimization. These limitations motivate RL-only adaptation. However, pure on-policy RL suffers from a cold-start problem, while mixed-policy RL still falls short: informative tokens in teacher outputs are learned too slowly in early training, and stale teacher outputs can hinder later improvement. We identify these two failure modes as Gradient Starvation and Teacher-Distribution Anchoring. To address them, we propose One-stage Policy Optimization (OnePO), which treats teacher outputs as transient guidance for policy improvement. OnePO combines Adaptive Objective Evolution to strengthen learning on informative low-probability teacher tokens and Teacher Retirement to discard teacher outputs once the current policy can surpass them. On medical adaptation, OnePO achieves 67.2 on HealthBench (Total) with only 20K training samples, outperforming SFT+RL and pure RL by 2.7 and 7.4 points, respectively. We further scale OnePO to produce HuatuoGPT-3, an open-source medical LLM series whose 27B variant reaches 70.1 on HealthBench (Total) and 71.4 on HealthBench Professional, surpassing frontier models such as GPT-6 Astra. Models and code are available at https://github.com/FreedomIntelligence/HuatuoGPT-3.
The Brief
2:28A short podcast summary of the paper. Written and voiced by AI.
Transcript
HuatuoGPT-3. RL-Only Domain Adaptation from Base Models. By Junying Chen and nine others, from the Chinese University of Hong Kong, Shenzhen, the Shenzhen Research Institute of Big Data, and two other institutions. Published on the fifth of October. Its code repository is public and real: fifteen source files.
Domain adaptation turns a general-purpose large language model into an expert in a target domain. The dominant pipeline does it in two stages: supervised fine-tuning, then reinforcement learning. That gives a convenient cold start, but it may reduce exploration diversity, and the multi-stage optimization adds complexity. Dropping the first stage is the obvious response, and it runs into trouble. Pure on-policy RL suffers from a cold-start problem of its own. Mixed-policy RL, which brings teacher outputs into the training, still falls short in two ways.
Early in training, the informative tokens in teacher outputs are learned too slowly. Later on, teacher outputs that have gone stale can hold back further improvement. The authors name these two failure modes Gradient Starvation and Teacher-Distribution Anchoring.
Their answer is One-stage Policy Optimization, or OnePO. It treats teacher outputs as transient guidance for improving the policy: useful for a while, and then meant to be let go.
OnePO has one component for each failure mode. Adaptive Objective Evolution targets the slow start, strengthening learning on teacher tokens that are informative but have low probability under the current policy. Teacher Retirement targets the late anchoring, discarding teacher outputs once the current policy can surpass them. The teacher helps where the policy is weak, and steps aside where it is no longer ahead.
On medical adaptation, OnePO scores sixty-seven point two on HealthBench Total with only twenty thousand training samples. That beats the two-stage pipeline by two point seven points, and pure RL by seven point four points. The method is then scaled up to produce HuatuoGPT-3, an open-source series of medical models. Its twenty-seven billion parameter variant reaches seventy point one on HealthBench Total and seventy-one point four on HealthBench Professional, surpassing frontier models such as GPT-6 Astra.
The argument runs from diagnosis to design. The difficulties of skipping supervised fine-tuning are given names, each name gets a mechanism, and the result is a single stage that keeps reinforcement learning's exploration while borrowing a teacher only for as long as the teacher has something to offer. Models and code are released.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a teacher who helps, then holds you back.
You want to turn a general language model into a medical expert. The usual route imitates teacher answers first, then runs reinforcement learning, but imitation may narrow exploration and two stages add complexity.
The road ahead
- 1 · The problem ~7 min Making a general language model a medical expert is usually done in two stages, and each simpler alternative so far has its own weak spot.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min OnePO treats teacher outputs as transient guidance in one RL stage: learned from strongly while the policy is weak, retired once it beats them.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min OnePO runs one RL stage from the base model, mixes in teacher outputs, boosts their unlikely tokens, and retires each once surpassed.
- 6 · The experiment ~6 min You’ll guess: Trained on 20K samples, by how many HealthBench Total points did OnePO beat the two-stage SFT+RL pipeline?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.