KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization

KV4-RL. Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization. By Sihwan Park and three others, from KAIST, NAVER Cloud, and one other institution. Published on the tenth of October. No code repository is linked to it.

Asynchronous rollout makes reinforcement learning with verifiable rewards more efficient, because it cuts the under-utilization that builds up while a long tail of generations finishes. Keeping many requests in flight has a side effect, though. Context processing gets concentrated into fewer decode steps, so each step carries a heavier key-value cache workload. The authors characterize that concentration, and in their matched-workload comparisons, quantizing the KV cache delivers substantially greater acceleration than weight-only or weight-activation quantization. The catch is that INT4 KV rollout can degrade learning, even with token-level truncated importance sampling in place.

KV4-RL keeps the cheap format almost everywhere and protects a few places. It combines calibration-free INT4 quantization of the KV cache with selective INT8 protection for two kinds of position: attention sinks, and recent context.

The protected set is bounded. Only a limited number of positions is held at the higher precision, and that is enough to reduce the mismatch between the sampler and the policy being trained, while retaining most of the speedup that INT4 offers.

The main test runs four hundred iterations of GRPO training on Qwen3-8B-Base. Against rollout in BF16, KV4-RL achieves a cumulative rollout speedup of one point eight times, and a cumulative full-iteration speedup of one point two seven times. Over the final fifty iterations alone, the rollout speedup reaches one point nine seven times.

Accuracy is the other side of the ledger. With every run scored by a common BF16 evaluator, final mean accuracy across six benchmarks lands within half a percentage point of BF16-rollout training, for both Qwen3-4B-Base and Qwen3-8B-Base.

The argument moves in three steps. Asynchrony concentrates work in the cache, so the cache is where quantization pays most. Four-bit caches on their own hurt learning. Guarding a small, bounded set of positions at eight bits recovers the learning while keeping most of the speed.