arXiv:2609.04518v2 Announce Type: replace
Abstract: Recent systems train coding agents with reinforcement learning (RL) on trajectories collected through multiple execution harnesses. However, gains under the training harnesses leave open whether the improvements transfer to unseen interfaces and how they depend on reward normalization. We present a controlled study that measures RL gains under fixed harnesses and compares the same models across harnesses. We train two Qwen3-8B RL variants from the same supervised fine-tuning (SFT) checkpoint using identical replay data. They differ only in whether rewards for each task are normalized within each harness (Within) or across harnesses (Cross). On SWE-bench Verified, we identify three main findings: (i) Multi-harness RL yields modest gains over the SFT baseline under both normalization rules. Across the four training harnesses, the estimated mean improvements are +0.48 and +0.77 percentage points for Within and Cross, respectively. (ii) Cross increases the fraction of trajectories with nonzero advantage but provides no consistent advantage over Within on an unseen harness across training seeds and evaluation budgets; neither rule significantly improves over SFT on this harness. (iii) Performance varies substantially more across harnesses than with RL training. After averaging over model variants, the highest and lowest source-harness solve rates differ by 7.13 percentage points, far exceeding the estimated training gains. Even after multi-harness RL, the observed performance differences across execution interfaces remain substantially larger than either the gains from training or the differences between normalization rules.
Abstract: Recent systems train coding agents with reinforcement learning (RL) on trajectories collected through multiple execution harnesses. However, gains under the training harnesses leave open whether the improvements transfer to unseen interfaces and how they depend on reward normalization. We present a controlled study that measures RL gains under fixed harnesses and compares the same models across harnesses. We train two Qwen3-8B RL variants from the same supervised fine-tuning (SFT) checkpoint using identical replay data. They differ only in whether rewards for each task are normalized within each harness (Within) or across harnesses (Cross). On SWE-bench Verified, we identify three main findings: (i) Multi-harness RL yields modest gains over the SFT baseline under both normalization rules. Across the four training harnesses, the estimated mean improvements are +0.48 and +0.77 percentage points for Within and Cross, respectively. (ii) Cross increases the fraction of trajectories with nonzero advantage but provides no consistent advantage over Within on an unseen harness across training seeds and evaluation budgets; neither rule significantly improves over SFT on this harness. (iii) Performance varies substantially more across harnesses than with RL training. After averaging over model variants, the highest and lowest source-harness solve rates differ by 7.13 percentage points, far exceeding the estimated training gains. Even after multi-harness RL, the observed performance differences across execution interfaces remain substantially larger than either the gains from training or the differences between normalization rules.
This article has been indexed from cs.AI updates on arXiv.org
Read the original article: