Abstract: Comparative feedback, in which a judge says which of two candidate outcomes is closer to what happened, is a practical way to train language world models that assign probabilities to an action's outcomes. We show that, fitted with a Bradley-Terry objective, it need not teach the true outcome probabilities and can distort accurate ones. On a
[…]
Content was trimmed to protect the source. Please visit the original article for the full text.
Read the original article:
Bradley-Terry Training Can Distort Outcome Probabilities of Initially Accurate Language World Models
