arXiv:2605.27752v3 Announce Type: replace
Abstract: Is verbalized confidence better calibrated than token likelihood? The answer depends on how the token likelihood is measured: which answer is scored, and under which prompt. Published comparisons diverge on this, and in a twelve-study audit five never state the choice. We fix one prediction event per question, the model's own answer together with its correctness label, and score that same answer under a plain query and inside the confidence prompt, holding the answer and its label fixed. Across four QA datasets and three 7-8B Instruct models this changes which signal performs better, by point estimate, in 4 of 12 settings under ECE and 9 of 12 under AUROC. The AUROC result cannot come from rescaling the likelihoods, since AUROC is invariant to any common order-preserving transformation; the items are ordered differently. Two further choices behave the same way: substituting the reference string for the model's own answer, and reading the first answer token instead of the answer span. Crossing three answer slots, two scored strings, and two readouts gives twelve measured operational variants that leave the sign of the ECE comparison ndetermined in 6 of 12 settings, whereas alternative calibration estimators move it substantially less, although one changes a single prompted-context winner. Verbalized confidence is sensitive to answer formulation as well: replacing an accepted TriviaQA alias with the canonical reference raises confidence by $0.072$ although both answers are correct. Comparing the two signals therefore requires an explicit answer, context, and evaluation protocol.
Abstract: Is verbalized confidence better calibrated than token likelihood? The answer depends on how the token likelihood is measured: which answer is scored, and under which prompt. Published comparisons diverge on this, and in a twelve-study audit five never state the choice. We fix one prediction event per question, the model's own answer together with its correctness label, and score that same answer under a plain query and inside the confidence prompt, holding the answer and its label fixed. Across four QA datasets and three 7-8B Instruct models this changes which signal performs better, by point estimate, in 4 of 12 settings under ECE and 9 of 12 under AUROC. The AUROC result cannot come from rescaling the likelihoods, since AUROC is invariant to any common order-preserving transformation; the items are ordered differently. Two further choices behave the same way: substituting the reference string for the model's own answer, and reading the first answer token instead of the answer span. Crossing three answer slots, two scored strings, and two readouts gives twelve measured operational variants that leave the sign of the ECE comparison ndetermined in 6 of 12 settings, whereas alternative calibration estimators move it substantially less, although one changes a single prompted-context winner. Verbalized confidence is sensitive to answer formulation as well: replacing an accepted TriviaQA alias with the canonical reference raises confidence by $0.072$ although both answers are correct. Comparing the two signals therefore requires an explicit answer, context, and evaluation protocol.
This article has been indexed from cs.AI updates on arXiv.org
Read the original article: