arXiv:2608.07943v2 Announce Type: replace
Abstract: Multi-page visually rich document understanding (MP-VRDU) systems can fail when evidence is not faithfully represented, not selected, or not used correctly, yet final accuracy alone does not reveal where failures occur. We present an empirical attribution framework that examines these failures through controlled interventions in a unified document question answering pipeline. Experiments on MMLongBench-Doc, with cross-dataset and cross-family checks, show that complementary modalities can reduce sensitivity to conversion errors: images mitigate parser errors, while extracted text reduces dependence on visual resolution. Adding related non-gold pages can both repair incorrect answers and break correct ones, even when all annotated evidence is present, revealing instability beneath aggregate accuracy. Abstention trades unsupported answers for false refusals, while its interaction with reasoning instructions differs across backbones. Qualitative examples and accuracy–cost comparisons connect these findings to guidance on representation, evidence selection, and reasoner choice under deployment constraints.
Abstract: Multi-page visually rich document understanding (MP-VRDU) systems can fail when evidence is not faithfully represented, not selected, or not used correctly, yet final accuracy alone does not reveal where failures occur. We present an empirical attribution framework that examines these failures through controlled interventions in a unified document question answering pipeline. Experiments on MMLongBench-Doc, with cross-dataset and cross-family checks, show that complementary modalities can reduce sensitivity to conversion errors: images mitigate parser errors, while extracted text reduces dependence on visual resolution. Adding related non-gold pages can both repair incorrect answers and break correct ones, even when all annotated evidence is present, revealing instability beneath aggregate accuracy. Abstention trades unsupported answers for false refusals, while its interaction with reasoning instructions differs across backbones. Qualitative examples and accuracy–cost comparisons connect these findings to guidance on representation, evidence selection, and reasoner choice under deployment constraints.
This article has been indexed from cs.AI updates on arXiv.org
Read the original article: