arXiv:2609.18958v1 Announce Type: cross
Abstract: The natural way to review a long recording or document with a multimodal model is to hand it the raw source and ask for a review in one call. We show that this quietly fails: the model satisfices, dropping roughly a third of the content and embellishing the rest. The failure is not perception–almost all of the dropped content reappears when the same model is simply asked to transcribe the source. The bottleneck is generation under load: a single pass cannot perceive, reason over, and write a long faithful review at the same time, because doing all three competes for one output. We rule out the obvious alternatives. It is not the modality: models read text and an image of the same text equally well. And it is not merely a matter of thinking harder: giving the single pass a far larger reasoning budget does not recover the lost content, because the model spends that budget planning a review rather than writing the source down. What works is to split the labor across two same-weights passes–first transcribe, then review the transcript–so each step gets a full output budget of its own. This transcribe-then-review decomposition improves both faithfulness and coverage across a 21-source suite. The benefit is not uniform: we observe that it helps most where the one-pass baseline is weakest and little where that baseline is already strong, a pattern that also tracks the source's length and modality. Decomposition comes with two failure modes–the review pass running out of room on very long sources, and confabulating from memory once the grounding source is removed.
Abstract: The natural way to review a long recording or document with a multimodal model is to hand it the raw source and ask for a review in one call. We show that this quietly fails: the model satisfices, dropping roughly a third of the content and embellishing the rest. The failure is not perception–almost all of the dropped content reappears when the same model is simply asked to transcribe the source. The bottleneck is generation under load: a single pass cannot perceive, reason over, and write a long faithful review at the same time, because doing all three competes for one output. We rule out the obvious alternatives. It is not the modality: models read text and an image of the same text equally well. And it is not merely a matter of thinking harder: giving the single pass a far larger reasoning budget does not recover the lost content, because the model spends that budget planning a review rather than writing the source down. What works is to split the labor across two same-weights passes–first transcribe, then review the transcript–so each step gets a full output budget of its own. This transcribe-then-review decomposition improves both faithfulness and coverage across a 21-source suite. The benefit is not uniform: we observe that it helps most where the one-pass baseline is weakest and little where that baseline is already strong, a pattern that also tracks the source's length and modality. Decomposition comes with two failure modes–the review pass running out of room on very long sources, and confabulating from memory once the grounding source is removed.
This article has been indexed from cs.AI updates on arXiv.org
Read the original article: