arXiv:2606.18101v3 Announce Type: replace
Abstract: GUI grounding requires vision-language models (VLMs) to identify small target elements in high-resolution screenshots and predict precise screen coordinates. On-policy self-distillation (OPSD) is a promising post-training approach for this coordinate-sensitive task because it provides dense token-level teacher signals beyond hard coordinate labels. However, naive OPSD is not well-suited to GUI grounding: since the teacher is evaluated on student-generated prefixes, its coordinate-token signals can become unreliable once a prefix deviates from the target coordinate. We propose quality-aware self-distillation for GUI grounding, centered on direct spatial correctness verification of teacher predictions. A soft gate checks whether the teacher's top coordinate-token prediction can still be completed into the ground-truth box under the student-generated prefix and down-weights the signal when this check fails. Teacher-probability scaling then uses the teacher's confidence to further calibrate the gated supervision. A key empirical finding is that neither component alone improves overall performance, whereas combining them consistently does. This suggests that the two mechanisms play complementary roles: gating suppresses unreliable coordinate-token supervision, while probability scaling calibrates the remaining signals. Experiments across six GUI grounding benchmarks show that our method consistently improves the base model and outperforms strong baselines.
Abstract: GUI grounding requires vision-language models (VLMs) to identify small target elements in high-resolution screenshots and predict precise screen coordinates. On-policy self-distillation (OPSD) is a promising post-training approach for this coordinate-sensitive task because it provides dense token-level teacher signals beyond hard coordinate labels. However, naive OPSD is not well-suited to GUI grounding: since the teacher is evaluated on student-generated prefixes, its coordinate-token signals can become unreliable once a prefix deviates from the target coordinate. We propose quality-aware self-distillation for GUI grounding, centered on direct spatial correctness verification of teacher predictions. A soft gate checks whether the teacher's top coordinate-token prediction can still be completed into the ground-truth box under the student-generated prefix and down-weights the signal when this check fails. Teacher-probability scaling then uses the teacher's confidence to further calibrate the gated supervision. A key empirical finding is that neither component alone improves overall performance, whereas combining them consistently does. This suggests that the two mechanisms play complementary roles: gating suppresses unreliable coordinate-token supervision, while probability scaling calibrates the remaining signals. Experiments across six GUI grounding benchmarks show that our method consistently improves the base model and outperforms strong baselines.
This article has been indexed from cs.AI updates on arXiv.org
Read the original article:
Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding