When Agents Play Differently: Localizing Human–Agent Behavioral Divergence with Conditional Distributional Testing

arXiv:2611.26012v1 Announce Type: new
Abstract: Evaluating whether artificial agents behave like humans requires comparing not just average outcomes but entire behavioral distributions across diverse contexts. We propose a stratified framework that computes nonparametric discrepancy statistics within contextual cells, and discovers subgroup rules identifying where agent behavior departs from human play. We show that session-level metrics dominated by win/loss outcomes offer little signal beyond mean comparisons, whereas process-level metrics such as move efficiency reveal genuine behavioral differences. Applied to a commercial match-3 game, it uncovers patterns invisible to aggregate metrics: a playtesting bot is diverging substantially from humans in certain game modes, while showing no detectable divergence in others. We release our implementation as open source to support reproducibility and adoption.
This article has been indexed from cs.AI updates on arXiv.org

Read the original article:

When Agents Play Differently: Localizing Human–Agent Behavioral Divergence with Conditional Distributional Testing