arXiv:2503.00992v3 Announce Type: replace
Abstract: Background. Evidence that large language models (LLMs) reproduce human judgments does not establish conceptual mastery: the correspondence may reflect memorisation or be sensitivite to incidental task features.
Objective. Across five experiments, we test whether 13 LLMs possess a generalisable competence in applying rules, including cases in which a rule's text and purpose point towards different outcomes.
Method. Study 1A compared LLM judgments with newly collected human data on published stimuli and matched vignettes created after the models' training cut-offs. Studies 2A/2B compared responses to time-pressure instructions, a manipulation with a mechanistic route to human judgment blocked for LLMs. Study 3 varied reasoning effort, as an analogue for time constrained human judgements. Studies 1B/2B alsovaried system prompt wording and numerical scale anchors.
Results LLM judgments closely tracked human judgments for both stimulus sets, while responding in the same unanticipated purposivist direction in the new set as humans did. Sensitivity to text and purpose was robust across prompt variations. Responses to time-pressure instructions were model-specific, suggesting a distinction between conceptual competence and human alignment. Replication of the human pattern was most apparent in models with fewer parameters, and these effects were susceptible to prompt variation. Increasing reasoning effort produced no detectable change in rule application for most models though a significant purposivist trend was observed in higher-effort for GPT-oss and Claude Sonnet 5. Response variance remained lower for LLMs than humans despite our per-model temperature calibration to match human sample variance.
Conclusions. Overall, the findings suggest that LLM rule application reflects a generalisable, standing semantic competence that does not typically depend on expanded deliberation.
Abstract: Background. Evidence that large language models (LLMs) reproduce human judgments does not establish conceptual mastery: the correspondence may reflect memorisation or be sensitivite to incidental task features.
Objective. Across five experiments, we test whether 13 LLMs possess a generalisable competence in applying rules, including cases in which a rule's text and purpose point towards different outcomes.
Method. Study 1A compared LLM judgments with newly collected human data on published stimuli and matched vignettes created after the models' training cut-offs. Studies 2A/2B compared responses to time-pressure instructions, a manipulation with a mechanistic route to human judgment blocked for LLMs. Study 3 varied reasoning effort, as an analogue for time constrained human judgements. Studies 1B/2B alsovaried system prompt wording and numerical scale anchors.
Results LLM judgments closely tracked human judgments for both stimulus sets, while responding in the same unanticipated purposivist direction in the new set as humans did. Sensitivity to text and purpose was robust across prompt variations. Responses to time-pressure instructions were model-specific, suggesting a distinction between conceptual competence and human alignment. Replication of the human pattern was most apparent in models with fewer parameters, and these effects were susceptible to prompt variation. Increasing reasoning effort produced no detectable change in rule application for most models though a significant purposivist trend was observed in higher-effort for GPT-oss and Claude Sonnet 5. Response variance remained lower for LLMs than humans despite our per-model temperature calibration to match human sample variance.
Conclusions. Overall, the findings suggest that LLM rule application reflects a generalisable, standing semantic competence that does not typically depend on expanded deliberation.
This article has been indexed from cs.AI updates on arXiv.org
Read the original article: