arXiv:2605.23262v3 Announce Type: replace
Abstract: Agent benchmarks have attracted substantial attention; yet, benchmarks for AI agents performing knowledge work are still mostly described only by their tasks, environments, and metrics. Starting from the definition of knowledge work, this paper derives four other design choices that often remain implicit. Three research threads on how knowledge work is organized each yield one choice: studies of professions, situated action, and boundary objects and coordination, while validity theory provides the fourth. Following this reasoning, we propose a benchmark representation centered on the work being performed. Based on this representation, we propose reporting practices for four largely missing fields: represented activity, tested setting, required work product, and evaluated result. Benchmarks designed in disparate ways thereby share a common description and can be compared side by side. To support the reporting of work activities, we cluster O*NET task statements into 18 shared work activities through unsupervised learning paired with human review. We then use this representation to analyze cases from three popular knowledge work benchmarks: GDPval, OfficeQA Pro, and APEX-SWE. Across all three benchmarks, the scope covered by the evaluated result is narrower than that of the required work product. We audited the tests of all 25 public APEX-SWE Integration tasks, independently coded by two authors. We found from the test code that in 13 of these tasks, a deliverable can alter established content from the source data and still pass all tests. We synthesize these four fields into a one-page record, accompanied by a worked example and a fillable template.
Abstract: Agent benchmarks have attracted substantial attention; yet, benchmarks for AI agents performing knowledge work are still mostly described only by their tasks, environments, and metrics. Starting from the definition of knowledge work, this paper derives four other design choices that often remain implicit. Three research threads on how knowledge work is organized each yield one choice: studies of professions, situated action, and boundary objects and coordination, while validity theory provides the fourth. Following this reasoning, we propose a benchmark representation centered on the work being performed. Based on this representation, we propose reporting practices for four largely missing fields: represented activity, tested setting, required work product, and evaluated result. Benchmarks designed in disparate ways thereby share a common description and can be compared side by side. To support the reporting of work activities, we cluster O*NET task statements into 18 shared work activities through unsupervised learning paired with human review. We then use this representation to analyze cases from three popular knowledge work benchmarks: GDPval, OfficeQA Pro, and APEX-SWE. Across all three benchmarks, the scope covered by the evaluated result is narrower than that of the required work product. We audited the tests of all 25 public APEX-SWE Integration tasks, independently coded by two authors. We found from the test code that in 13 of these tasks, a deliverable can alter established content from the source data and still pass all tests. We synthesize these four fields into a one-page record, accompanied by a worked example and a fillable template.
This article has been indexed from cs.AI updates on arXiv.org