arXiv:2604.08552v3 Announce Type: replace-cross
Abstract: Descriptive scientific metadata in public repositories are often incomplete and inconsistent with community standards and ontologies, limiting data FAIRness. Large language models (LLMs) offer a promising approach to automatically standardizing such metadata when provided with relevant standards in machine-actionable form, such as metadata templates from the CEDAR Workbench. Prompt engineering, however, provides only fixed snapshots of these standards and relies on an LLM's pretrained knowledge to interpret and satisfy their constraints. We evaluate whether giving an LLM access to metadata specifications and authoritative terminology at runtime improves automated metadata standardization. Methods: We present ARMS, a tool-augmented LLM agent that retrieves complete CEDAR metadata templates and dynamically queries authoritative biomedical terminology services at execution time. We compared ARMS with a prompt-based approach on 839 legacy metadata records from the Human BioMolecular Atlas Program (HuBMAP), using expert-standardized records as the reference standard. Results: ARMS outperformed the prompt-based approach, increasing precision from 0.56 to 0.93 and recall from 0.51 to 0.85, with improvements across all field categories and assay types. The largest gains occurred for ontology-constrained fields, where precision increased from 0.36 to 0.92. Conclusion: LLMs cannot convert legacy metadata to standards-adherent form without knowledge of the relevant standards. ARMS improves metadata standardization by providing runtime access to authoritative resources that define valid metadata. Machine-actionable metadata standards enhance LLM-based rectification of legacy metadata, especially when they can be queried dynamically.
Abstract: Descriptive scientific metadata in public repositories are often incomplete and inconsistent with community standards and ontologies, limiting data FAIRness. Large language models (LLMs) offer a promising approach to automatically standardizing such metadata when provided with relevant standards in machine-actionable form, such as metadata templates from the CEDAR Workbench. Prompt engineering, however, provides only fixed snapshots of these standards and relies on an LLM's pretrained knowledge to interpret and satisfy their constraints. We evaluate whether giving an LLM access to metadata specifications and authoritative terminology at runtime improves automated metadata standardization. Methods: We present ARMS, a tool-augmented LLM agent that retrieves complete CEDAR metadata templates and dynamically queries authoritative biomedical terminology services at execution time. We compared ARMS with a prompt-based approach on 839 legacy metadata records from the Human BioMolecular Atlas Program (HuBMAP), using expert-standardized records as the reference standard. Results: ARMS outperformed the prompt-based approach, increasing precision from 0.56 to 0.93 and recall from 0.51 to 0.85, with improvements across all field categories and assay types. The largest gains occurred for ontology-constrained fields, where precision increased from 0.36 to 0.92. Conclusion: LLMs cannot convert legacy metadata to standards-adherent form without knowledge of the relevant standards. ARMS improves metadata standardization by providing runtime access to authoritative resources that define valid metadata. Machine-actionable metadata standards enhance LLM-based rectification of legacy metadata, especially when they can be queried dynamically.
This article has been indexed from cs.AI updates on arXiv.org
Read the original article: