arXiv:2412.17228v4 Announce Type: replace
Abstract: Background: Clinical trials are essential to advancing cancer treatments, but fewer than 10% of adults with cancer enroll in therapeutic trials. Open-source AI trial matching tools could democratize access to trial options. Methods: We created MatchMiner-AI, co-developed with practicing clinical oncologists and trained on synthetic electronic health record (EHR) data. It uses open-weight LLMs to summarize patient histories from unstructured EHR text and extract target populations from trial eligibility documents. Embedding and re-ranking models were distilled to retrieve and rank trial and patient suggestions. Multifaceted evaluation was performed, including retrospective quantification of distillation fidelity; applying a closed-source LLM as judge of patient summarization and matching; and evaluation of candidate matches by oncologists. Results: Across retrospective evaluations of distillation fidelity, the pipeline outperformed a baseline text-embedding model, improving mean average precision (MAP) at 20 from 0.44 (95% CI 0.44-0.45) to 0.95 (95% CI 0.95-0.96) for trial-enrolled patients and from 0.38 (95% CI 0.37-0.38) to 0.94 (95% CI 0.93-0.94) for patients who received standard of care therapies. In a 50-patient sample selected for comparison between MatchMiner-AI and a rules-based tumor genomic trial matching algorithm, MatchMiner-AI retrieved trials for all patients, as opposed to 19 patients (38%) who had tumor genomic data available. Among those 19 patients, 80% of 256 trial suggestions retrieved by MatchMiner-AI were deemed reasonable considerations by a frontier LLM, vs 53% of 113 suggestions retrieved by the rules-based approach. Conclusion: MatchMiner-AI is an open-source, open-weights, clinical trial matching AI pipeline for oncology. Synthetic training data, model weights, inference tools, and demonstration frontends are publicly available.
Abstract: Background: Clinical trials are essential to advancing cancer treatments, but fewer than 10% of adults with cancer enroll in therapeutic trials. Open-source AI trial matching tools could democratize access to trial options. Methods: We created MatchMiner-AI, co-developed with practicing clinical oncologists and trained on synthetic electronic health record (EHR) data. It uses open-weight LLMs to summarize patient histories from unstructured EHR text and extract target populations from trial eligibility documents. Embedding and re-ranking models were distilled to retrieve and rank trial and patient suggestions. Multifaceted evaluation was performed, including retrospective quantification of distillation fidelity; applying a closed-source LLM as judge of patient summarization and matching; and evaluation of candidate matches by oncologists. Results: Across retrospective evaluations of distillation fidelity, the pipeline outperformed a baseline text-embedding model, improving mean average precision (MAP) at 20 from 0.44 (95% CI 0.44-0.45) to 0.95 (95% CI 0.95-0.96) for trial-enrolled patients and from 0.38 (95% CI 0.37-0.38) to 0.94 (95% CI 0.93-0.94) for patients who received standard of care therapies. In a 50-patient sample selected for comparison between MatchMiner-AI and a rules-based tumor genomic trial matching algorithm, MatchMiner-AI retrieved trials for all patients, as opposed to 19 patients (38%) who had tumor genomic data available. Among those 19 patients, 80% of 256 trial suggestions retrieved by MatchMiner-AI were deemed reasonable considerations by a frontier LLM, vs 53% of 113 suggestions retrieved by the rules-based approach. Conclusion: MatchMiner-AI is an open-source, open-weights, clinical trial matching AI pipeline for oncology. Synthetic training data, model weights, inference tools, and demonstration frontends are publicly available.
This article has been indexed from cs.AI updates on arXiv.org
Read the original article: