The Performance and Reliability Evaluation for Continuous Modifications and Usability of Artificial Intelligence (PRECISE-AI) program by ARPA-H focuses on maintaining the accuracy and effectiveness of AI models in healthcare. As AI-enabled medical devices become more prevalent, their accuracy is critical for clinical decision-making. However, research has shown that machine learning (ML) models in clinical settings often degrade over time due to changes in input data, such as shifts in patient demographics, clinical workflows, or IT infrastructure. This degradation can lead to inaccurate outputs, which jeopardize patient safety and healthcare efficiency.
Currently, no clinical AI models are routinely tested or updated to counteract performance degradation, and there are no regulatory requirements to do so. Detection of such issues relies heavily on physicians’ clinical intuition, which is variable and unreliable, increasing the risk of misdiagnosis. PRECISE-AI aims to address these shortcomings by developing automated tools to monitor AI model performance, detect degradation, and implement corrections without human oversight. These tools will also enhance transparency by identifying degradation sources and communicating model uncertainty, enabling better clinical use of AI.
The program focuses on five technical areas. TA 1 will extract and integrate data across clinical use cases to establish accurate patient “ground truth.” TA 2 will implement continuous performance monitoring, root cause analysis, and automatic corrections. TA 3 aims to quantify and communicate model uncertainty effectively to stakeholders, enhancing trust and usability. TA 4 will aggregate data across institutions to support development across technical areas, fostering collaboration. TA 5 will independently validate progress made in the other technical areas, ensuring robustness and reliability.
PRECISE-AI relies on interdisciplinary collaboration, bringing together experts in machine learning, health informatics, and clinical medicine. Teams are encouraged to form partnerships to achieve the program's ambitious goals. A Proposers' Day recording, FAQs, and a teaming page are available to support potential applicants in preparing their submissions. Proposals for innovative solutions are due by January 15, 2025, with adjustments to page length requirements noted in the final Innovative Solutions Opening (ISO).
This program aims to ensure that AI tools in healthcare maintain peak clinical performance, reducing risks and enhancing outcomes for patients. By addressing the challenges of AI model degradation and improving usability, PRECISE-AI represents a transformative step in integrating reliable AI into healthcare systems. ARPA-H's role in fostering interdisciplinary coordination positions the program to deliver impactful, scalable solutions.
