Registrieren

David Toro-Tobon and colleagues argue that “clinician in the loop” is shifting responsibility for AI safety from developers to doctors and cannot be relied on as a failsafe for patients Consider a routine encounter: an endocrinologist examines a patient’s thyroid by ultrasonography. A nodule the clinician judges to be a benign cyst is flagged by an AI tool as highly suspicious of malignancy. Accepting this output could lead to unnecessary and potentially harmful care. Overriding it would require the clinician to provide documentation, reassurance, and justification for forgoing biopsy and, if the algorithm proves correct, exposes the patient to harm and the clinician to liability for delaying the diagnosis and treatment of thyroid cancer. This dilemma extends beyond thyroid imaging. Healthcare AI comprises diverse technologies with distinct governance needs: predictive risk scores in emergency departments (often locally developed and unregulated), diagnostic AI in radiology (regulated as medical device software), and generative AI in primary care (often general purpose and unregulated). As AI reaches the bedside, the debate shifts from technical performance to human responsibility: who appraises these systems, interprets the accuracy and relevance of their outputs for individual patients, and ensures their safe and effective use? The Food and Drug Administration,1 European Union,2 and the World Health Organization3 all recommend clinicians review and approve AI outputs before implementation. This approach to oversight, sometimes referred to as “clinician in the loop,” is borrowed from engineering and aims to offer safety while enabling the accelerated deployment of potentially transformative AI systems (fig 1).4 However, it requires that clinicians, already caring for patients at the limits of time and attention, also oversee and override “black box” algorithms and interpret outputs for each patient’s clinical, cultural, and moral reality. The model rests on a fragile assumption: that clinicians can …