ChatGPT Only Gets Diagnoses Correct Half of the Time
By Lori Solomon HealthDay Reporter
THURSDAY, Aug. 8, 2024 -- ChatGPT is not accurate as a diagnostic tool, but does offer some medical educational benefits, according to a study published online July 31 in PLOS ONE.
Ali Hadi, from the Schulich School of Medicine and Dentistry at Western University in London, Ontario, Canada, and colleagues investigated ChatGPT’s diagnostic accuracy and utility in medical education. The analysis included 150 Medscape case challenges (September 2021 to January 2023) that were inputted into ChatGPT.
The researchers found that ChatGPT answered 49 percent of cases correctly, with an overall accuracy of 74 percent, a precision of 48.67 percent, sensitivity of 48.67 percent, specificity of 82.89 percent, and an area under the curve of 0.66. ChatGPT struggled with the interpretation of laboratory values and imaging results, but was generally correct in ruling out a specific differential diagnosis and providing reasonable next diagnostic steps. Just over half of answers were complete and relevant (52 percent) and a similar percentage of answers were characterized as low cognitive load (51 percent).
"Additional research should focus on enhancing the accuracy and dependability of ChatGPT as a diagnostic instrument," the authors write. "Integrating ChatGPT into medical education and clinical practice necessitates a thorough examination of its educational and clinical limitations. Transparent guidelines should be established for ChatGPT’s clinical usage, and medical students and clinicians should be trained on how to effectively and responsibly employ the tool."
Disclaimer: Statistical data in medical articles provide general trends and do not pertain to individuals. Individual factors can vary greatly. Always seek personalized medical advice for individual healthcare decisions.

© 2025 HealthDay. All rights reserved.
Posted August 2024
Read this next
Declining Childhood Vaccination May Increase Risk for Vaccine-Preventable Infections
WEDNESDAY, April 30, 2025 -- Declining childhood vaccination rates may increase outbreaks of eliminated vaccine-preventable infections within the United States, leading to a...
AACR: Incidence-Based Mortality Dropping for Young Women With Breast Cancer
TUESDAY, April 29, 2025 -- Incidence-based mortality (IBM) declined from 2010 to 2020 among women aged 20 to 49 years diagnosed with breast cancer, according to a study presented...
AACR: Nonsurgical Treatment Feasible for Mismatch Repair-Deficient Tumors
TUESDAY, April 29, 2025 -- A neoadjuvant programmed cell death 1 (PD-1) blockade enables nonoperative management among patients with early-stage mismatch repair-deficient (dMMR)...
More news resources
- FDA Medwatch Drug Alerts
- Daily MedNews
- News for Health Professionals
- New Drug Approvals
- New Drug Applications
- Drug Shortages
- Clinical Trial Results
- Generic Drug Approvals
Subscribe to our newsletter
Whatever your topic of interest, subscribe to our newsletters to get the best of Drugs.com in your inbox.