ChatGPT Only Gets Diagnoses Correct Half of the Time
By Lori Solomon HealthDay Reporter
THURSDAY, Aug. 8, 2024 -- ChatGPT is not accurate as a diagnostic tool, but does offer some medical educational benefits, according to a study published online July 31 in PLOS ONE.
Ali Hadi, from the Schulich School of Medicine and Dentistry at Western University in London, Ontario, Canada, and colleagues investigated ChatGPT’s diagnostic accuracy and utility in medical education. The analysis included 150 Medscape case challenges (September 2021 to January 2023) that were inputted into ChatGPT.
The researchers found that ChatGPT answered 49 percent of cases correctly, with an overall accuracy of 74 percent, a precision of 48.67 percent, sensitivity of 48.67 percent, specificity of 82.89 percent, and an area under the curve of 0.66. ChatGPT struggled with the interpretation of laboratory values and imaging results, but was generally correct in ruling out a specific differential diagnosis and providing reasonable next diagnostic steps. Just over half of answers were complete and relevant (52 percent) and a similar percentage of answers were characterized as low cognitive load (51 percent).
"Additional research should focus on enhancing the accuracy and dependability of ChatGPT as a diagnostic instrument," the authors write. "Integrating ChatGPT into medical education and clinical practice necessitates a thorough examination of its educational and clinical limitations. Transparent guidelines should be established for ChatGPT’s clinical usage, and medical students and clinicians should be trained on how to effectively and responsibly employ the tool."
Disclaimer: Statistical data in medical articles provide general trends and do not pertain to individuals. Individual factors can vary greatly. Always seek personalized medical advice for individual healthcare decisions.
© 2025 HealthDay. All rights reserved.
Posted August 2024
Read this next
Health Care Expenditure Expected to Top GDP Growth 2024 to 2033
THURSDAY, July 3, 2025 -- The annual growth in national health spending is expected to be faster than average gross domestic product (GDP) growth during 2024 to 2033, according to...
Considerable Variation Seen in Commercial Pricing for Surgery Services
THURSDAY, July 3, 2025 -- Commercial pricing varies considerably for general surgery services, with facility prices exhibiting greater variability, according to a research letter...
Urinary Metal Levels Tied to Increased Risk for Later Heart Failure
THURSDAY, July 3, 2025 -- There are consistent associations between elevated urinary metal levels and increased heart failure risk over time across geographically diverse cohorts...
More news resources
- FDA Medwatch Drug Alerts
- Daily MedNews
- News for Health Professionals
- New Drug Approvals
- New Drug Applications
- Drug Shortages
- Clinical Trial Results
- Generic Drug Approvals
Subscribe to our newsletter
Whatever your topic of interest, subscribe to our newsletters to get the best of Drugs.com in your inbox.