ChatGPT Only Gets Diagnoses Correct Half of the Time
By Lori Solomon HealthDay Reporter
THURSDAY, Aug. 8, 2024 -- ChatGPT is not accurate as a diagnostic tool, but does offer some medical educational benefits, according to a study published online July 31 in PLOS ONE.
Ali Hadi, from the Schulich School of Medicine and Dentistry at Western University in London, Ontario, Canada, and colleagues investigated ChatGPT’s diagnostic accuracy and utility in medical education. The analysis included 150 Medscape case challenges (September 2021 to January 2023) that were inputted into ChatGPT.
The researchers found that ChatGPT answered 49 percent of cases correctly, with an overall accuracy of 74 percent, a precision of 48.67 percent, sensitivity of 48.67 percent, specificity of 82.89 percent, and an area under the curve of 0.66. ChatGPT struggled with the interpretation of laboratory values and imaging results, but was generally correct in ruling out a specific differential diagnosis and providing reasonable next diagnostic steps. Just over half of answers were complete and relevant (52 percent) and a similar percentage of answers were characterized as low cognitive load (51 percent).
"Additional research should focus on enhancing the accuracy and dependability of ChatGPT as a diagnostic instrument," the authors write. "Integrating ChatGPT into medical education and clinical practice necessitates a thorough examination of its educational and clinical limitations. Transparent guidelines should be established for ChatGPT’s clinical usage, and medical students and clinicians should be trained on how to effectively and responsibly employ the tool."
Disclaimer: Statistical data in medical articles provide general trends and do not pertain to individuals. Individual factors can vary greatly. Always seek personalized medical advice for individual healthcare decisions.
© 2025 HealthDay. All rights reserved.
Posted August 2024
Read this next
Racialized Economic Segregation Linked to Advanced Cancer Diagnosis
FRIDAY, Aug. 1, 2025 -- People living in racially and economically segregated neighborhoods are more likely to be diagnosed with advanced-stage breast and cervical cancer...
New Subtype of Diabetes Identified in Sub-Saharan Africans, Black Americans
FRIDAY, Aug. 1, 2025 -- A new diabetes subtype has been identified in Sub-Saharan Africans and Black Americans, according to a study published online July 21 in The Lancet...
Changes in Missouri Law Linked to Increased Breast Cancer Screening Rates
THURSDAY, July 31, 2025 -- Changes in Missouri law expanding breast cancer screening coverage have led to increased screening rates among Medicaid patients and an increased...
More news resources
- FDA Medwatch Drug Alerts
- Daily MedNews
- News for Health Professionals
- New Drug Approvals
- New Drug Applications
- Drug Shortages
- Clinical Trial Results
- Generic Drug Approvals
Subscribe to our newsletter
Whatever your topic of interest, subscribe to our newsletters to get the best of Drugs.com in your inbox.