• All News
• Consumer
• Pro
• New Drugs
• Pipeline
• Clinical Trials
• FDA Alerts

How Accurate Is ChatGPT When Asked About Your Health?

By Dennis Thompson HealthDay Reporter

Medically reviewed by Carmen Pope, BPharm. Last updated on Aug 5, 2025.

TUESDAY, Aug. 5, 2025 — ChatGPT is likely to be hit-or-miss when it comes to figuring out symptoms for a particular illness, a new study says.

The AI program has 49% to 61% accuracy when it comes to identifying symptoms associated with specific diseases, researchers reported recently in the journal iScience.

This might have to do with how AIs like ChatGPT are trained, said lead researcher Ahmed Abdeen Hamed, a research fellow in systems science and industrial engineering at Binghamton University, State University of New York.

ChatGPT is a large language model, a type of AI that learns by analyzing vast amounts of text. But it might be tripping over the medical jargon used by doctors, Hamed said in a news release.

“ChatGPT uses more of a friendly and social language, because it’s supposed to be communicating with average people,” Hamed said. “In medical literature, people use proper names.”

For example, ChatGPT stumbles when matching “sad mood” to “depression,” “facial redness” to “inflammation,” or “seizures” to “febrile convulsion,” the study says.

“The LLM is apparently trying to simplify the definition of these symptoms, because there is a lot of traffic asking such questions, so it started to minimize the formalities of medical language to appeal to those users,” Hamed said.

On the other hand, ChatGPT was great at identifying things for which the medical jargon matches everyday lingo, such as disease terms, drug names and genetic information, researchers said.

Hamed had thought ChatGPT would have “at most 25% accuracy” when tested on those matters.

But the AI was 88% to 97% accurate at identifying disease terms, 90% to 91% accurate in understanding drug names, and 88% to 98% accurate at assessing genetic information, results show.

“The exciting result was ChatGPT said cancer is a disease, hypertension is a disease, fever is a symptom, Remdesivir is a drug and BRCA is a gene related to breast cancer,” Hamed said. “Incredible, absolutely incredible!”

Hamed said he is investigating AI’s ability to answer medical questions, because many are turning to the programs for quick advice.

“People talk to ChatGPT all the time these days, and they say: ‘I have these symptoms. Do I have cancer? Do I have cardiac arrest? Should I be getting treatment?’” Hamed said.

“It can be a very dangerous business, so we wanted to see what would happen if we asked these questions, what sort of answers we got and how these answers could be verified from the biomedical literature,” Hamed said.

Hamed’s goal is to expose these flaws so the models can be adjusted for better responses.

“If I am analyzing knowledge, I want to make sure that I remove anything that may seem fishy before I build my theories and make something that is not accurate,” he said.

For example, ChatGPT was completely flummoxed when asked questions about data stored in GenBank, a National Institutes of Health database that assigns a number to every identified DNA sequence, such as NM_007294.4 for the breast cancer gene BRCA1.

When asked for these numbers as part of its testing on genetic info, ChatGPT just made them up out of whole cloth, a phenomenon known as “hallucinating,” Hamed said.

“Maybe there is an opportunity here that we can start introducing (medical jargon) to the LLMs to provide much higher accuracy, get rid of all the hallucinations and make these tools into something amazing,” he said.

Disclaimer: Statistical data in medical articles provide general trends and do not pertain to individuals. Individual factors can vary greatly. Always seek personalized medical advice for individual healthcare decisions.

© 2025 HealthDay. All rights reserved.

Posted August 2025

More news resources

• FDA Medwatch Drug Alerts
• Daily MedNews
• News for Health Professionals
• New Drug Approvals
• New Drug Applications
• Drug Shortages
• Clinical Trial Results
• Generic Drug Approvals

Subscribe to our newsletter

Whatever your topic of interest, subscribe to our newsletters to get the best of Drugs.com in your inbox.