Skip to main content

Poor Performance for ChatGPT on AUA Self-Assessment Study Program

Medically reviewed by Drugs.com.

By Elana Gotkine HealthDay Reporter

TUESDAY, June 27, 2023 -- ChatGPT performs poorly on the 2022 American Urological Association Self-Assessment Study Program, according to a study published online June 5 in Urology Practice.

Linda My Huynh, from the University of Nebraska Medical Center in Omaha, and colleagues assessed use of ChatGPT on the American Urological Association Self-Assessment Study Program as an educational adjunct. A total of 135 questions from the 2022 Self-Assessment Study Program exam were screened. ChatGPT's output was coded as correct, incorrect, or indeterminate; responses were regenerated up to two times if indeterminate.

The researchers found that ChatGPT was correct on 26.7 percent of open-ended and 28.2 percent of multiple-choice questions (36 and 38 questions, respectively). Indeterminate responses were generated in 29.6 and 3.0 percent of open-ended and multiple choice questions, respectively. Of the correct responses, 66.7 and 94.7 percent were on initial output; 22.2 and 2.6 percent on second output; and 11.1 and 2.6 percent on final output, respectively. Indeterminate responses decreased with regeneration, but there was no increase in the proportion of correct responses. ChatGPT provided consistent justifications for incorrect answers for open-ended and multiple-choice questions, and remained concordant between correct and incorrect answers.

"While ChatGPT provided explanations that were well written, easy to read, and concordant with the selected answer choice, these explanations were also lengthy and lacked mechanistic or pathophysiological justifications," the authors write. "As is, utilization of ChatGPT in urology has a high likelihood of facilitating medical misinformation for the untrained user."

Abstract/Full Text

Editorial 1 (subscription or payment may be required)

Editorial 2 (subscription or payment may be required)

Disclaimer: Statistical data in medical articles provide general trends and do not pertain to individuals. Individual factors can vary greatly. Always seek personalized medical advice for individual healthcare decisions.

© 2024 HealthDay. All rights reserved.

Posted June 2023

Read this next

First Responders With More Debris Exposure Have Higher Risk of Early Dementia

FRIDAY, June 14, 2024 -- More severe exposure to dust or debris among World Trade Center (WTC) responders is significantly associated with a higher risk of dementia at <65...

1990 to 2019 Saw Increase in Life Expectancy in Seniors With T1DM

FRIDAY, June 14, 2024 -- From 1990 to 2019, there was an increase in life expectancy in older people with type 1 diabetes mellitus (T1DM), according to a study published online...

Second Tumors Are Rare After Chimeric Antigen Receptor T-Cell Therapy

FRIDAY, June 14, 2024 -- Second tumors are rarely seen after chimeric antigen receptor (CAR) T-cell therapy, according to a study published in the June 13 issue of the New England...

More news resources

Subscribe to our newsletter

Whatever your topic of interest, subscribe to our newsletters to get the best of Drugs.com in your inbox.