ChatGPT accuracy for breast cancer screening advice

The newly available ChatGPT provides correct answers most of the time, but sometimes gives inaccurate or fictious information, according to a recent study conducted by researchers at the University of Maryland School of Medicine (UMSOM) and published in Radiology.

Researchers from UMSOM developed 25 questions related to breast cancer screening in February 2023. Each question was submitted to ChatGPT 3 times, as it is known for providing varying answers each instance a question is given.

Responses were analyzed by 3 radiologists fellowship-trained in mammography. Of the 25 questions, ChatGTP responded to 22 with accurate information. One answer contained outdated information, while the other 2 had widely varying responses each time the question was posed.

Correct answers were provided by ChatGTP 88% of the time, which Paul Yi, MD, study corresponding author and assistant professor of diagnostic radiology and nuclear medicine at UMSOM, considered impressive. Yi also stated the information provided was summarized in a form easily digestible for consumers.

However, the responses from ChatGTP were not as comprehensive as those which could be given through a Google search.

“ChatGPT provided only 1 set of recommendations on breast cancer screening, issued from the American Cancer Society, but did not mention differing recommendations put out by the Centers for Disease Control (CDC) and Prevention or the US Preventative Services Task Force (USPSTF),” said Hana Haver, MD, lead study author and radiology resident at University of Maryland Medical Center.

Once inappropriate response from ChatGPT contained outdated information on planning a mammogram around COVID-19 vaccination. ChatGPT recommended delaying a mammogram 4 to 6 weeks after receiving a COVID-19 vaccine dose. However, in February 2022, this guideline was changed, and CDC has endorsed USPSTF guidelines to not wait.

Yi noted ChatGPT will sometimes support its claims through made up journal articles and health consortiums. He also reminded consumers to rely on their doctor for advice before ChatGPT. 

When analyzing ChatGPT’s responses on breast cancer risk and where individuals may receive a mammogram, it was found that ChatGPT gave inconsistent responses. Currently, Yi and his colleagues are evaluating how ChatGPT handles recommendations for lung cancer screening, studying ways to make the chatbot more accurate and complete.

Reference

ChatGPT helpful for breast cancer screening advice with certain caveats, new study finds. EurekAlert. April 4, 2023. Accessed April 5, 2023. https://www.eurekalert.org/news-releases/984922