ChatGPT's Medical Knowledge Evaluated: How Well Does It Answer Health Questions?

ChatGPT shows promise as a source of medical knowledge, according to a recent study. Researchers discovered that the answers provided by the AI-powered language model to 280 medical queries were mostly correct, though it is not yet 100 percent trustworthy.

The positive marks of ChatGPT were observed by Dr. Douglas Johnson, senior researcher at the Vanderbilt-Ingram Cancer Center. When a second AI software examined its replies, accuracy increased.

According to US News & World Report, 33 doctors from various specialities designed the questions for the study. On a scale of 1 to 6, with 5 being easy, 4.7 being medium, and 4.6 being tough, ChatGPT received an average score of 4.8. On a 3-point scale, it received a 2.5 for providing responses that were mostly comprehensive.

Better accuracy ratings were given to several specialty. When it came to answering "yes/no" questions, ChatGPT excelled.

ChatGPT Provides Complete and Accurate Answers to Health Questions

Overall, both accuracy and completeness were quite well performed. Despite its flaws and partial dependability, it provided information that was comparatively accurate and thorough in light of the study's open-ended questions.

Notably, ChatGPT excelled in specific medical disciplines, earning an accuracy score of 5.7 for queries about common ailments and 5.2 for queries about melanoma and immunotherapy.

Additionally, research determined that ChatGPT's comments were quite comprehensive, earning a score of 2.5 on a 3-point scale for completeness.

Between January and May 2023, the researchers-led by Dr. Johnson-started this investigation to gauge ChatGPT's competency in responding to medical questions. The researchers wanted to study the possibilities of AI systems like ChatGPT in the realm of medicine as individuals and healthcare professionals currently use search engines like Google and Bing for medical information.

Not a Substitute for Human Medical Experts

Even though ChatGPT provides insightful information, experts cautioned that it cannot take the position of medical specialists. AI can be a useful tool for physicians and patients, but for it to be used in an ethical and trustworthy way, additional research and regulatory control are required.

Some medical experts voice their worries about AI chatbots, noting problems with prejudice, liability, transparency, privacy, safety, and the existing lack of regulatory control. Mason Marks, a Florida State University health law expert, finds this strategy unsatisfactory and issues a warning against using AI as a fast answer for more serious systemic issues.

"Companies might want to push a particular product over another. "The potential for exploitation of people and the commercialization of data is unprecedented," according to Marks, as quoted by NPR.

To ensure reliable information and solve these concerns, experts at Georgetown University suggests that medical AI models like ChatGPT must be regulated. The formation of regulatory agencies tasked with overseeing the development and application of AI models in healthcare is of utmost relevance. These regulatory agencies have the power to impose strict requirements on the accuracy, reliability, and security of AI models. They can also assure ethical and transparent model development.

"There are entities like the European Union and China that are trying to regulate AI over concerns of privacy, bias, and fake news generation. Considering similar problems in the United States, the US government should also take steps in this direction," as quoted in the article published by the institution.