ChatGPT Generates Persuasive but Phony Medical Report - Study Claims

Italian researchers have discovered that ChatGPT, the popular language model developed by OpenAI, has the ability to generate persuasive yet fictitious medical data, raising concerns about the potential misuse of artificial intelligence in medical research, as reported in Tech Xplore.

Giuseppe Giannaccare, an eye surgeon at the University of Cagliari, led the research team that found ChatGPT, specifically the GPT-4 version, capable of creating a fake dataset supporting one surgical eye procedure over another in a matter of minutes.

This discovery sheds light on the potential darker side of AI, highlighting the ease with which AI models can generate and manipulate data to produce biased results and false medical evidence.

ChatGPT's Hallucination

While ChatGPT has been celebrated for its capabilities, there have been instances of the model generating inaccurate or false information. The Cambridge Dictionary even named "hallucinate" as the word of the year, referring to the tendency of large language models to spontaneously produce false information.

However, it is worth noting that OpenAI has been open about the chatbot's capability to hallucinate. ChatGPT's website displays a disclaimer saying, "ChatGPT can make mistakes. Consider checking important information."

The researchers guided GPT-4, connected to a sophisticated Python-based data analysis model, in the creation of clinical trial data for two methodologies aimed at addressing keratoconus, a prevalent eye disorder.

Through detailed prompts encompassing information about eye conditions, subject statistics, and outcome criteria, the researchers steered the model to endorse one procedure over the other, constructing a compelling argument grounded in entirely fabricated information.

The implications of such capabilities extend beyond academic nuisances or financial penalties. In the medical field, where decisions are made based on research outcomes, the potential for AI-generated false data to influence medical procedures is a serious concern.

"It was one thing that generative AI could be used to generate texts that would not be detectable using plagiarism software, but the capacity to create fake but realistic data sets is a next level of worry," said Elisabeth Bik, a research-integrity consultant in San Francisco.

"It will make it very easy for any researcher or group of researchers to create fake measurements on non-existent patients, fake answers to questionnaires or to generate a large dataset on animal experiments."

Closer Scrutiny

The study, titled "Large Language Model Advanced Data Analysis Abuse to Create a Fake Data Set in Medical Research," acknowledges that closer scrutiny could reveal signs of possible fabrication in the data.

The researchers also emphasize the need for the scientific community to develop better approaches for fraud detection in response to the potential misuses and threats connected to AI.

Giannaccare underscores the dual nature of AI, acknowledging its potential benefits for scientific research while cautioning against its misuse.

The discovery prompts reflections on how the scientific community will address the challenges posed by AI-generated data and maintain academic integrity in the face of evolving technologies.

The findings of the research team were published in the journal JAMA Ophthalmology.