As medical diagnostics increasingly use artificial intelligence (AI), a new study is causing concerns regarding its long-term consistency. Released on Dec. 20, the study indicates that large language models (LLMs) and AI-based chatbots could suffer from cognitive decline, similar to human beings.
This discovery undercuts the idea that human doctors will be quickly replaced by AI and identifies possible threats in medical diagnostics.
AI Made Medical Diagnostics Faster
AI has revolutionized healthcare by rapidly analyzing medical histories, X-rays, and other datasets to detect anomalies before they become apparent to the human eye.
These advancements have significantly improved diagnostic speed and accuracy. However, a recent study suggests that AI-powered chatbots, including OpenAI's ChatGPT, Anthropic's Sonnet, and Alphabet's Gemini, may lose their effectiveness over time, according to Live Science.
Read more: Chrome Extension 'FixTheGulf' Defies Trump's Claim of Renaming Gulf of Mexico to Gulf of America
Testing AI for Cognitive Abilities
To determine if AI systems suffer from cognitive decline, researchers made use of the Montreal Cognitive Assessment (MoCA) test. This test, commonly employed by neurologists to assess cognitive ability in humans, quantifies capabilities in areas including:
- Attention and Memory
- Language and Abstraction
- Spatial Skills and Executive Function
The test involves such activities as repeatedly subtracting seven from 100, remembering items from a read list, and marking a given time on the face of a clock. For humans, 26 out of 30 represents normal cognitive status, and less than that might indicate cognitive deficit.
How AI Chatbots Scored
The results of the BMJ study showed a remarkable trend. ChatGPT-4 had the highest score at 26 out of 30, reaching the human level of normal cognitive functioning. Gemini 1.0 was far less successful, with a score of just 16, indicating deterioration in cognitive function with older versions of LLM.
Whereas AI performed best on naming, attention, and language tests, it did poorly on spatial and executive function tests, especially delayed recall and abstract thinking.
Does AI Truly Suffer Cognitive Decline?
While the results point to decreased performance in older AI models, scientists warn the findings are observational. How the AI processes data is inherently unlike the human brain, so the two cannot easily be compared. The study still suggests that there may be flaws in AI systems that could prevent them from functioning fully in medical diagnosis.
One of the primary concerns is that AI performs badly in visual abstraction and executive function tasks—two domains that are essential for sound medical diagnosis. If AI chatbots cannot dependably decode complex visual information, their application in clinical environments could be undermined.
Implications for AI in Healthcare
These results have glaring implications for the future of AI in medicine. While AI is an incredibly powerful tool, its limitations need to be overcome before it can be entirely relied upon in important diagnostic functions. Some of the main points are:
AI should not substitute human physicians but help them. AI can rapidly scan enormous quantities of data, but human intervention is required for proper interpretation.
Regular updates and retraining are necessary. As older AI models are less effective, updating LLMs is important to ensure accuracy.
AI might need "neurological check-ups." The humorous idea of AI itself being given cognitive tests opens the door to regular checks to guarantee ongoing reliability.
A Reality Check for AI Users
Though AI has come a long way in medicine, this study reminds us that technology is not perfect. The suggestion that AI could suffer from "cognitive decline" makes it clear that it needs to be fine-tuned and supervised by humans.
