ChatGPT New 'Code Interpreter' Plugin Put to the Test by Researchers — Is It Effective?

The newly introduced Code Interpreter, the latest official plugin for ChatGPT, has caught the attention of West Virginia University (WVU) researchers who see its potential in educational contexts.

However, their evaluation has revealed that the Code Interpreter has certain limitations, particularly when applied to biomedical research and bioinformatics.

Pros and Cons of Code Interpreter for ChatGPT

Gangqing "Michael" Hu, an assistant professor in the Department of Microbiology, Immunology and Cell Biology at the WVU School of Medicine and director of the Bioinformatics Core, expressed his perspective on the plugin's functionality.

While acknowledging its value in making coding in STEM fields more accessible to students and aiding learning, he pointed out that the plugin falls short in addressing the specific needs of the bioinformatics domain.

"However, it doesn't have the features you need for bioinformatics. These are technical issues that can be overcome. Future developments of Code Interpreter are likely to extend its use to many fields such as bioinformatics, finance and economics," Hu said in a statement.

Hu and his team embarked on a thorough assessment of the Code Interpreter plugin to gauge its effectiveness. Their findings highlight both its merits and limitations.

One of the plugin's strengths lies in its ability to provide easy access to coding for individuals without a science background. It offers a cost-effective solution and fosters curiosity among students, sparking their interest in data analysis.

However, users must possess the ability to interpret data accurately, recognize the reliability of results, and effectively interact with the chatbot, according to the team. Bioinformatics, which involves the use of coding, software, and data interpretation for biomedical research, necessitates precision.

The team found that while Code Interpreter aids in minimizing inaccuracies known as hallucinations -where responses present misinformation - it falls short in some key areas specific to bioinformatics.

The research team also found that the plugin excelled in converting data into charts and graphs, enhancing data visualization capabilities. However, they identified several areas for improvement.

Suggestions for ChatGPT Plugin, the Code Interpreter

The team's suggestions included incorporating internet access for downloading genome data, integrating software tailored for bioinformatics, expanding storage capacity, supporting additional programming languages, enhancing privacy and security features to comply with regulations, and facilitating access to larger datasets.

While the Code Interpreter plugin currently supports only one programming language, Python, and lacks seamless internet data access, Hu envisions its potential in education.

He plans to introduce the plugin to his class to teach students about data visualization and coding despite its current limitations. Hu acknowledges the rapid evolution of the AI field and remains open to exploring innovative uses and advancements in AI programming and features.

Reflecting on his continuous involvement in testing and monitoring new AI products, Hu remains enthusiastic about the untapped potential that awaits discovery.

"AI is a fast-moving field. I hope by that time OpenAI may overcome some of the limitations so it can be used for a broad range of bioinformatics coding," he noted.

The team's findings were published in the journal Annals of Biomedical Engineering.