ChatGPT Sucks at Accounting, Humans Still Better According to Study

In the age of artificial intelligence, it is no surprise that machines are rapidly becoming smarter and more advanced. However, a recent study shows that when it comes to accounting, humans still reign supreme.

The study, conducted by Brigham Young University and 186 other universities, pitted ChatGPT against actual university accounting students. The researchers wanted to see how the AI would fare against real human beings, and the results were eye-opening.

ChatGPT Needs More Work

While ChatGPT has certainly made waves since its debut in November 2022, boasting an impressive 100 million users in under two months, it seems that its accounting skills could use some work.

In fact, according to the study, students performed better than ChatGPT overall, with an average score of 76.7% compared to the bot's score of 47.4%.

Of course, ChatGPT did have some moments of glory. It performed particularly well on accounting information systems (AIS) and auditing questions, scoring higher than the student average on 11.3% of questions.

However, when it came to tax, financial, and managerial assessments, ChatGPT struggled with the mathematical processes required for these types of questions.

ChatGPT also struggled with certain types of questions. It performed better on true/false questions and multiple-choice questions but stumbled on short-answer questions.

In fact, higher-order questions proved to be more challenging for the chatbot to answer. The researchers also noted that ChatGPT sometimes provided incorrect answers along with authoritative written descriptions, or provided accurate descriptions but selected the wrong multiple-choice answer.

Despite these shortcomings, the researchers behind the study see great potential for ChatGPT in the future. They believe that it could be used to improve teaching and learning and to help design and test assignments.

"Fool's Errand"

While ChatGPT may not be perfect, it is clear that the technology is rapidly advancing and is sure to improve exponentially in the coming years.

The study's lead author, David Wood, emphasizes that the researchers are not trying to focus on the negative aspects of ChatGPT's performance. Instead, they want to highlight what the technology can do now that it could not do before.

"We're trying to focus on what we can do with this technology to improve the teaching process for faculty and the learning process for students," he said in a press release statement.

In the end, it seems that while ChatGPT may have some catching up to do in the realm of accounting, there is no denying that it's a game changer that will change the way we teach and learn. As Jessica Wood, one of the students who participated in the study, puts it, "Trying to learn solely by using ChatGPT is a fool's errand."

Despite the challenges uncovered in the study, the authors remain optimistic that GPT-4 will significantly enhance its performance on accounting questions and address the concerns they mentioned.

They are particularly encouraged by the potential of the chatbot to enhance the education experience, such as facilitating the creation and assessment of assignments or potentially aiding in the creation of project drafts.

The findings of the study were published in the journal Issues in Accounting Education.