Can AI Replace Humans in Future Market Research? New Study Has Some Answers

A recent study explored whether artificial intelligence (AI) can effectively replace humans in market research endeavors, shedding light on potential shifts in traditional research methodologies.

The study examines the viability of utilizing large language models (LLMs) as substitutes for human participants in generating research outputs.

Using AI in Market Research

Led by researchers Peiyao Li and Zsolt Katona from the University of California, Berkeley, Noah Castelo from the University of Alberta, and Miklos Sarvary from Columbia University, the study delves into automated market research facilitated by advanced AI technologies.

The study highlights that LLMs, leveraging their advanced linguistic capabilities, can produce outputs comparable to those derived from human surveys. By analyzing agreement rates between human- and LLM-generated data sets, researchers observed agreement rates ranging from 75% to 85%.

Peiyao Li emphasizes the versatility of LLMs in text generation, particularly in the context of automated market research. LLMs can effectively interpret prompts and generate text responses, offering a novel approach to perceptual analysis across various product categories.

The research methodology devised by the study authors involves harnessing readily available data from the internet, allowing market researchers to rely solely on LLMs for conducting research.

Zsolt Katona underscores the foundational principle that LLMs, while eliminating the need for direct human interviews, ultimately derive their data from human sources.

"It is important to note that with LLMs, while market researchers may not require interviews with human research subjects, the ultimate data does originate from human beings, using available data," Katona said in a press release statement.

"LLMs have been engineered to accurately replicate human responses based on machine learning of actual human perceptions, attitudes, and preferences."

The Core LLM

Noah Castelo explains the core functionality of LLMs, which involves processing prompts and generating coherent textual continuations.

"The core LLM takes a prompt as an input and generates a continuation of text as output. With proper prompting, the LLM can then generate comparisons and assessments of various brands or products in a given category and produce results that are, at the moment, 75%-85% in agreement with research featuring human participants," Castelo said.

Despite the promising potential of LLM-powered market research, the study authors caution against complete reliance on automation, particularly in scenarios where human input remains indispensable.

While automation may enhance efficiency and reduce costs, its applicability across all product categories remains subject to scrutiny.

"While we are very excited about the possibilities we've seen through our research, we recognize that this is just the beginning and going forward, LLM-based market research will be able to answer more nuanced questions as the market research field begins to tap and develop its potential," says Sarvary.

So, can AI replace humans in future market research? The researchers said "not always".

The findings of the study were published in Marketing Science.