In nearly every aspect that counts, receiving condensed information from AI models was found to be less informative than conducting a search independently. Since ChatGPT was introduced in late 2022, a large number of individuals have turned to advanced language models for accessing information. The appeal is clear: pose a question, receive a refined summary, and move on – it gives the impression of effortless learning.
Nonetheless, a recent paper that I collaborated on presents experimental proof that this convenience might have drawbacks: When individuals depend on large language models to summarize information rather than researching through a standard Google search, they tend to acquire superficial knowledge about the subject. In our study, co-authored with Jin Ho Yun, both marketing professors, we shared this discovery based on seven studies involving over 10,000 participants.
The majority of the studies followed a similar structure: Participants were tasked with learning about a topic – like cultivating a vegetable garden – and were randomly assigned to utilize either an LLM such as ChatGPT or the traditional method of navigating search results using Google.
Participants were free to use the tools as they wished; they could browse Google for as long as needed and could continue seeking information from ChatGPT if desired. Subsequently, they were asked to offer advice to a friend on the topic based on their research.
Consistent patterns emerged from the data: Individuals who learned about a topic via an LLM rather than a web search felt that they gained less knowledge, put in less effort into crafting their advice, and ultimately produced advice that was concise, less factual, and more generic. When this advice was reviewed by an independent group of readers unaware of the learning method used, they deemed it less informative and helpful and were less likely to adopt it.
These distinctions remained consistent across various scenarios. For instance, one reason LLM users provided shorter and more generic advice could be that LLM results exposed them to less diverse information compared to Google results. To address this possibility, we conducted an experiment where participants received identical facts in both their Google and ChatGPT searches.
Similarly, in another experiment we kept the search platform constant – Google – but varied whether participants learned from standard Google results or Google’s AI Overview feature.
The results confirmed that even when keeping the facts and platform consistent, learning from synthesized LLM responses led to shallower knowledge compared to gathering and interpreting information independently through standard web links.
Why did using LLMs seem to hinder learning? One fundamental principle of skill acquisition is that individuals learn best when actively engaging with the material they are trying to grasp. Conducting a Google search involves more “friction”: navigating different sources, reading materials, and interpreting information independently.
Though more challenging, this friction contributes to forming a deeper understanding of the topic. However, with LLMs handling this process on behalf of users, learning shifts from active participation to passive reception.
To clarify, we do not advocate avoiding LLMs as they offer undeniable advantages in many situations. Instead, we suggest becoming more discerning users of LLMs by understanding when they are beneficial or detrimental to one’s objectives.
If you need quick factual answers, feel free to consult your preferred AI assistant. But if your goal is to develop profound and transferable knowledge in a field, relying solely on LLM summaries may be less beneficial.
As part of my research on new technology psychology and new media trends, I am exploring ways to make LLM learning more engaging. In one experiment testing this concept, participants interacted with a specialized GPT model that presented real-time web links alongside its synthesized responses.
However, we observed that once participants received an LLM summary, they lacked motivation to delve deeper into original sources. Consequently, these participants still gained shallower knowledge compared to those who used standard Google searches.
In future research endeavors, I plan to investigate generative AI tools that introduce productive frictions for learning tasks – particularly exploring which types of constraints best encourage users to actively seek additional knowledge beyond simple synthesized responses.
Such tools could be especially valuable in secondary education where educators face the challenge of preparing students for essential skills while acknowledging that LLMs are likely to play a significant role in their daily lives.