AI Slop and the Expert's Paradox: Cognitive Biases in the Era of Generative Search

In the article Activism in the Shadow of Cognitive Biases and Confirmation Bias, I described the case of Adam Pustelnik and the cognitive bias mechanism resulting from Google’s Rich Results. That error was, let’s say, “analog”—it stemmed from how people scan search results on a screen without delving deeper. Today, we face a much more serious problem: AI as an active generator of confabulations, not just a passive aggregator of links. And since a revolution in SERPs and the internet is now underway, it’s worth taking a closer look at this issue.

The “EFF Everbridge” Case: When AI Creates Non-Existent Concepts

I am currently writing an article about emergency notification systems in the USA. Everbridge is a company that provides platforms for managing critical events. These include mass SMS alerts and notifications about natural disasters. EFF is, of course, the Electronic Frontier Foundation, an organization dedicated to digital liberties and privacy.

I decided to check what Google’s SERP (search engine result page) would show for the simple query “EFF everbridge.” I was looking for materials on potential privacy controversies related to the mass data collection by alert systems. Two phrases, mashed together exactly as most people search.

I received what most people get now: an “AI Overview” at the top of the results, telling me that “EFF Everbridge is a Critical Event Management (CEM) system.” Google AI combined the two phrases and generated a non-existent concept that sounds like an established technical term. For someone without context, this answer looks completely credible. Of course, a simple check of the first letters of the English term shows that CEM is not EFF. I am familiar with the scopes and definitions of both terms, so for me, the AI slop was clear.

Only by entering the AI mode (the second tab) did I get the correct answer:

Here, Google clearly distinguishes: EFF is a non-profit organization fighting for digital freedoms, and Everbridge is a technology company. It adds the key information: “no official partnership or joint project under the name EFF Everbridge has been recorded.” I personally believe that, as with query optimization at the GEMINI PRO level, we are partly dealing with caching and routing queries to different models (like Flash), which causes a drastic semantic disconnect at the basic level (I described this in my text Gemini PRO is broken). Simply put, the model lacks the computational power to catch a simple error like CEM not equaling EFF. Only by going deeper do additional mechanisms get activated that capture the contexts separately.

The Problem: Keyword-Based Searching in a World of Generative Answers

For over 20 years, Google has trained users to search using keywords. This drama of SERP searching and result hacking exists, but most people prefer simplicity. They don’t really know how to use operators or control semantics. People take shortcuts. You just type in a few phrases and get a list of links. The recipient evaluates the sources themselves.

How much do people take shortcuts? Data from Semrush’s State of Search 2022 (an analysis of 160 million keywords) shows the scale of the problem:

Over half of all queries are one or two words. My “EFF everbridge” fits into the dominant category—and that’s precisely why it’s representative of how most people use search engines.

What’s more, a Backlinko study (1801 user sessions) shows how users interact with the results:

Nine seconds to the first click. Nine. That’s not enough time to analyze sources—it’s scanning and reacting to the first result that looks sensible. And now, that first result is an “AI Overview” occupying the top of the screen.

In the article about the digital palimpsest, I described in detail the AI prediction model that builds the model recipient in Google (BERT/MUM), which is why these search results are so contextually tailored. This model worked, and cognitive biases arose from user laziness (scanning headlines instead of reading), but at least the responsibility for interpretation lay with the human. Spammers and the entire marketing industry benefited from this.

Now, the same jumble of phrases generates an authoritatively sounding narrative. For most people, this featured snippet, which has gained a reputation greater than universities, is a significant and fundamental difference. Previously, Google said, “here are pages that might contain the answer”; now it says, “here is the answer.” And that answer can be pure confabulation.

The Facade of Credibility: What AI Search Inherits from Google

The habits formed by 20 years of keyword searching, thanks to the monopolist Google, are carrying over to generative search engines, but with more serious consequences. Researchers from Stanford University audited four popular systems (Bing Chat, NeevaAI, perplexity.ai, YouChat), analyzing 1450 queries from various sources.

The key finding for us is the correlation between perceived utility and citation precision, which is r = -0.96. The more “helpful” an answer seems, the more likely it is to contain unsupported claims. Systems that copy from sources (high precision) are perceived as less useful. This happens because users prefer “creative” synthesis, which, however, more often deviates from the truth.

But let’s not be too harsh on algorithms and AI, because we owe this mechanism of inheritance to Google, which for years trained us to scan quickly and trust the first result. AI search produces fluent, convincing text that the user does not verify because it “looks credible” and is easy to read.

The Stanford research cited here, from May 2023, in my opinion, shows just the tip of the iceberg: about 50% of generative search engine responses have no source citations, and 25% of the provided citations do not actually support the claims made. This is changing, but let’s be honest, how many people click on the context and the next result? The researchers discovered an inverse correlation: “the most fluent and convincing answers are the least likely to be supported by sources, while the most clumsy answers are better documented.” Does this stem from semantics and the fact that smooth words and corporate jargon are precisely the AI-style, while clumsy and thus unconventional statements are better processed as entities by AI?

Something that many people don’t understand is the technique of inverting intuition or, in other words, testing paranoid interpretations and falsifying them. In the normal world, the more convincingly something sounds, the more we trust it. In the world of AI slop, we should use the exact opposite heuristic. However, with an emphasis on falsification, i.e., verification.

The Expert’s Paradox: To Know AI is Lying, You Already Have to Know Something

And it is precisely this falsification, which I write so much about in the context of conspiracy theories, that brings us to the heart of the epistemic problem. I knew that EFF was the Electronic Frontier Foundation. I have been following their publications and activities for years. I knew that Everbridge was a separate company. That’s why I immediately saw the confabulation. But what if I didn’t have this prior knowledge? Of course, an attentive observer would see the “CEM does not equal EFF” error I mentioned earlier.

For a layperson, “EFF Everbridge as a critical event management system” sounds like a description of an established, existing concept. Not the fantasy of a statistical system. The confabulation is dressed in the language of facts. I think that people like the journalist Karolina Opolska, who allegedly wrote a book by copying from AI (I’m basing this on press reports and expert publications), are not just taking the easy way out, but probably simply lack the competence.

This is the expert’s paradox: to immediately identify AI slop, you must already possess the knowledge you are seeking. Without it, you need critical competence and time for verification—and in the era of 9-second scanning, few can afford either. On an epistemic, cognitive level, this is a catastrophe.

The Shift of Authority: From Sources to the System

If we go further, we arrive at something I observe with growing concern. Authority is shifting from people and sources that can justify something to a system that reproduces statistical patterns without the ability to verify them. That’s why I’ve been saying for two years now that the time for storytellers and people with narrow specializations is over. The time has come for people with a surplus of knowledge and competence.

Musk solved the common man’s problem; he made a brilliant move. I have to admit it, because “brilliant” from his perspective, of course, as from the world’s and ethics’ point of view, the move is questionable, but given the scale of the changes we are observing, Musk is a symptom. What did he do? On the one hand, he systematically dismantles traditional authorities, creating a sense that he is fighting for free speech and independent research. He brutally attacks the media, experts, institutions, journalists, and scientists. On the other hand, he created Grok, which is an element of an algorithmic meta-authority, supposedly “autonomous” and “uncensored.” Grok is like that cool professor friend with whom you can chat about memes, but who will also explain to you whether something is true or not. Grokopedia, a replacement for Wikipedia, which Musk hates, is another layer of this process, which Musk introduced as an alternative knowledge base that is better than the “woke-Wikipedia.”

A pluralism of sources passes through a single filter. This is a concentration of knowledge in the hands of “philosopher kings,” about whom I wrote earlier in reference to Plato. These new philosopher kings do not so much possess knowledge as control semantics and meanings. They tell new myths. They decide what is “truth,” not through argumentation, but through the architecture of the system. It is worth adding that Plato also introduced the concept of the noble lie, meaning that those who control semantics-myths have the right to bend facts to manage the rabble.

Google’s Authority is Greater Than a University’s

And here, if we are considering epistemology, i.e., cognitive possibilities and cognitive biases, it is worth referring to the study “Epistemic Injustice in Generative AI,” where the concept of “amplified testimonial injustice” appears. This means we are currently at a stage of internet development where AI amplifies and replicates false narratives on a massive scale. According to the authors, the system rarely creates new lies — it more often reinforces existing errors and biases from the training data. And it does so with the authority of an algorithm, which for many users is greater than the authority of a human expert. Many experts have written extensively about the potential dangers associated with this, which I address in other places, including in the text about the manipulation of recommendation systems. However, it is worth noting that the authors of the aforementioned text call what we are describing “hermeneutical access injustice”—unequal access to information and knowledge mediated by AI. When one system becomes the gateway to knowledge, control over that system becomes control over what people can know.

Perfectly said, and in many minds, the name Foucault, who said that power is knowledge, probably now appears. Well, the semblance of knowledge, and especially controlling the narrative and semantics, is an open field for control. This is a meta-level, but at the most elementary level of the screen and keyboard, we have exclusion caused by the competence of the questioner (who is also the recipient of the message). In my opinion, we are dealing here with an asymmetry of knowledge or informational asymmetry, which I have written about many times, pointing out how access to knowledge and information can exclude people from processes, for example, at the local level.

What to Do About Informational Asymmetry in the Age of AI Slop?

Education and improving competencies is good advice, like telling a seriously ill person to take care of their health and exercise. I do not intend to end this text with nihilism. In the previous article, I gave specific recommendations for activists. Here the situation is more difficult because the problem is systemic, but a few things can be done.

First, be aware of the expert’s paradox. When you are looking for information about something you don’t know, AI, if you don’t have the skills for OSINT-level research, is the worst possible source. Generative answers are most dangerous precisely where you need them most. In my opinion, for the last 10 years, schools should have had lessons on what a SERP is and what the dangers of SERPs are. Is the information contained in a SERP reliable? Such training is reportedly being implemented in Estonia.

Second, go back to the sources. I often do source research to broaden my context and learn with the help of AI (Gemini Deep Research, Claude Research, Perplexity), but I verify every result step-by-step. Then I read the sources, listen to podcasts, and delve deeper into the topic. AI is an assistant for collecting materials, not an oracle. If you don’t want to make mistakes like Karolina Opolska, you must always, and I mean always, when using materials provided by AI, go to the original sources and check the citations (in Opolska’s book on conspiracy theories, the citations were fabricated). If you don’t apply this methodology, you’ll end up in a mode of zombie prompting with an illusion of competence.

Third, learn to recognize the language of confabulation. AI has a characteristic way of presenting invented facts. It often uses constructions like “it is,” “it refers to,” “it includes” without any caveats. Real knowledge usually contains more nuances and uncertainties. Of course, I have taken courses in manipulation, eristics, logic, and have a degree in philosophy, but many of these errors can be caught with a little mindfulness.

Fourth, test the system. My test with “EFF everbridge” was intentional; while expecting research results, I wanted to see how the AI mode would behave versus the standard SERP. Do such tests on topics you know well. This builds an intuition for where the system can be trusted and where it cannot.

Fifth, remember the confirmation bias from the first part. AI slop is particularly dangerous when it confirms what you already believe. The system is trained to give answers that sound convincing and will be liked, not to tell the truth. The goal of modern algorithms is not TRUTH, i.e., quickly arriving at an answer, but ATTENTION and your time spent on a given service. Facebook is probably the best example of this.

Do We Still Have Time for Mindfulness?

In the previous article, I wrote about activists who created conspiracy theories based on Google search results. That error stemmed from the human tendency to scan and simplify. Today’s problem is deeper: the system itself generates material for conspiracy theories, and it does so with the authority of an algorithm.

The Stanford researchers end their report with cautious optimism: “if the research community can rally around the shared goal of making generative search engines more verifiable, I think things will get better.” I would like to share this optimism. But observing the direction of the fight between OpenAI, Google, Anthropic, and the introduction of commercially underdeveloped tools… that is, concentration, closed models, the race for engagement instead of truth—I have quite serious doubts. Mindfulness in an age of rapid content consumption is also not an easy skill.

Let’s remember that the philosopher kings who dreamed of setting the semantics, and thus the concept of the noble lie, have to eat; companies must have income. And we must have the knowledge to distinguish myth from fact. The thing is, no one will provide us with this knowledge through the magic of prompting… you still have to acquire it yourself before you ask the machine. Or maybe we should ask directly whether we still want mindfulness and criticality, or perhaps a world of appearances is more attractive to us?

To be fair: I collected materials for the upcoming article on Everbridge and their systems with the help of research-AI, which I then verified. I did the SERP test with “EFF everbridge” out of curiosity, to see how the AI mode would behave.

Noteworthy Sources

Epistemic Injustice in Generative AI

STANFORD UNIVERSITY: Liu, N.F., Zhang, T., & Liang, P. (2023) Evaluating Verifiability in Generative Search Engines | GitHub

It does not matter to me that the text is 2 years old; as you can see, the theses are still relevant, and the blame for this lies with companies implementing underdeveloped technologies for consumers and businesses.