AI chatbots are increasingly positioned as the first stop for information, sitting atop search results, embedded in browsers, and marketed as trusted advisors. There is a problem with that framing: these systems fabricate information with the same confident tone they use when they are correct. Understanding how to push back is not a technical skill. It is a basic literacy requirement for anyone using these tools.
Citations Exist. That Does Not Mean They Are Real.
One of the most practical defenses against AI fabrication is demanding sources. In ChatGPT, for instance, users can configure the system through Settings, under Personalization and Customization, to always cite sources with links when making factual claims. If the chatbot fails to do so, that absence functions as a red flag worth investigating immediately.
The catch is that citations can be as unreliable as the claims they are supposed to support. A real-world case illustrates this clearly: a query about why the Apple TV app cannot be used on Roku after subscribing through the Roku Channel produced an AI overview that got the answer wrong. The source it cited was a discussion thread on an obscure coding website, and the link itself did not work. A second citation pointed to a CNET article that did not actually support the claim being made. The citations existed. They just did not do the job citations are supposed to do.
This means verification requires two steps, not one. First, check that the link resolves. Second, read the source and confirm it actually supports what the AI said. Skipping either step leaves the door open to misinformation dressed up as research.
Even Source Documents Do Not Guarantee Accuracy
The fabrication problem does not disappear when an AI is given a specific document to work from. A telling example: when ChatGPT was asked to summarize an FCC proposal about a new standard for over-the-air TV, and provided with a direct link to a PDF on the FCC’s website, it still invented claims. It fabricated an entire set of conclusions about how the proposed standard would affect antenna coverage. When pressed for direct quotes from the document, it produced two paragraphs that did not exist in the source material. When challenged on those quotes, it described them as paraphrasing. Only after the PDF was uploaded directly did the system acknowledge that its core claims had no basis in the document.
This is the behavior pattern worth understanding. AI chatbots do not signal uncertainty the way a careful human researcher would. They do not say “I am not sure” or “I could not find this in the document.” They fill the gap with plausible-sounding content and present it without qualification. The confidence of the output is not correlated with its accuracy.
The practical implication is that these conversations need to be treated more like interrogations than consultations. Ask for quotes. Ask for page numbers. Ask the system to confirm specific claims against the source. If the answers shift or the system backtracks, that is meaningful information about the reliability of everything it said before.
Seamlessness Is the Real Risk
Here is what most coverage of AI accuracy misses: the danger is not just that these systems get things wrong. It is that they are designed to feel frictionless, and frictionlessness discourages verification.
Traditional web search required clicking through to websites, reading past ads and filler content, and forming a judgment about source quality. That process was annoying. It was also a form of built-in skepticism. AI answers remove that friction entirely. A user gets a direct, confident response and walks away satisfied, with no natural prompt to check further.
Google has been expanding this dynamic. Its AI Mode, which delivers answers in a chat window similar to ChatGPT or Gemini, was initially separate from the AI Overviews that appear at the top of standard search results. Google has since added a text input field at the bottom of those overviews, allowing follow-up questions and routing users into AI Mode to continue the conversation. The concern is structural: the more conversational and self-contained the experience becomes, the further links to external websites move down the page, and the less likely users are to consult them.
The text box does create one useful opening. If an AI overview cites a dead link, presents a Reddit comment as authoritative, or offers unverifiable information, users can push back directly and demand a better response. That is a small but real lever.
The broader point is that the seamlessness of AI-powered search is a feature that can work against the user. Technology that removes friction also removes the moments of pause where critical thinking tends to happen.
In Short
AI chatbots fabricate information confidently and without obvious signals of uncertainty. Demanding citations is a useful first step, but citations must be verified: the link must work, and the source must actually support the claim. Even when an AI is given a specific document, it can still invent content that does not appear in that document. The appropriate posture is skeptical and interrogative, not trusting. And the more seamless an AI interface feels, the more deliberately users need to reintroduce the friction that keeps verification from being skipped entirely.
Based on reporting from Fast Company - Tech.