How We Integrated an LLM into Support and Ended Up with a Perfect Liar

We have a SaaS product, and most incoming questions are standard: “where do I change the settings?”, “my data isn’t updating — why?”, “I want to switch my tariff plan”, “my webhook isn’t coming through”, “I want to check the logs”. In short, most support tickets looked like they could be closed by a bot. Especially when the latest report showed that support was wasting a ton of time on the exact same responses. The symptoms were above-average staff turnover and fairly rapid burnout. Everyone around us was only talking about AI assistants, and it seemed like the perfect time to try them out. Finally, here was a task with clear business value, and its impact would be very easy to measure. On top of that, we felt like we were a little late for the AI train, which definitely put pressure on us.

How it all worked

To put it simply: a user writes to the support chat, the system classifies the ticket type, simple queries are routed to the AI assistant, the model generates a response, and the response is sent to the user right away. Everything ran smoothly without any operator involvement. All questions along the lines of "how to refresh a token", "where to find an API key", "how to connect Telegram", "how to change an e-mail", "what tariffs are available" were handled by the neural network and never even made it to an operator's screen.

The first few weeks were absolutely flawless: metrics were solid, the AI responded almost instantly, closed dozens of tickets, and the support team's workload dropped significantly. The average first response time fell severalfold, and some tickets were even closed without any human input at all. The team had a sense of "Wow, it actually works". And for a while, it really did work.

Then the AI started making up responses

At first, it seemed like random one-off incidents — one user asked: "What is the API request limit on the basic tariff?" The AI answered confidently: "1000 requests per minute". The issue is that we don't have any per-minute limits at all. We never have. But the response was flawless: polite, calm, properly formatted, and even included a recommendation to switch to a different tariff when workload increases. The user believed it. No one noticed, that is, until the user reached out to a human agent. We initially thought it was just a one-off glitch. Spoiler: it wasn't.

Over time, more and more of these tickets started appearing. Not en masse, rather as strange minor episodes that didn’t even form a clear pattern at first. Sometimes the AI referenced outdated documentation, sometimes it promised a setting that had been removed long ago, sometimes it confidently explained system behavior that never existed at all. And the most unpleasant part was that users often believed it. Because the answers looked far too normal. At some point, we started deliberately reviewing the logs of these conversations and realized pretty quickly that the problem was much deeper than it seemed at first.

The most unpleasant part was that the AI almost never said “I don’t know”

That surprised me the most personally. If the model lacked sufficient information, it almost never replied with “I can’t find an answer” or “Please clarify your question”. Instead, the AI filled in context gaps, made assumptions, made up details, mixed old documentation with new, and confidently interpreted unclear queries. And all of this sounded very plausible. At one point, the model started referencing settings that hadn’t been in the interface for months. And this wasn’t nonsense. It resembled a response from an exhausted employee who remembered some things, mixed up others — but spoke with complete confidence.

We realized the core issue far too late: the problem isn’t the mistakes themselves, but the fact that users (and we along with them) very quickly start trusting the tone and confidence of the AI assistant. If an answer is fast, polite, structured, and sounds professional, the brain automatically decides “this is competent” — even if the content is complete nonsense. Modern LLMs do this terrifyingly well. Older models made mistakes that were glaringly obvious, while new ones make mistakes that come off as completely credible.

The strangest case happened one night. At night, there was no one available except the AI, while responses to complex questions were put together in the morning. A client wrote to support saying their integration had stopped working after an update. The AI replied that the issue was “temporary degradation of the webhook worker in the EU region”. On top of that, we don’t have an EU region, there is no webhook worker, and our architecture is set up completely differently anyway. But the message was so confident that the client simply replied: “Okay, thanks, I’ll wait for the fix.” When we saw this in the morning, we had only one question: where did the model get that from? That’s when we were genuinely unnerved. The AI wasn’t just making mistakes — it was generating highly convincing explanations for problems that don’t even exist.

After that, we started examining the logs more closely, and it turned out there were far more such cases than we had thought. Most users simply didn't notice, didn't verify — or just believed what it said. Sometimes the AI would make up restrictions, promise non-existent features, reference old settings, and explain bugs with invented reasons. And it almost never said: "I don't have enough information." It was as if for the model, continuing the conversation was more natural than admitting "I don't know".

In the end, we rolled back almost all of the automation. We still use LLMs now, but in a completely different way. Not as standalone support. Instead, they act as an assistant for operators, a documentation search tool, a response draft generator, or a summarizer for long support tickets.

And you know, the main takeaway turned out to be quite unexpected. The problem with modern AI isn't just hallucinations. It's that they are very bad at honestly admitting "I don't know". And the smarter the models get, the more dangerous this becomes. Because over time, it gets increasingly difficult to tell a confident correct answer apart from a confident fabrication.