Real implementation of LLM in business processes

Hello everyone! My name is Sergey Temchenko, I work as a system analyst at EORA. We implement LLM solutions in companies such as Dodo, Stoloto, and others.

Today I will talk about three cases of LLM implementation in large companies: Magnit, Skyeng, and Tochka. I personally contacted representatives of the first two to find out the details. These cases cover different areas and help solve various problems, but they all have one thing in common: the competent use of LLM for quick process optimization. At the end of the article, I will summarize my attitude towards the implementation of LLM based on these examples.

Case 1: LLM implementation for review moderation in Magnit

Task:
The Magnit Tech team checked how well LLM can automate the moderation of user reviews in the application. It was assumed that the task could be solved with a single LLM call with a well-thought-out prompt.

Current system:
Currently, an "ensemble" of simple dictionary filters, patterns, and a small ML model is used. About 67% of texts pass automatic verification, and the remaining 33% are manually moderated.

Experiments with models:

• ChatGPT 3.5 showed ~80% accuracy with basic few-shot prompting and up to 85% with the multi-agent method (several requests for one review).

• YandexGPT initially performed worse (output format issues, unpredictable responses), but after fine-tuning on its own dataset, the accuracy increased to 81%.

• Other LLMs were also tested, read more in the Magnit article.

Interesting insight about accuracy:
When compared to the "gold standard" (human markup), it turned out that two moderators disagree about 10% of the time. That is, the real maximum accuracy (if compared to a human) is 90%, not 100%.

Conclusion:
GPT 3.5 or fine-tuned YandexGPT copes with such a simple task as comment moderation.

Link to the original article.

Case 2: LLM helps Skyeng speed up the production of educational content

Task:

1. Create and personalize lessons (task cards) for specific student requests.

2. Speed up content release (Time to Market) and reduce development costs.

Solution:

• Integrated ChatGPT into our own CMS so that methodologists could quickly generate tasks.

• Used ready-made prompt templates and taught AI to adapt to Skyeng's internal markup.

• Achieved a 3-fold increase in development speed and a 4-fold reduction in exercise creation costs.

Why not automate everything entirely?

• ChatGPT sometimes makes mistakes and "invents" facts, so methodologists, proofreaders, and fact-checkers need to check and refine the content.

• In narrow specializations (e.g., "English for a concrete plant"), questions require expert human involvement.

• Teachers have an AI generator within the lesson, but they decide which tasks to show the student themselves. Thus, people remain in the process and control quality.

Conclusion:
At the current stage of LLM development, in complex scenarios (such as lesson creation or support responses), it is more reasonable to keep a human in the decision-making loop to optimize their work rather than trying to completely replace specialists.

Link to the original article.

Case 3: how Tochka Bank implemented LLM for welcome messages in business networking

User pain:
Entrepreneurs find it difficult to start a dialogue with a potential partner: the "right" first phrases are needed. Many do not write at all or send "spam" and get banned.

Solution:
In the Tochka Network networking service, a "Greeting Options" button was added, which generates 2-3 texts using ChatGPT/GPT-4/GPT-4o. At the same time, the context from the questionnaires (experience, "how can I help", "what help is needed", goals) is taken into account.

Results:

• About half of the users who viewed the selection of interlocutors used the neural network for welcome messages.

• 86% of them approve of the suggested options, which indicates a real simplification of the start of communication.

• It is important to keep the person in the loop: AI sometimes makes mistakes or gives templates, and the user can edit the text before sending.

Conclusion:
LLM in Tochka is not "magic for the sake of magic," but a targeted solution to a real problem. Thanks to this feature, users establish business connections faster.

Link to the original article.

General conclusions

To effectively implement LLM in business processes, it is important to clearly understand what you want to optimize or what "pain" of the user you are addressing. In complex scenarios, it is important to keep people in the loop to control the quality and correctness of decisions. The optimal strategy is to launch small features with a short development cycle, test hypotheses, and gradually scale solutions based on metrics and feedback.