Why OpenAI Assistants Are Unsuitable for Complex AI Agents

About the article:

This is a case study of how we at Soft Skills Lab developed AI agents for an EdTech product:

• Why OpenAI Assistants were not suitable for complex scenarios

• What problems arose with our backend

• Why we needed to create our own platform for managing agents

If you are integrating AI into a product and are wondering which path to take — this experience may be useful.

Launching an AI agent on OpenAI Assistants is easy. We at Soft Skills Lab did just that: we created 20 AI simulators for negotiations. Everything worked smoothly, OpenAI's infrastructure is great.

But when we started complicating the scenarios — we hit a wall.

Do we need to explicitly manage the agent’s memory? No. Conditional logic in the middle of a dialogue? Workarounds. Switch the model to another one? Rewrite the integration. Understand why the agent gave a strange answer? There are practically no logs.

We realized: for complex product scenarios, we need not just stability, but operational control. Here’s how we came to this conclusion (and what we did next).

First attempt - OpenAI Assistants

We started with OpenAI Assistants — a quick start, stable infrastructure. It works great for simple scenarios.

But then we wanted to develop an exercise called "Skills Testing": the agent asks questions, analyzes answers, evaluates competencies. And that’s where the trouble began...

Flexibility issues:

1. Conditional logic based on the response

In skills testing, it is very important to apply different checking logic depending on what the person answers.
For example: if the person answered A — ask a clarifying question, if B — move to the next block, if C — give a hint and ask again.
In Assistants, this is either through functions or is generally impossible to implement explicitly.

2. Control over memory

The agent needed to remember the pattern of responses: if the person made a mistake in one category of skills three times in a row — this is a signal for a deeper check.
In Assistants, memory works automatically through threads — convenient, but you do not control WHAT exactly the agent remembered and HOW it uses it. Sometimes it pulled the wrong data from context, and the logic broke down.

3. Works like a black box

When the agent gives an unexpected assessment or misses an important pattern — you want to understand why.
But in Assistants, you only see the final result. There are no detailed logs — the system works like a black box.
For debugging complex testing logic, this is a problem.

Conclusion on OpenAI Assistants — a cool tool for a quick start. But when explicit conditional logic, memory control, and detailed analytics of each step are needed — you run into limitations.

Attempt 2 - our exercise constructor

We decided to create our own backend: FastAPI, our prompts, full control.

Plus: we can implement any logic, with no restrictions. The infrastructure is stable.

And then problems arose...

1. Debugging turns into a quest

The agent gave a strange evaluation in testing. I want to understand why — I go to the developers.

They look at the logs on the server, figure out what happened at which step, searching for where exactly the agent went wrong.

Analyzing one strange case took from one hour to several hours of team work.

2. Switching models = pain

We wanted to test Claude instead of GPT-4 — this means changing the integration, testing, deploying. Each model switch carries the risk of breaking something.

3. Provider lock-in

Even while creating our own backend, we were essentially tied to the OpenAI API.
When we wanted to try GigaChat or DeepSeek — we realized that it would take weeks of work for refactoring.

At some point, I understood: we were spending 40-50% of our time not on new exercises, but on supporting existing ones, debugging, and making small fixes.

We didn't have a unified system: logs in one place, prompt settings in the code, dialogue analytics — nowhere at all.

Attempt 3 - Agent Builder from OpenAI

We saw Agent Builder from OpenAI — wow, this solves our problems! A visual constructor, logic can be set up without coding, iterations are fast.

We almost started redoing exercises on Agent Builder...

But...

Then I asked myself: what if tomorrow we realize that for a specific exercise, Gemini performs better than GPT-4?

Or that DeepSeek provides the same quality but is 5 times cheaper for certain tasks?

Agent Builder solved the flexibility problem but created a new one: we would be forever stuck on OpenAI models.

All our exercises would be trapped with one provider. Migration = rewriting all 20+ simulators from scratch.

Realization:

We understood: different tasks require different models. In some places, GPT-4 is the best, in others, Gemini performs better, and sometimes you can save on DeepSeek without losing quality.

We need the flexibility of Agent Builder, but without being tied to one provider and its models.

Assemblix — an operating system for AI agents

We compiled a list of what is REALLY needed to support AI agents in production:

1. Visual builder for flexible logic (like Agent Builder)

2. Transparency of each call — detailed logs

3. Independence from the provider — choosing the best model for each task

4. A unified system to manage all agents

5. Memory between agents and sessions

Initially, we created it for ourselves in SSL. Then we realized: this problem is not only ours.

Attempt 4 (hope it's the last):

1. Visual builder independent of the provider

We assemble exercises visually: nodes, conditions, prompts — like in Agent Builder, but on our infrastructure.

The same simulator "Skills Testing": if answer A — a clarifying question (node), if B — the next block (transition), if C — a hint (condition). All this visually, without code.

We change the logic right in the interface, without releases and deployments.

2. Seeing each step of the agent

We see each step of each agent: what it answered, how much time it took, what logic was triggered.

The agent gave a strange evaluation? We open the log of this session, see the entire path: what conditions were triggered, what the model answered at each step.

Debugging a strange case — a few minutes instead of several hours of team work.

3. Model selection for each task

OpenAI, Claude, GigaChat, DeepSeek, Gemini — we choose a model for a specific exercise.

In some cases, GPT-4 shows the best quality, in others, Gemini handles specific logic better, and sometimes DeepSeek provides the same quality at a lower cost.

Switching between providers is literally a choice from a list in the interface. Everything is out of the box, without technical difficulties.

4. Shared memory between agents

We have a chain of exercises: first skills testing, then practical tasks based on the test results.

Previously, data was transmitted manually between exercises. Now we simply pass client_id — both agents automatically see the entire interaction history.

The memory works out of the box.

5. Unified operational panel

All agents, all exercises, all user sessions — in one place.

We can filter: show all sessions of a specific student, or all exercises with a certain model, or all dialogues from the last week.

This is operational control over the entire AI infrastructure.

What have we achieved?

1. Debugging is no longer a pain for developers

In the last month, the development team has not once been approached with a request to figure out why an exercise is not working correctly.

Previously, this was a regular pain: developers were distracted from tasks, digging through logs, looking for issues.

Now, methodologists and trainers open the logs in Assemblix themselves, see every step of the agent, and find the problem without developers.

2. Speed of creating exercises

Previously: new exercise = several weeks of development.

Now: several days of active work in the builder.

The main work is done by methodologists; developers are only needed for final integration.

3. The best model for each task

We stopped using one model for everything.

Part of the logic has been delegated to OpenAI (generating responses to users), and part to Gemini (analyzing user responses) — each model where it shows the best results.

Previously, such switching would have taken weeks of refactoring. Now — selecting a model from a dropdown.

4. Scale

On Assemblix, we created about 7 large exercises, which we are actively integrating into our courses.

(In the next article, I will share a specific case with details — how one of the exercises works under the hood.)

What's next?

We launched Assemblix as a product for teams that create AI agents in production.

In the next article, I will share a specific case: how one of our exercises works under the hood, what problems we encountered, and how we solved them.