How we teach robots to understand the physical world: the path from specialists to generalists

My name is Evgeny Mikhaylenko, I lead the Physical AI business and product team at Yandex Robotics. In this article, I will discuss how modern architectures and the concept of Guidance, which we combined with the Wozniak test, help bridge the gap between specialist robots and future generalists.

Why progress is moving towards generalist robots

In one of our previous publications on Habr, we talked about a robotic arm with artificial intelligence — Picker. This is a specialist robot, and it excels at one specific task: we taught it to pick up an object it sees for the first time and carefully move it to the desired location.

However, Picker cannot go beyond its task yet. Moreover, depending on the application area of the robot, it needs to be able to perform many different actions — for example:

• Assembly of non-standard parts and products — each time the components are different, and it has to adjust the force, angle, and order of actions.

• Flexible production lines — small batches, constant reconfiguration of equipment.

• Household assistants — an infinite number of possible scenarios: cleaning, cooking, home maintenance.

The conclusion is this: to solve truly universal tasks, a generalist robot is needed, which is given a common goal, and it independently selects and combines the necessary actions to achieve it.

What is needed for a robot to become a generalist

Modern robots — even the most advanced ones — are still hard to call universal generalists. The question arises: what is required from technologies to make the creation of such a robot possible?

If we imagine that a team of roboticists could algorithmically describe all possible environmental scenarios and all the robot's reactions, we would have a generalist. But in reality, this is unattainable — there are too many combinations. This is where the power of neural networks comes into play, as they can generalize. They are capable of solving tasks for which they were not directly trained. We saw this with LLM and VLM models, as well as during the development of our Picker, which worked correctly with new objects and in new settings — for example, under different lighting and backgrounds.

By using neural networks and large amounts of data, we are gradually getting closer to having robots understand the context of the physical world almost like humans do. For instance, it realizes that if it turns over a cup of water, it will spill.

The next step: to create a universal "brain" that can control different robots. For the universal brain to work with various bodies, we use the Embodiment approach. It consists of several stages:

• we apply simulators and real-world data;

• we collect new data through teleoperation;

• we add them to the training set so that the "brain" sees all the possibilities of a specific body.

We have already tested this method on the Picker: a robot trained to work with a pinching gripper successfully adapted to a vacuum gripper — confirming the effectiveness of the Embodiment approach.

What Makes a Generalist Robot

Currently, two main necessary skills can be identified for generalist robots:

Understanding the context of the world and predicting the consequences of their actions. The robot must understand cause-and-effect relationships: if it puts a glass under a stream of water, the glass will fill; if it tips the glass over — the water will spill. This ability to foresee the results of actions is the foundation of the robot's intelligence.

Agency, meaning the ability to take initiative, make decisions, and control its actions and environment. For example, if a household task "Clean the kitchen" is given to a highly autonomous robot, it will:

• Understands what it means to "clean up" — formulates the goal.

• Finds dirty dishes and puts them in the dishwasher.

• Notices spilled juice and wipes it up.

• Sees trash and takes out the bag.

• Replaces the cloth if it is dirty.

Current developments in robotics are gradually moving towards absolute intelligence — when a robot truly learns to deeply understand the surrounding world, and interacting with it can be done in a familiar way, "as with a human." Although today robots do not yet possess such skills, they can already be universal assistants — for example, if given very detailed instructions about what we want from them. Therefore, in training the robot, we focus on the Guidance approach.

Guidance Concept

This is an approach where a person explains the task to the robot in such a way that it can remember and reproduce it, and adapts the infrastructure and conditions of the task to the current abilities of the robot. In general, this resembles prompting in AI, only instead of text — it is a very detailed guide on how to perform the task, and preparation of the surrounding space, and instead of generation — execution of the task in the physical world.

Thus, we remove the main barrier to mass robotization: the necessity for long and expensive integration for each case. The more complex the task, the more detailed the instructions for the robot must be.

Let's imagine that we want our robot to learn how to make coffee. It already has basic knowledge about coffee and how to make it, but it needs to understand how this process works in the physical world. We prepare Guidance for it: describing what types of coffee machines exist, what types of cups there are, which buttons to press on different coffee machines, how to place cups on which coffee machines, what signals we need to wait for, and so on. That is, it should be a sufficiently detailed and comprehensive instruction, anticipating various nuances that may arise during the operation. But we only need to prepare such a document once, and after that, we can ask the robot to make coffee in different kitchens and with different coffee machines.

A typical example of a Guidance element in terms of preparing infrastructure for training is the yellow button on the coffee machine. The person in the instructions writes: "Americano - this is the yellow button. Cappuccino - the green one" and sticks labels on the buttons of all coffee machines in the office. It is much easier for the robot to find the yellow button than to figure out the coffee machine's menu. One could spend a lot of resources teaching the robot to understand the coffee machine's menu, or one could simply use colored stickers. As our robot becomes "smarter," the need for guidance will decrease (and the stickers will no longer be needed).

In other words, one can spend a long time creating very strong intelligence, or one can quickly and cheaply adapt the infrastructure around the robot that will allow it to work and provide instructions. Thus, Guidance allows us to deliver benefits before absolute intelligence emerges. This is not an alternative to absolute intelligence, but a product shortcut.

A similar approach to preparing infrastructure can be used for other scenarios: for instance, assembling a pallet in a warehouse or packaging goods in production. For example, if a robot struggles with a certain size of packaging box, it makes sense to modify it according to the current capabilities of the robot.

How to understand that a robot is smart enough: combining the Wozniak test and Guidance

When it comes to testing AI intelligence, the classic Turing Test is often mentioned: if a person cannot distinguish a machine from a human in a dialogue, then the machine demonstrates intelligence. But what should be considered analogous to such a test in robotics?

The community has proposed numerous options, and one of the most famous due to its simplicity is the Wozniak Test. The test checks fundamental abilities:

• orientation in an unfamiliar physical environment;

• recognition of real-world objects;

• manipulation of objects (take, turn on, pour);

• planning and executing a chain of actions under unpredictable conditions;

• flexibility in responding to unexpected events (spilled coffee, water runs out, door won't open).

If we return to the example of the coffee machine, the essence of the test is that the robot must enter an unfamiliar room, find the kitchen, discover the coffee machine (or another device for making coffee), pick up a cup, coffee, and water — and independently prepare a cup of coffee.

In the near future, we plan to apply Guidance to the Voznjak test: to provide the robot with a pre-prepared detailed instruction and see how far it can go. Ideally, the robot will be able to operate at different coffee points in Yandex offices.

Today, robots perform individual narrow tasks excellently, but the real world is too diverse to program everything in advance. For a robot to become universal, it needs a "brain" that understands context and can adapt to new situations.

Modern neural networks already show that such generalization is possible. Now the task is to transfer these abilities into the physical world and teach one brain to work with different robots. Embodiment helps with this — transferring skills to different types of "bodies".

But even the most advanced models are still insufficient, which is why we use Guidance. A person explains the task to the robot in detail once, and then it is capable of repeating it in new locations and conditions.

Thus, we are gradually approaching the moment when the robot will be able to complete the Voznjak test even before the emergence of full-fledged AGI.