TAPe-diary, day 7: first departure from transformers and “almost free” segmentation

If this is your first time here, you can first check out:

• the basic article about TAPe+ML — TAPe + ML: a universal computer vision architecture

• FAQ on TAPe-detection — FAQ on TAPe object detection (how we learn to detect objects instantly and dozens of times more efficiently/cheaply than ML)

• how TAPe performs against SOTA — How our “homegrown” AI outperformed DINOv2, ViT, and dozens of ML models in video annotation using TAPe

• previous parts of the dear TAPe-detection diary on COCO:

  • Day 1. TAPe and YOLO

  • Day 2. 115k parameters for us vs 2M+ for YOLO

  • Day 3. Why we don’t need gradient descent

  • Day 4. COCO dataset. The beginning.

  • Day 5. 98% on 2% of COCO, less “background” and the first boxes.

  • Day 6. Synthetic data, embeddings, and the first step away from transformers

A brief reminder: what TAPe is and why we need it

TAPe (Theory of Active Perception) is a mathematical theory and technology that describes (and models) the mechanism of active perception: it breaks an image into stable features and defines the structure of connections between them. In TAPe+ML, we use these elements and then work with them rather than raw pixels.

Within TAPe+ML, TAPe algorithms transform an image into TAPe elements — structured “building blocks” with known relationships between them, providing a compact vector representation instead of raw pixels. On these representations, we build experiments: from self-supervised tasks in the spirit of DINO/iBOT to classical ML methods (clustering, search, classification) and object detection on COCO, which we are tracking in this diary.

Why we were able to move away from transformers

In the experiments, as planned, we moved away from transformers. Recall: the reason this is possible is that TAPe data already has a structure such that Global Attention is not really needed – the global context is covered because TAPe data is stable and structured.

This led to a significant reduction in the number of parameters. Currently, the number of parameters required for embeddings is orders of magnitude smaller than any analogs.

Moving away from transformers allowed progress to the next level – conditional grouping of any number of patches for object extraction and further classification. Even in the first experiments with this, there is success (at least at first glance), and a few interesting observations.

Local associations of TAPe patches

First, some localization plays a role. That is – if you choose a “move” from a certain patch, only a certain surrounding area will be associated with it, because contextually, as the area grows, it looks different.

Each patch can have associations in its local neighborhood with a certain number of other patches, which from training:

• are similar to it in one way or another (for example, all have skin); this follows from the initial embedding training;

• belong to the same object; this specifically follows from training associations to an object (i.e., we indicate that all patches in this area belong to an object of type “person”).

At the same time, attempting to further modify such a patch for task-solving does not cause problems, it just needs to be slightly more granular.

Example with a face: how associations look

As a good example – an image:

This is one of the images in the COCO database. The central patch is marked in yellow. In orange – all patches that are associated as related to this patch. As can be seen, all orange patches are semantically associated with the same thing as the yellow patch, namely the human face.

At the same time, fully outlining a person’s face is a simple clustering task. We don’t train to distinguish a person’s neck from their face, so the outline is not guaranteed to stay on the face and may capture other patches where there is skin in general, for example, but in our experiments, faces often fully fell within their area.

As a result, we can in some way classify the concept of “skin.” Yes, we don’t do this directly, but it emerges from training because clothing forms a natural boundary within the object “person,” and we find these segments because we rely on the most contrasting patches to assemble a general description of the entire object.

Center shift and the “seed” of segmentation

If you move the yellow patch to the side, the association remains the same, with minor changes, because the relationship between patches is asymmetrical (this follows from TAPe’s principles).

If you start moving the central patch beyond the first zone, the new and old zones begin to overlap—while still preserving the same contour of the “object” itself (the person in this case, after all). In fact, we “get” a segmentation model for free.

Limitations, future, and speed

Let’s add a couple of things here:

• The growth of these patches to cover the entire object will also happen—it’s just pointless for such a mini-demo right now;

• Some inaccuracies (at the level of one patch going in the wrong direction) still exist—the model shown here was trained literally for one minute. This will be fixed with proper (i.e., longer, complete) training, as well as overall model improvement;

• Segmentation is a future research direction; for now, all work is heading toward detection, which these results lead toward. The end result is very close.

• The model currently runs at 120 frames per second.