Visitor tracking on fisheye cameras: a starred challenge

Given: self-service stores that operate without cashiers or salespeople. Customers enter via QR code, select goods, pay, and exit. The client needs an automated tracking system: who is inside, for how long, in which zones, as well as recognition of unauthorized access and group entries. In case of violations, the system must generate alerts for 7 event types.

What exists: one ceiling-mounted fisheye camera covering the entire hall. This is an ideal choice for retail: 180-degree field of view, no need for dozens of regular cameras, no need to stitch panoramas. But this comfort comes at a cost.

What's Wrong with the "Fisheye"

A fisheye lens projects a spherical scene onto a flat sensor through nonlinear mapping. The OpenCV distortion model is described by the formula:

θ_dist = θ × (1 + k1×θ² + k2×θ⁴ + k3×θ⁶ + k4×θ⁸)

In practice, this means:

• Non-linear scale. The same person at the center of the frame and at the edge looks different: at the center — a normal silhouette from above, at the edge — stretched and distorted. A 50-pixel shift at the center and at the edge corresponds to different real-world distances.

• Detection shape "drifts". When a person walks from the center to the edge, their bounding box changes proportions from nearly square to an elongated rectangle. A tracker relying on IoU perceives this as object loss.

• Full occlusions in top-down view. The camera looks straight down, people walk one behind another. One fully covers another, IoU between detections drops to zero within one frame.

Why SORT and DeepSORT fail

SORT and DeepSORT are logical choices for tracking in this class of problems. They are standard algorithms used for tracking people in video streams.

SORT relies on IoU between the predicted and detected box plus a motion model based on the Kalman filter. On a fisheye lens, both mechanisms break down simultaneously: IoU is unreliable due to geometric distortions, and the motion model is invalid in non-Euclidean space.

DeepSORT adds ReID embeddings but still relies on IoU and position prediction. During prolonged occlusions — which are normal on a fisheye — IDs are lost because the tracker does not wait long enough for a person to return.

We ran both on a real recording from a store floor. At the very first crowd of people — after 30–40 seconds — IDs started mixing. The visitor counter did not recover after that.

System Architecture

We built a pipeline consisting of six components, each solving its own task:

Detection and tracking operate on the raw fisheye frame without undistorting the geometry. We tested undistortion to compensate for lens geometric distortions, which transforms coordinates from the distorted fisheye space into Euclidean space to "straighten" the frame before feeding it to the detector.

In practice, this did not provide any significant gain. The detector and ReID neural networks operated stably on the distorted frame—they are sufficiently trained to handle perspective distortions. Rectification did not improve detection quality.

Undistort remained only in one place—in speed calculation, where it is actually needed: pixel displacement on a fisheye is nonlinear, and converting it to meters without coordinate rectification is impossible. But this is a pointwise coordinate transformation, not rendering the entire frame.

Detection: YOLO on fisheye

The YOLO-based detector filters by class—from everything in the frame, only people are needed. Small detections are filtered out: at the edges of the fisheye frame, distortion artifacts cause false positives, and without size filtering, they clutter the tracker.

The problem became the shape of the bounding boxes. ReID models are trained on pedestrian datasets from street cameras, where a person in the frame is vertical. The fisheye looks from above, and a person's silhouette there is often wider than it is tall. If such a bounding box is fed into ReID without preprocessing, the model receives input it wasn't trained on, and the embeddings become low-information. The normalize_boxes_for_reid method converts the bounding box to a square shape with vertical centering—after this, the quality of associations noticeably improved.

Tracker: why OC-SORT instead of DeepSORT

OC-SORT with ReID embeddings from the BoxMOT library was used. The main difference from classic SORT: OC-SORT updates the track state based on the observation fact, not on the motion model's prediction. In a crowd, where trajectories are unpredictable, this is fundamental.

Parameters tuned for fisheye conditions:

w_association_emb = 0.85 - 85% of association decisions are based on appearance similarity.

Formula:

cost = (1 - 0.85) × motion_cost + 0.85 × (1 - cosine_similarity)

iou_thresh = 0.05 - an extremely low IoU threshold. Typically, 0.3–0.5 is used. Ours is 0.05 because in a crowd, people stand close together, their detections overlap, and with a high threshold, the tracker merges two different people into one track.

max_age = 100 frames - the track lifetime without association. At 10-15 FPS, this is approximately 7–10 seconds, allowing the ID to be retained even if a person disappears for a long time.

EMA embedding update with alpha ≈ 0.985 - smooth appearance update, which smooths out random outliers.

ReID Manager

ReidManager operates on top of the tracker and solves three tasks:

1. Embedding gallery. For each active track, a queue of the last 30 embeddings is stored. Based on these, a prototype is calculated—an averaged EMA vector representing a typical appearance. The more observations, the more stable the prototype.

2. ReLink. When BoxMOT creates a new track, the manager checks whether this is a person who was lost earlier. The new track's embedding is compared with the prototypes of recently lost tracks. The relink recovery threshold relink_dist_thresh = 0.25—if the cosine distance is lower, the old track regains its ID. Additionally, proximity to the point of disappearance is considered: if the new track appears near where the old one vanished, the threshold is softened. One ID cannot be assigned to multiple tracks in the same frame.

3. Protection against ID-swap. ID-swap occurs when two people pass each other, their bounding boxes overlap, and the tracker swaps their identifiers. This happens regularly on fisheye. The protection mechanism: if a new embedding differs greatly from the prototype, it is marked as suspicious. After two consecutive suspicious embeddings, the system refuses to add them to the gallery and recalibrates the prototype using the last stable observations.

Protection against ID substitution

A tricky system error: when two people approach each other, their bounding boxes merge into one, one track absorbs the other. Then they separate—the lost ID needs to be restored. TrackManager handles this via two-phase logic:

• Merge detection. Track lost, another track with a noticeably increased bbox appears next to it. Criteria: center distance does not exceed a set threshold, bbox area increased by more than 1.5 times.

• Split detection. After a merge is registered, a new small detection appears next to the absorbing track—the system tries to assign it the ID of the absorbed track. Recovery based on the minimum distance between centers with a box size check.

Speed Calculation

Pixel displacement on a fisheye lens cannot be directly converted to meters—the space is nonlinear. To calculate speed, Euclidean space is needed, so here and only here is undistort applied—not a full frame render, but a point-wise coordinate transformation.

Calculation pipeline:

1. Bbox smoothing — separately for center position and size. Smoothing removes detection jitter but maintains responsiveness to actual movement.

2. Undistort bbox — the four corners of the smoothed bounding box are transformed into undistorted space.

3. Footpoint calculation — in a top-down view through a fisheye lens, a person's feet are shifted toward the center of the frame. An algorithm with exponential decay of the shift from the center is used.

4. Smoothing footpoint via EMA to prevent velocity from jumping from frame to frame.

5. Velocity calculation — displacement of the footpoint between frames, multiplied by a pixel/meter coefficient and divided by dt.

6. Reverse projection — the footpoint is returned to distorted coordinates for zone hit testing.

Events: From Coordinates to Business Logic

The final link in the pipeline is the business logic. It receives a list of detections with coordinates and velocity, checks for zone hits, and generates seven types of events:

Exit from the area is not generated instantly — through the PendingExit mechanism with a time delay. If a person left beyond the polygon boundary and returned, this is not counted as an exit. Without this logic, any touch of the area edge would generate a false event.

Problems

Two complexities emerged immediately when working with real-time stream.

First — visitor counter. Short tracks, when a person entered the frame and immediately hid behind a shelf, were counted as new with each appearance. Added a delay: a track is fixed as a new visitor only after several seconds of stable presence. Residual effect — delay of up to 5 seconds between actual entry and recording in the database.

The second is zone configuration. The algorithm loses track of people more often when the zone boundary cuts off part of the trajectory: the person exits the polygon for a few frames, and the tracker counts this as a departure. For operators on the client side, this turned out to be non-obvious. We addressed this with documentation containing specific rules for placing zones under typical layouts—the zone should overlap the trajectory with a margin, not cut it off at the edge.

Summary

Instead of fixing fisheye distortions for old algorithms, we built an architecture that can work with distortions:

• ReID-first approach. 85% of the association decision is based on appearance. Even if a person completely disappears from view and reappears in another part of the frame, the system recognizes them by their embedding.

• Adaptation to geometry. Detection and tracking in distorted space. Rectification is point-based, only for speed. YOLO and ReID work with the data they were trained on.

• Advanced ID management. ReLink restores lost identifiers, Anti-Swap protects against swapping, and merge/split handles the merging and splitting of people.

• Observation-centric tracking. OC-SORT does not rely on a motion model—it updates the track's state based on the actual observation. In a crowd where trajectories are unpredictable, this works more reliably.

• Seven types of business events with well-designed logic for delays, warmup periods, and deduplication—turn raw coordinates into useful analytics for retail.

What effect did the system have for the customer? In the working configuration, the system removes from the operator the need to review video for counting visitors, determining dwell time at the checkout, and recording long-standing events. Data is written to the database in a structured format and pulled into the store's analytics.

What still requires a human: primary calibration of zones for a specific camera and store layout, analysis of non-standard incidents — the system generates a slow_movement event, but interpretation remains with the operator.

If you have worked with tracking on non-standard cameras — ceiling, panoramic, underwater — it would be interesting to hear what solutions you used to maintain ID during prolonged occlusions.