8 external factors affecting the effectiveness of facial recognition from video cameras

First of all, if you decide to build a facial recognition system, high-resolution cameras will not solve all your problems. Unfortunately, in most cases, the result will only get worse, and the project costs will increase.

Hello, I am Mikhail, a product manager at TRIDIVI, and I lead facial recognition projects on video for "Safe City" class tasks.

I want to talk about the internal and external factors that significantly affect the effectiveness of facial recognition, as well as share practical recommendations on how to reduce their negative impact through the correct architecture of information systems, selection of appropriate equipment, and its installation locations.

Research Approach

In 2023, in preparation for two "Safe City" class projects, we decided to conduct a series of specialized tests in various street operating conditions to identify direct and indirect factors affecting the performance of the designed systems.

The tests lasted from November 2023 to July 2024, covering 4 seasons of the year and 3 cities (Moscow, St. Petersburg, and Chelyabinsk) in different latitudes and time zones.

In total, out of ~5,500 passerby faces, 1,056 identification attempts were made on a test database of 528,000 faces.

A methodology and testing program were developed, various types of cameras were selected, and several groups of participants were involved to check the correctness of identification under the influence of conditions such as:

• Resolution and viewing angle of cameras;

• Camera installation height: from 2 to 4 m — to assess the impact of exceeding the requirements for head tilt angles;

• Camera installation locations: pedestrian traffic light poles, trolleybus and tram poles — to assess the impact of vibrations and shaking;

• Adverse weather conditions: snow, rain, fog, smog;

• Adverse lighting conditions (less than 50 lux): twilight, artificial city lighting;

• Backlighting from traffic lights and advertising structures;

• Camera orientation by cardinal directions (in northern areas in the morning and daytime, the sun can give backlighting on cameras oriented to the south, and in southern areas — to the east).

During the study, the impact of the system architecture was also analyzed, taking into account the bandwidth of the urban network:

• Data Center (DC) — streaming from cameras to the data processing center;

• Edge — processing streams from cameras directly at the installation sites with subsequent sending to the DC of identification results and "thinned" video streams.

After analyzing the results, I identified 2 groups of factors that have the greatest impact on the effectiveness of identification in face recognition systems.

• 8 external — impact at the level of the environment and operational problems.

• 6 internal — impact at the system level.

In this article, I will discuss how you can manage external factors.

1. Vibrations (wind, shaking from transport)

Degree of impact — low (3 losses in 1056 attempts).

The greatest vibrations and shaking were experienced on tram poles at the moment of transport passage, but they had little effect on the overall result. The system manages to process 25 frames per second, so there is no significant blurring of the image from shaking. Our general recommendation is that the image shift should be no more than 1% of the frame size.

2. Weather conditions (snow, rain, fog)

Degree of influence — low (7 losses in 1056 attempts).

In daylight or with sufficient lighting (>200 lux), snow and rain create interference in images comparable to the natural noise of camera sensors.

The situation worsens in low light conditions when precipitation and airborne particles begin to amplify the effect of backlighting.

3. Distance from the camera to the object

Degree of influence — significant (9 losses in 1056 attempts).

Sometimes it is necessary to use telephoto cameras to ensure face identification at a distance. This may be necessary to "reach" the appropriate identification zone (for example, where people are at rest and waiting for a traffic light signal), or to level the excessive tilt angle of the camera due to the need to raise it above 2.5 meters.

As the distance to the identification point increases, the influence of the two previous factors (precipitation, airborne particles, shaking, and vibrations) begins to increase significantly.

4. Density of human flow

Degree of influence — high (26 losses in 1056 attempts).

In a dense crowd, there are more overlaps and obstructions, and therefore a higher chance of missing successful frames with a face image when:

• The person is looking in the direction of the camera;

• The face is in focus, and the image is not blurred due to active movement;

• There is no excessive noise, artifacts, and interference on the face: snowflakes, rain, hair, cigarette smoke, etc.

5. Speed of movement of people in the frame

Degree of influence — high (28 losses in 1056 attempts).

Choose areas where the speed of people is ≤ 5 km/h. This means that people moving by running, on a scooter/bicycle, etc., may be missed. If the face has shifted a distance greater than its own size between two adjacent frames, this can result in excessive image blurring and track loss.

6. Backlighting (from the sun, lamps, advertisements)

Impact level — critical (37 losses in 1056 attempts).

Backlighting in the morning hours, glare on wet asphalt, or ice shine may not last long and seem like an insignificant problem against the background of the overall duration of the camera's operation during the day or year, but we lost quite a lot of identifications during the tests precisely because of this reason.

Fog at night can also worsen the impact of backlighting from traffic lights, advertising structures, and passing cars, making face images unsuitable for identification.

7. Shooting angle (head turn and tilt angles)

Impact level — critical.

People look at their feet and phones. Placing the camera too high sharply reduces the chance of getting a frontal face image, optimal for identification.

Excessive camera shift to the side of the main traffic flow increases the risk of blurring.

8. Face illumination less than 200 lux

Degree of influence — critical (98 losses in 1056 attempts).

Night tests without the use of additional lighting in identification areas (in addition to the existing standard city lighting, providing <50 lux illumination) showed a threefold drop in performance.

How to manage external factors?

1. Recommended manufacturers of specialized camera lines (with the possibility of using telephoto lenses): AXIS, IDIS, Hikvision

2. When installing, check for backlighting depending on the direction of light in the morning and evening hours, as well as from advertising structures, traffic lights, and lamps.

3. Set up cameras for night operation. Without special lighting for most of the day (from 18:00 to 09:00 in winter, from 22:00 to 07:00 in summer), up to 60% of identifications may be lost.

4. Do not use overview cameras for identification. Avoid the "identification spot" falling on the passage area. It is better to replace the lens with a telephoto lens and ensure face capture in an area where people are stationary and without exceeding permissible angles.

5. The quality of the majority of face images obtained from cameras should strive for quality comparable to NIST Mugshot or Wild (35 pixels between the eyes, even lighting, no more than 20° angle of rotation/tilt from the frontal position). This can be checked using automated services provided by vendors of face recognition systems from the NIST FATE Quality list.

   Practice shows that if the database is formed from faces with NIST Visa/Border quality, and face images with NIST Mugshot/Wild quality come from video cameras, the system will work efficiently and identifications will be reliable. Otherwise, the likelihood of false identifications or misses increases.

   

Conclusion

My research has shown that at the stage of setting up and placing cameras, specialists often work "blindly"! Although there are available methodological materials and even tools that automate the assessment of the effectiveness of camera settings, they are not used as widely.