The impact of software and accelerator architecture on performance

Comparison of chip characteristics

The NM Card module is based on the K1879VM8YA processor, also known as NM6408, manufactured at the TSMC factory using 28 nm process technology in 2017. The die area is 83 mm², contains 1.05 billion transistors, and the power consumption does not exceed 35 W [1]. The chip includes a PCIe 2.0 x4 controller and a DDR3 memory interface. The peak (theoretical) performance is 512 GFLOPS in fp32 data format. The NMC4 processor cores operate at a clock frequency of 1 GHz.

The closest analogs of this chip appear to be two graphics processors from NVIDIA: GK208B of the updated Kepler generation for discrete graphics cards [2] and GM108S of the Maxwell generation for the laptop segment [3]. Both chips are almost identical, released in March 2014. Since the GM108S is only available as part of laptops, and their price on the secondary market is too high for simple curiosity, the first option was chosen. However, it is worth noting that Maxwell is a newer architecture, supported by a more up-to-date software stack and potentially capable of achieving higher performance.

To avoid duplicating the text description, the comparison of chip characteristics is summarized in the table:

It can be noted that the chips are almost identical in characteristics. However, the GK208B has a certain flexibility, so it is important to choose the right graphics card for a correct comparison.

Comparison of modules

As a module based on NM6408, as already mentioned, the NM Card was chosen [4]. In terms of performance, all modules with a single NM6408 show identical performance, which is provided in the documentation for the inference framework from Module called NMDL [5], which I will refer to later.

After a couple of days of searching through local flea markets, I managed to find an MSI GT 730 (N730K-2GD3H/LPV1) graphics card for 14 euros [6], in Russia at the time of writing this article it can be found on Avito for 500-1$0.00. The graphics card was manufactured in 2016 and is quite close to the NM Card module. Both cards have passive cooling, however, the GT730 has only 2 GB of memory compared to 5 GB of the NM Card. This affected the research by limiting the batch size, which potentially limited the peak performance for several networks. The second difference is in the standard frequency: 902 MHz versus 1000 MHz for the NM6408, but it can be increased in Afterburner with one action. After that, the peak performance will be 768 GFLOPS, which corresponds to a 1.5 times difference relative to the NM6408.

Test bench and software

As a test bench, I got a computer with very modest specifications: i3-7100 and 8 GB of DDR4 memory. However, since the performance of the graphics card is being measured, this should not have a significant impact. Due to the rather old architecture, the latest version of CUDA that works with this card is 11.6. The modern onnxruntime comes with a dependency on CUDA 12.x, for CUDA 11.x you need to build onnxruntime yourself.

As a result, to get a working version of onnxruntime with cuDNN and TensorRT, the following versions of the components were found and selected:

A simple program using onnxruntime was written as a benchmark, which loaded the model, ran inference with different batch sizes, and measured execution time over 10 seconds, after which the average execution time was taken. For the sake of experiment purity, an attempt was made to use reference models as much as possible, which are distributed as additional data to the NMDL framework.

Data Processing Modes

NM6408 can operate in two modes: multi-unit and single-unit. To understand these two modes, it is necessary to briefly describe the chip architecture. NM6408 contains four independent computing clusters, each of which contains four NMC4 cores and one arm control core. Each cluster has its own DDR3 controller, to which 1 GB of memory is connected. In addition to these four clusters, there is an additional (control) arm core with its own DDR3 controller and an additional gigabyte of memory. All clusters have direct access to each other's memory, but this access is limited to the first 512 MB of memory, which is caused by the use of a 32-bit address space.

The single-unit mode represents the independent operation of each cluster. In the case of the same model, this can be represented as working with a batch size of 4 and in data parallelism mode.

In multi-unit mode, the common input tensor is processed simultaneously on all four clusters. This can also be represented as working with a batch size of 1 and in spatial parallelism mode.

The main difference between the NM6408 and traditional GPGPU operation is that in the former case, the entire model execution graph is on the device and runs with minimal host device involvement. To some extent, this is similar to how modern NPUs work. At the same time, GPGPUs work with so-called kernels, i.e., individual layers. This approach allows for better distribution of work across multiple parallel computing devices, but more on that another time.

In this study, the use of traditional GPGPU allows for greater flexibility in choosing batch size, thereby increasing device utilization.

Comparison results

An important difference in the measured quantities is that NMDL returns the execution time on the time scale of the arm control core without taking into account the transfer of input and output tensor data. Onnxruntime does not provide such a mechanism, so in this experiment, timestamps are taken on the CPU scale before and after running inference through onnxruntime. Thus, all other things being equal, the performance values obtained from onnxruntime will be lower.

For batch size 1, the following results are observed:

Two out of 12 models were slower on the GT730, the performance of two models was on par, the rest received a significant performance boost. The average improvement relative to the NM Card was 56.8%

The next experiment consisted of finding the "optimal" batch size in powers of two, i.e., batch size of 1, 2, 4, ... 64.

In this experiment, the fact that the GT730 has 3 GB less memory starts to show. At one time, versions of this video card with 4 GB of GDDR5 were released, but for the purity of the experiment, I specifically bought a card with DDR3. From these results, it can be seen that alexnet shows a colossal increase, if this model is excluded, the improvement is about 24%.

Conclusion

According to the measurements taken, it can be judged that the GT730 generally shows better performance with identical technological standards for the production of the system-on-chip and similar design solutions for the manufacture of the module. A certain interest could be represented by a comparison also with the GT940M based on the aforementioned GM108S, but the cost of laptops in working condition with this chip ranges from 100-150 euros, which somewhat cools my curiosity.