Ultra Ethernet: what is this technology and why is it needed for data centers

Ethernet maintained its position due to its mass adoption and vast ecosystem. However, with the proliferation of large artificial intelligence models, the nature of workloads has changed drastically: thousands of accelerators exchange terabytes of data, collective operations require strict synchronization, and a brief overload can slow down the training of the entire cluster. In such conditions, traditional networking approaches begin to hit their limits — hence the interest in solutions like Ultra Ethernet.

The initiative was born in 2023 thanks to corporations like Intel, AMD, Broadcom, Cisco, Microsoft, and others. They formed the UEC consortium. Today, it already includes over a hundred companies. The idea was to take the proven Ethernet and rework it for the workloads that dominate today: neural network training and high-performance computing. In 2025, they presented Specification 1.0 with a complete description of the stack, including the transport protocol, congestion control mechanisms, and telemetry. The first chips and network cards with support have already appeared: Broadcom showcased its Thor Ultra with 800G ports, AMD released the Pensando Pollara adapters, and Nokia and Arista are testing switches. The first real products have started to enter the market, but the ecosystem is just forming. Let's analyze what exactly has changed and why this could become the new standard for large systems.

How the Ultra Ethernet Architecture Works

The foundation remains the same — standard Ethernet according to IEEE 802.3. Cables, optics, and the physical layer are compatible with what is already used in most data centers. This means that Ultra Ethernet does not require a complete replacement of the channel-level hardware infrastructure. However, to utilize new technologies, primarily UET transport and advanced congestion management, network adapters and switches with the corresponding support are needed.

The main changes occur not at the physical level, but above — in the transport layer of the stack. For such loads, a new protocol is introduced over Ethernet — Ultra Ethernet Transport (UET), which is designed for intensive data exchange between nodes, including direct memory access, as in RDMA. It is initially focused on collective operations like AllReduce and AllGather, characteristic of model training.

Within UET, the logic is divided into several parts: one is responsible for the types of operations and their processing, another for packet delivery (with or without guarantees), and a third for congestion control taking signals from the network into account. This approach allows for more precise traffic management and better adaptation to the loads of large clusters.

Thanks to this design, packets of a single stream can take different routes through the entire network, without a strict requirement to maintain order. This reduces tail latency. Additionally, they have integrated advanced In-Network Telemetry (INT), which collects real-time data on the state of links, queues, and flows directly within the switches. Problems are detected in advance — before they turn into serious failures.

Basic APIs like libfabric or MPI remain compatible, so developers do not have to rewrite applications entirely. However, to maximize the benefits of the new mechanisms, adaptation of the stack to the specifics of UET is possible. As a result, an approach emerges that aims to combine familiar infrastructure with significant improvements for AI and HPC.

What technical innovations have emerged

Most of the changes concern how the network behaves under load. Previously, the signal about a problem came from the receiver — through mechanisms for pausing transmission or congestion notifications. In the new scheme, the sender regulates the speed itself, based on signals from the switches and the receiving side. It quickly reduces the pace if it sees congestion, so situations where one node blocks another with data happen much less frequently.

Another important change is the coordinated distribution of traffic across multiple routes. Packets are not sent along a single fixed path but are spread across the network in such a way that congested areas do not arise. At the same time, delivery outside of strict order is allowed, and the receiver correctly reassembles the stream. As a result, the network makes better use of available communication channels, and overall throughput increases significantly, especially in clusters with thousands of nodes.

Special attention was paid to protection against situations where many nodes simultaneously send data to a single recipient, overloading it. To address this, dynamic management of available bandwidth on the receiver's side was introduced, priority queues, and other mechanisms that prevent data transmission from stopping even during intensive collective operations. Several service classes for different types of traffic have emerged, along with more precise control at the level of individual connections.

As a result, the network behaves much more stably and predictably. Built-in telemetry collects data on the status of channels and queues in real-time and allows for quick identification of where congestion begins. According to the results of the first equipment tests in 2025, the traffic management mechanisms proved to be effective: congestion occurs less frequently, and efficiency in model training tasks is significantly higher, especially where there is intensive exchange of small data packets.

How this affects the performance and reliability of clusters

In large systems with thousands of GPUs, even small improvements in the network translate into a tangible gain for the entire cluster. Ultra Ethernet provides microsecond-level latencies from node to node, with tail values having decreased due to the mechanisms described above. For collective operations like AllReduce, where all nodes need to work in unison, this is particularly important: in several scenarios, synchronization time can be reduced significantly.

Throughput increases because all available paths are utilized efficiently, and ports already support 800 Gbit/s and higher with PAM-4 modulation. Scaling approaches linearity: adding new machines nearly proportionally increases overall bandwidth without sharp drops due to bottlenecks.

Reliability has also increased. If one link fails, traffic smoothly shifts to other routes thanks to coordinated spraying, and degradation is almost imperceptible. The overload management mechanisms prevent the emergence of hot spots that could previously paralyze an entire segment of fabric even with partial failure.

In tests and independent reviews from 2025–2026, the results show metrics approaching top InfiniBand configurations in terms of latency and throughput for AI workloads. At the same time, the standard remains fully open, and equipment from different vendors is easier to integrate. For many companies, this becomes an important argument, especially when it is necessary to avoid dependence on a single supplier.

How it differs from conventional solutions

To make it clearer, let's evaluate the capabilities of the standards in a table format. Just the basics, without diving into complex technical details.

RoCE on standard Ethernet works even today but requires very careful tuning, and tail latencies still remain a sore spot even with modern improvements. InfiniBand provides excellent speed and reliability in tasks with tightly coupled nodes; however, it ties you to a single vendor and is expensive. Ultra Ethernet takes the best from both options: a huge ecosystem from the first and advanced performance and resilience mechanisms from the second.

The difference is generally visible in the approach. Here, a new stack was built from scratch under the UEC 1.0 specification, which is initially designed for future workloads — from exaflop systems to large GPU clusters.

What’s the outcome

Ultra Ethernet is likely to start appearing where the network has already become a bottleneck: in large clouds that build enormous clusters for training models, and in research centers where thousands of accelerators need to work as a cohesive whole. For such systems, not only low latencies are important but also predictable network behavior under increasing load. In scientific computing with clusters of hundreds of thousands of nodes, the situation is similar: the larger the scale, the more painful any overload or traffic skew becomes.

At the same time, the transition does not look like a revolution — the physical foundation remains the same, while new capabilities are added as the hardware is updated. For now, this is an early stage: there are initial implementations and tests, but mass adoption is still not there. How widely Ultra Ethernet will take root will become clear based on whether major players are truly willing to bet on it in their infrastructures.