Data Center: Topologies and Architectures

Data Centers are typically large buildings with rooms that house endless racks of servers, communication equipment, storage systems, and other infrastructure elements. And sometimes it is extremely difficult to understand how all this technical diversity interacts with each other. In this article, we will talk about the topology and architecture of data center networks.

The network architecture of a data center should provide a balance between reliability, performance, flexibility, scalability, and cost. At the same time, it should support both current and future applications and have a power reserve that may be required for the deployment of new applications.

First, let's talk about what topologies can be used in data center networks.

Centralized Topology

The centralized model is a suitable topology for small data centers (less than 1500 square meters, although sometimes even in smaller areas a significant amount of equipment can be accommodated).

As shown in the figure, there are separate local area network (LAN)/storage area network (SAN) environments, each with its own cables going to each server cabinet and corresponding zone. Each server is connected by cables to the main switches located in the central zone (Main distribution). Thanks to this topology, we can use the ports on the switches more efficiently and it is easier for us to manage our components. The centralized topology is well suited for small data centers but does not scale well, which significantly complicates the process of adding resources.

Thus, adding a large number of network cables leads to overloading of cable routes and cabinets and increases the overall cost. Moreover, sooner or later bottlenecks in the performance of network equipment will begin to appear.

Therefore, if your data center cannot be called small, it is better to consider other types of topologies, which will be discussed later.

Zoning

The zonal topology consists of distributed switching resources. As shown below, switches can be distributed between end-of-row (EoR) or middle-of-row (MoR) locations, with chassis-based switches typically used to support multiple server cabinets.

To be precise, End of Row is a switching model in which the switch is located at the end of a row of racks and handles traffic from servers in multiple racks arranged in a row. Middle of Row is a network architecture where network equipment, such as switches, is located in the middle of each row of server racks. This solution is recommended by ANS/TIA-942 data center standards and is characterized by high scalability, easy reproducibility, and predictability. Zonal architecture is generally the most cost-effective solution, providing the most efficient level of switch and port utilization while minimizing cabling costs.

Thus, the local area network (LAN) ports of two servers (which exchange large amounts of information) can be placed on a single switch at the end of the row according to the EoR model, providing low-latency switching between ports. A potential drawback of EoR switching is the need to run cable back to the switch at the end of the row. Assuming each server is connected to backup switches, the amount of cabling can be quite significant.

Top of rack

An alternative can be Top-of-rack (ToR) switching, which typically consists of two or more switches located at the top of the rack in each server cabinet, as shown below. This topology can be a good choice for dense server environments consisting of a single rack (1RU).

All servers in the rack are connected by cables to both switches to ensure redundancy. The switches located at the top of the rack are connected to the next level of switching. The top of the rack significantly simplifies cable management and minimizes cable isolation requirements. This approach also provides quick switching of servers in the rack from port to port and predictable bandwidth consumption on the uplink.

The disadvantage of this approach is the increased cost of switches and high costs due to underutilization of ports. In large-scale deployments, it can be difficult to manage top-of-rack switching.

As a result, some data centers deploy top-level switches in MoR or EoR architecture to better utilize switch ports and reduce the overall number of switches used.

Having considered the main network topologies, let's move on to the architecture of data center networks.

Architecture

Mesh architecture, often referred to as "network fabric," consists of mesh connections between end devices and switches. The collection of network connections provides the ability to connect any device to any device with predictable bandwidth and lower latency, making this architecture most suitable for cloud services. Thanks to the numerous switching resources distributed throughout the data center, the mesh network is inherently redundant, allowing for increased application availability.

Deploying and scaling such distributed networks can be much more cost-effective compared to very large traditional centralized switching platforms.

Multilevel architecture

Multilevel architecture is the most common model used in enterprise data centers. This model mainly consists of web servers, application servers, and database servers running on various platforms, including blade servers, 1RU servers, and mainframes.

It's quite simple here. We have several levels, each of which operates equipment of a certain type. At each of these levels, the equipment is connected in a fault-tolerant configuration.

Super spine mesh

The Super spine architecture is typically used by large organizations that deploy large-scale data center infrastructures or campus-style data centers.

This type of architecture handles huge volumes of data transmitted by the east-west rule through data halls.

This architecture combines mesh and multi-tier topology, which ultimately provides the advantages of both, but significantly increases the overall network cost.

Conclusion

We have considered various types of network topologies used in data centers, as well as architectural options that can be applied in different types of data centers.

To continue the topic, I recommend attending open lessons that will be held as part of the "Data Center Network Design" course:

• October 2: "Data Center Networks — SP vs. DC. Which network design approach has become more widespread and why?". Registration link

• October 17: "Traffic Balancing in Data Centers". Registration link