The network where it all begins: how Underlay is structured in MWS Cloud Platform

If we imagine the cloud as a living organism, then the network is its circulatory system. It is the very infrastructure that connects computing resources, storage systems, security services, and connects all of this to the outside world — to the internet. Without its uninterrupted and predictable operation, no service, no virtual machine could simply function. Although I am, of course, exaggerating — it could function, but only the disk would be in read-only mode, as it is networked :)

In this article, I will detail the Underlay network — the very physical foundation of the new MWS cloud. We will discuss why we made the strategic decision to design it from scratch, rejecting traditional, often overly complex vendor approaches. We will talk about the principles that underlie the architecture: simplicity, fault tolerance, and scalability. And finally, we will look "under the hood": we will examine the leaf-spine fabric, the BGP and IPv6 protocols that have become its nervous system, and how we manage this complex distributed system through automation and monitoring.

This material is an attempt to explain complex engineering solutions without excessive simplification, but also without "magic." It is important to remember that a reliable cloud starts with a well-designed network. Okay, let's start with the most important question: why did we not copy a proven network architecture and instead go our own way?

Why did we not copy a proven network architecture

In general, if you already have a functioning public cloud — it seems logical to use proven solutions for the new one as well. At MWS Cloud Platform, we faced this very dilemma: we already had a cloud built on a classic and seemingly reliable networking platform from one of the leading global vendors, following its best practices. Its architecture relied heavily on EVPN VXLAN for building overlay networks. But for the new platform, we made a strategic decision not to replicate this approach but to take a different path. And here's why.

Firstly, complexity. EVPN VXLAN is a powerful but very complex protocol. The more complex the protocol, the larger its codebase, the number of states, and potential corner cases. In networking engineering (and probably in any field), there is a simple rule: the greater the complexity, the higher the likelihood of failure. We aimed to build a system that was as simple and predictable as possible, where each part is understandable and manageable. A monolithic, feature-bloated architecture is not our way.

Secondly, we needed vendor agnostic. Events of recent years have shown that the ecosystem can change instantly. A rigid dependence on one vendor and its unique implementations of protocols (like EVPN VXLAN) poses a significant strategic risk. We needed a network that could be built on equipment from various manufacturers without losing functionality and reliability. To achieve this, we had to abandon the most complicated, "heavy" protocols in favor of the simplest and most universal stack that works reliably everywhere.

Thirdly, control and transparency. By using vendor design "out of the box," you largely hand over control over the operational logic and diagnostics to the vendor. We wanted to understand our network down to the lowest level, to be able to modify it according to our unique requirements and scale. For example, one of the key principles we established was the use of commodity hardware. Wherever possible, network functions (like gateways) run on standard x86 servers with our software instead of specialized "boxes" with closed firmware.

In other words, we consciously opted out of the "same as everyone else" or "as it was before" path. Instead of inheriting a complex vendor architecture, we decided to build a network from scratch, based on the principles of minimalism, universality, and complete control. This was a strategic choice in favor of long-term sustainability rather than immediate convenience. And this choice immediately defined the requirements for the architecture: it must be simple and standardized enough to be replicated in any data center with any compatible equipment.

Being closer to the client: geography instead of abstraction

Our strategic choice in favor of simplicity and universality was not an end in itself. It became the foundation for solving a much larger task. The strategy of MWS Cloud Platform in the cloud business is to be as close to our client as possible, physically reducing latency and increasing service availability.

What does this mean in practice? It rejects the model of one or two hyper-scale data centers located somewhere in the center of the country. Instead, we are building a distributed cloud, with elements placed as close as possible to the points of business and user concentration. This "closer to the client" philosophy fundamentally changes the requirements for the network.

Imagine a map. Somewhere on it is our client, connected to the internet through their provider (ISP), and somewhere else is their virtual resource in our cloud. The task of the network is to provide a reliable, fast, and secure communication channel between them. To achieve this, we create Points of Presence (PoP) — nodes located at the boundary between the internet and our platform. All external traffic enters through these points.

But the cloud is not a monolith. Inside it, critical workloads are placed in isolated Availability Zones (AZ), which are physically independent data centers. And here our old good engineering principle comes into play: everything that is critical must be duplicated.

Between two AZs, we necessarily lay down two independent physical communication channels, separated by a distance of several kilometers. Why? So that no earth-moving mechanism, whether an excavator or a drilling rig — the worst enemy of any network engineer — can damage both channels simultaneously. This is the harsh practice of building fault-tolerant trunk lines.

We are also developing Edge computing infrastructure by placing compact data processing nodes even closer to the user — for example, at the sites of telecommunications operators. This allows tasks to be executed with the minimal possible latency, which is critical for 5G services, IoT, or interactive applications.

Finally, for corporate clients who cannot or do not want to trust their traffic to the public internet, we provide the option of direct, dedicated connectivity (Interconnect). The client can extend their channel to our PoP and receive a secure, predictable latency private channel directly into their cloud environment.

So what do we have in the end? Not just a "network somewhere out there," but a distributed platform with clear geographic coordinates. A platform where fault tolerance is embedded at the level of physical connections, and low latency is achieved through architecture.

However, such a distributed model poses a non-trivial question: how do we safely and privately route the traffic of thousands of different clients over a single physical infrastructure? To answer this, we had to clearly distinguish between two large concepts — Underlay and Overlay.

The Magic of Isolation: One Physical World, Many Virtual Ones

A distributed physical infrastructure is powerful, but not enough. Imagine: thousands of clients enter the cloud simultaneously — from a startup with a single web server to a large bank with a distributed ERP system. All their traffic inevitably passes through the same trunk channels and switches in our data centers. A fundamental question arises: how to guarantee that one client's data never mixes with another's? How to ensure privacy and security in a shared environment?

The answer lies in the division of network logic into two distinct, independent layers: Underlay and Overlay. Let’s break them down using an analogy.

Imagine a powerful, sprawling railway system (Underlay). Its task is to provide basic transportation. The tracks, switches, schedules—all serve one purpose: to deliver cargo from point A to point B as reliably and quickly as possible. The railway does not know or care what exactly is being transported in the cars. Its world consists of locomotives, stations, and tracks.

Now imagine that private closed cars (Overlay) travel along these common tracks. Each cloud client receives its own set of such cars. The client loads its "cargo" (traffic) into them, the cars are closed, and the railway delivers them to the desired station (host). Throughout the journey, the contents of the cars remain invisible to outsiders and are completely isolated from the cargoes of other clients.

Here’s how this analogy is embodied in our platform:

Underlay ("railway") is a factory of switches, which we will detail further. Its sole purpose is to ensure IP connectivity with predictable latency and fault tolerance between all critical points: between points of presence (PoP) and availability zones (AZ), among servers within the factory. Underlay operates on BGP and IPv6 protocols and only cares about efficient routing. It is "blind" to what lies in the payload of the packets.

Overlay ("private cars") is a virtual layer of isolation and services. When a client creates a virtual machine, the orchestration system automatically programs tunnels for them based on Segment Routing over IPv6 (SRv6). These tunnels start and end not on specialized equipment but directly on hypervisors (compute hosts) and gateways—ordinary x86 servers where a high-performance VPP (Vector Packet Processing) stack and our own SDN agent operate. Overlay is responsible for ensuring that the client’s traffic is packed into its "private car" (SRv6 tunnel) upon entering the network and delivered strictly to its virtual network (VPC) without the possibility of crossing with others.

The main conclusion is simple: Overlay does not hang in the air. All its "magic" of isolation, security, and flexible routing relies entirely on the speed, reliability, and predictability of Underlay. If the "railroad" stops, no "private cars" will go anywhere.

This is why we paid primary attention to the design of Underlay. In the next section, we will finally take a look "under the rails" and figure out how this network factory is structured, capable of handling the load from millions of "private cars".

The Heart of Underlay: Leaf-Spine Network Fabric

So, Underlay is the foundation. But how do you design this foundation to be simultaneously simple, fault-tolerant, and capable of processing terabits of traffic from thousands of clients? The answer is the modern architecture known as Clos, or Leaf-Spine, which has become an industry standard.

At its core are two distinct levels:

• Spine — powerful core network switches.

• Leaf — access switches.

The key principle: each Leaf switch is connected to all Spine switches. This forms a dense, fully connected network of channels. End devices such as servers and storage systems connect to Leaf. And for connectivity with the outside world (with points of presence — PoP and backbone channels), specialized switches called Border Leaf are used, which are also directly connected to all Spine, becoming an integral part of the fabric.

Why has this topology become our only correct choice?

Firstly, it provides determinism. In Clos architecture, the path of traffic from one server to another is always strictly defined and minimal (the classic three "hops": Leaf → Spine → Leaf). This is critically important for distributed applications, databases, and storage systems, which generate the lion's share of traffic in the cloud — the so-called "east-west" traffic (between services within the data center). It is this type of traffic, not "north-south" (from the internet to the cloud), that constitutes 80–90% of the entire load, and our fabric is optimized specifically for it.

Secondly, it provides fault tolerance through redundancy. The failure of one Spine switch or even several links does not paralyze the network. Because each Leaf is connected to all Spines, traffic is instantly redistributed across the remaining channels. This mechanism is called ECMP (Equal-Cost MultiPath). BGP, which operates on all devices in the fabric, sees multiple equal-cost paths to a single destination and balances the load among them. Thus, ECMP not only enhances fault tolerance but also allows for the aggregation of the bandwidth of many links.

Thirdly, it scales logically and linearly. We identify two scenarios:

1. Need more bandwidth within the fabric? We increase the number or bandwidth of the links from Leaf to Spine. If the capacity of the Spines is not enough, we add new Spine switches to the fabric, which increases the total bandwidth for all Leafs.

2. Need to connect new racks with servers? We install new Leaf switches and connect them to all existing Spines. The architecture remains unchanged, and the new computing power simply merges into the fabric.

Finally, and most importantly: in this fabric, a server is not a passive consumer but a full participant in the domain of dynamic routing. Each server runs a routing daemon (FRR), which "introduces" itself to neighboring Leaf switches via BGP. Thus, the server actively informs the network about its addresses, participating in constructing the connectivity picture. This is a fundamental departure from the model where a server was merely an endpoint device in the switching segment (L2).

The result is that we obtain a network fabric: unified, homogeneous, managed as a whole. It is unaware of virtual machines and clients; its task is to deliver IP packets between its ports with maximum efficiency. And it does this brilliantly, providing the very "railroad" over which the "private cars" of Overlay run.

However, the fabric made of iron and links is just the "body." To come alive and become an intelligent, self-healing system, it needs a "brain" and a "nervous system." In our Underlay, these were several carefully selected protocols.

The brain of the fabric: the protocols that bring everything to life

So, the factory made of iron and links is a powerful skeleton. But for it to become a living, self-repairing system, it needs "nerves" and "reflexes." In the world of networks, protocols serve this role. Our approach here was as minimalist as our choice of architecture: to use exactly as many technologies as necessary, and not a single one extra.

The entire brain of our Underlay is built on three pillars: BGP, BFD, and IPv6.

BGP (Border Gateway Protocol): the Internet protocol in the data center

The choice of BGP for the internal network of a data center may seem non-obvious. After all, traditionally IGPs (Internal Gateway Protocols) like OSPF or IS-IS are used for this purpose. We deliberately opted out of them. Why?

Scalability and Proven Reliability. If BGP can handle routing across the entire global Internet, with its hundreds of thousands of prefixes and constant changes, then it will handle our factory with ease. It is the most stress-resistant and refined protocol existing.

Flexibility and Control. BGP provides an incredibly powerful toolkit for managing routing policies. This is important for fine-tuning traffic management, for example at the border with external networks.

This is a controversial topic: eBGP vs iBGP. We use eBGP (External BGP), where each participant — Spine, Leaf, Border Leaf — and even each server has its own unique autonomous system (AS). Why not the classic iBGP within a single AS? Because eBGP with unique AS gives us unprecedented visibility for troubleshooting. By looking at the AS path (AS_PATH) in the route announcement, we can instantly determine from which specific server it came, without digging through additional databases. This simplifies the diagnosis of complex problems significantly.

BFD (Bidirectional Forwarding Detection): reflex to failure

BGP is a reliable protocol, but not instantaneous. For quick response to link or neighbor failures, BFD is used. This is a lightweight protocol that constantly exchanges "hello packets" between neighbors at intervals of tens of milliseconds. If several packets in a row do not arrive, BFD immediately (in the same fractions of a second) informs BGP, which then rebuilds the routing table by removing the "dead" path. BFD is that "reflex" which turns a fault-tolerant architecture into a genuinely quickly recoverable system.

IPv6: infinite space instead of workarounds

From the very beginning, we made a strategic decision: there is and will be no IPv4 in Underlay. Only IPv6. Why? The IPv4 address space is exhausted, and working with it often requires workarounds like NAT (Network Address Translation), which complicate architecture and diagnostics. IPv6 solves this problem once and for all. We have as many addresses as may be needed for growth in the foreseeable and unforeseeable future, allowing for simple and logical addressing schemes.

This provides clarity and simplicity: no imposed address translation, no conflicts of private ranges. Each device in the factory has its own unique, routable address. Moreover, the large length of the IPv6 address (128 bits) opens up interesting possibilities for encoding service information within them, which we utilize in our Overlay based on SRv6.

The result of our stack: we consciously abandoned a separate IGP (OSPF/IS-IS) and completely excluded L2 protocols (like Spanning Tree/LACP) from Underlay. The entire factory operates on pure IP (L3), where connectivity is ensured through dynamic routing via eBGP, and stability is maintained by ultra-fast BFD.

Thus, the "brain" of our network is not a collection of disparate technologies, but a cohesive, minimalist system. BGP builds routes, BFD reacts instantly to changes, and IPv6 provides unlimited room for maneuvering. Together, they transform the factory's hardware into an intelligent, self-regulating platform.

But even the most advanced network needs management, monitoring, and maintenance. How do we tackle this task on the scale of a distributed cloud? This is what the concluding part is about.

Life with the factory: management, automation, and monitoring

Designing and building a network is only half the battle. The second, equally important half is managing it, scaling it, and keeping it operational 24/7. When it comes to a distributed cloud with dozens of factories, hundreds of switches, and thousands of servers, manual management through CLI is out of the question. Our salvation is automation based on the principles of Infrastructure as Code (IaC), and multi-level monitoring that sees everything.

Automation: your code, a single truth, and OOB channel

The foundation of all our automation is Netbox. It is not just a database, but a single source of truth (Single Source of Truth — SSOT). It stores a complete model of the infrastructure: what devices exist, their names, their IP addresses and AS numbers, and how they are connected to each other. All subsequent automation takes data from here and only here, which eliminates desynchronization and human error.

Why don't we use ready-made tools like Ansible? Our experience has shown that for specific tasks, especially when working with infrastructure "at the network edge" (that's where the main difficulty lies!) and deep integration with internal systems, maximum flexibility is required. Therefore, we chose the path of creating our own automation framework. Essentially, we created an internal product — a set of Python libraries and utilities that directly integrate with Netbox as a data source, generate configurations from templates (Jinja2), and manage the lifecycle of devices. This has allowed us to precisely address our unique tasks without adapting to the limitations of off-the-shelf solutions and to perfectly integrate network management into the overall processes of platform development and operation.

An important detail: all automation work with network equipment (configuration uploads, information gathering) is conducted not through the main factory, but through a dedicated Out-of-Band (OOB) management channel. This ensures that even in the event of a serious failure in the data plane, we maintain control over the infrastructure and can, for example, remotely reboot a "hung" switch.

Monitoring: from good old SNMP to modern telemetry

Once I heard the principle: monitoring is something that can and should be improved endlessly.

SNMP still serves us well for collecting basic metrics: CPU and memory load, temperature, port status. It is a proven, universal standard.

BMP (BGP Monitoring Protocol) — this is our "window" into the control plane. We passively listen to BGP sessions on switches of different levels and get a complete picture of routing in real-time, as well as the history of all its changes. This is invaluable for incident investigation: you can rewind time and see which route disappeared and when, leading to the problem.

NETCONF/YANG is used where more granular and real-time data is needed. For example, we subscribe to a stream of changes in the state of BGP sessions on switches and receive these events via a push model, rather than polling the device once a minute.

Synthetic Monitoring actively simulates the behavior of real user traffic. Servers within the factory constantly "ping" each other, checking not only for availability but also for latency and packet loss. In the future, we are looking towards the TWAMP standard for more accurate end-to-end measurements, although the support for this technology among different vendors currently leaves much to be desired.

The shift to telemetry is our strategic vector. Our goal is to obtain all metrics (from port counters to chip states) via a push model, where the device itself sends data to the collector at a high frequency. This provides incomparably better detail and allows for faster response to anomalies. However, the implementation depends on the capabilities of the specific vendor, and here we are moving gradually, choosing the most mature solutions.

Conclusion: simplicity is a strategy

Looking back at everything we've discussed—from moving away from complex vendor stacks to a minimalist set of protocols and automation based on a single truth—one common thread becomes clear. This is a strategic choice in favor of simplicity, control, and predictability.

We have built not just a network for the cloud. We have built a network platform—uniform, automated, scalable, and observable. A platform where every component is understood, and every solution is justified. A platform that is not afraid of excavators, grows easily alongside the business, and serves as a completely reliable foundation for any services that will operate on top of it.

Now our task is to make the most of this platform, creating services based on it that our clients will appreciate. More detailed information about the services can be found on the MWS Cloud Platform website.