Online migration of EVPN-VXLAN fabric between data centers: euNetworks → QupraDC without service downtime

My name is Rene, I’m a network engineer at FirstVDS. In the first part, I talked about how we launched a small European platform in Amsterdam: one Leaf, one Spine, routed host networking for hypervisors, EVPN-VXLAN as the service plane, DDoS protection in a separate VRF, OOBM, and a Flow collector.

This part is no longer about the initial design, but about testing it against reality. The euNetworks data center is shutting down, we need to move the equipment, we can’t stop client workloads, we can’t change the IP addressing, and we can’t halt sales of new services either.

The good news is that the initial architecture was not built around one big all-in-one box, but as a small but properly designed fabric. This is exactly what allowed us to avoid making one risky “switch everything at once” cutover, and instead carry out the migration through several controlled intermediate states.

Background: The data center is shutting down, services must stay up

The platform ran normally for a while. We launched services, connected standard transit, added DDoS protection, and settled into regular day-to-day operations.

Then came the news: the euNetworks data center is shutting down, and we need to move the equipment.

For the business, the question was simple: what do we need to buy to move? Next came a second, more interesting question: can we avoid buying any extra equipment, or rent it temporarily, and migrate without interrupting client services?

The requirements ended up being as follows:

• Avoid purchasing additional network equipment where possible;

• Do not halt sales or interrupt client services;

• Migrate servers gradually;

• Retain the existing client IP addressing;

• Do not condense the entire migration into one large maintenance window where everything needs to be switched over at once.

Translating this into networking terms, I realized we just need one more Leaf in QupraDC that can be temporarily integrated into the existing fabric. We leave the old Spine in euNetworks, physically install the new Leaf in QupraDC, and set up a temporary IP transport link between the two sites.

The initial idea was to use this Leaf as temporary equipment only for the duration of the migration. But the provider informed us they would not be able to take it back after the move. I then proposed not to treat the purchase as a forced loss, but to use the situation as an opportunity to improve the architecture: leave the second Leaf in the new location, and after the migration spread the server connections across two switches.

This forced move also turned into a project to improve fault tolerance.

Temporary DCI and the first servers in QupraDC

To connect the new Leaf in QupraDC to the existing fabric, we needed a channel between the two data centers.

In fact, we almost didn’t care exactly how the provider implemented this channel within their network: dark fiber, lambda, VLAN transport across the operator’s switching fabric, L2VPN, L3VPN, or any other transport type. For our task, the technology didn’t matter as much as specific technical characteristics:

• bandwidth of at least 40G;

• ability to transmit IP packets between the old and new site, preferably via a single p2p link with /31 addressing;

• MTU 9216;

• ability to quickly decommission the channel after migration is complete;

• low cost.

We didn’t need Dot1q tags, because we weren’t carrying user VLANs between data centers, and even if we had that requirement, we would have handled it using EVPN-VXLAN. The underlay interfaces on the Leaf and Spine were standard L3 ports with IP addressing. The temporary channel was supposed to carry underlay IP packets, with VXLAN-encapsulated overlay traffic running on top of them.

MTU was critical. Inside the underlay, not only regular IP packets are transmitted, but also virtual machine traffic with VXLAN encapsulation. If a client VM sends a standard 1500-byte packet, VXLAN/UDP/IP overhead is added to it. If this is overlooked on an inter-data center link, you can run into unpleasant fragmentation issues or black holes for part of the traffic. A separate practical reason is the service requirement to use jumbo frames for network storage: such scenarios also require a healthy MTU headroom in the fabric.

On underlay interfaces, we use an MTU of 9216 bytes. This is the maximum packet transmission size for our switches, so there is no practical sense to set a lower value inside the fabric we manage. This MTU provides sufficient headroom for VXLAN encapsulation, simplifies operations, and is used as a unified standard across all our IP fabrics.

It is important to emphasize: we did not build stretched L2 between data centers. The temporary L2 link from the provider was used as transport for our underlay L3 connectivity. For the fabric, the new Leaf looked like just another Leaf that simply happened to be a bit farther away than previously expected.

No stretched client VLANs over xSTP, no attempts to merge the two sites into a single large L2 domain. Only underlay IP connectivity and EVPN-VXLAN running on top of it.

The new Leaf had to be exactly the same model, so that the network chips and software versions were as identical as possible. This is important during a migration under pressure and with limited time. Theoretically, EVPN should work fine across different platforms, but in practice, mixing different chips, different software versions, and different control plane behavior profiles can turn the migration into a debugging session for yet-unresolved bugs.

After the new Leaf was physically installed in QupraDC, the work sequence was as follows:

1. Connected the new Leaf to the existing Spine via a temporary 40G link.

2. Configured basic underlay IP connectivity.

3. Verified reachability between fabric members.

4. Established underlay and overlay BGP sessions.

5. Monitored the stability of BGP sessions and link quality.

6. Verified that the new Leaf participates in the EVPN control plane and receives the required routes correctly.

At this stage, it was important not to rush with client workloads. First, the control plane had to work stably: reachability of VTEP addresses, MP-EBGP EVPN, transmission of routing, MAC/IP information and EVPN Type 5 routes, as well as programming of the EVPN database. Only after that could the first servers be migrated.

When the new Leaf became part of the fabric, we took one parent server out of client workload, physically moved it to QupraDC and connected it to the new Leaf.

The key task was to preserve the service model and addressing scheme. The client should not have noticed that the server was now located in a different data center. To do this, we transferred the required service configurations to the new Leaf, verified routing to the hypervisor, virtual machine availability and traffic flow in both directions.

This resulted in a temporary but operational topology: some servers remained in euNetworks, some were already located in QupraDC, the old Spine remained physically in euNetworks, the new Leaf was in QupraDC, a temporary channel operated between the sites, and logically all of this remained a single fabric.

The route to a virtual machine on the migrated server could look like this: internet → old Leaf → Spine → inter-data center channel (DCI) → new Leaf → hypervisor → VM.

This was not the most direct path, but it was acceptable for the intermediate state: latency increased by single-digit milliseconds, which is not so critical for our VDS workload. The main thing was that the service remained available, and we got the opportunity to migrate servers in batches.

After a successful test, we began migrating client workloads gradually. Approximately half of the servers moved to QupraDC and started connecting to the new Leaf, while the old site continued to serve the remaining portion of the workload.

In network migrations, the winner is often not the one who does everything at once with a single switchover, but the one who knows how to operate safely in an intermediate state for a long time. In our case, the intermediate state was clear, observable and controllable.

Migration of external connectivity and infrastructure

When roughly half of the load was in QupraDC, it became inefficient to route all external traffic through the old facility. The servers were already physically located in the new data center, but all incoming and outgoing traffic still passed through euNetworks, the old Leaf, and a temporary inter-datacenter link.

Luckily, our IP transit operator also had a presence in QupraDC. This allowed us to migrate one of the BGP sessions with the upstream provider to the new location.

It is important to clarify this point: in QupraDC, we did not replicate the old 2×10G physical setup. When migrating the peering link to the new location, we immediately enabled a 100G connection on the new Leaf. From a service perspective, this looked like migrating one of the two BGP sessions along with its VLAN and /31 point-to-point addressing, while physically it was a switch of the new facility to a 100G uplink.

After this, the setup looked like this:

• one BGP session with the upstream provider remained on the old Leaf in euNetworks via the old 10G peering link;

• the second BGP session with the same upstream provider was established on the new Leaf in QupraDC via the 100G connection;

• both Leafs continued to participate in the shared fabric;

• external traffic started being distributed between the two facilities.

From our side, a second connection to the same operator emerged, this time from the new location and on new physical infrastructure. Each Leaf had its own local default route from the upstream provider. Additionally, the default route could be propagated inside the overlay as an EVPN Type 5 route, to maintain connectivity between parts of the temporary setup across the fabric.

On the operator's side, they also used multipath routing and tried to deliver incoming traffic as symmetrically as possible. As a result, we got two beneficial effects.

The first was that we offloaded the temporary channel between data centers. Traffic to servers that were already in QupraDC could now arrive and depart via the uplink in the same location, rather than necessarily passing through euNetworks.

The second was that we gained the first practical element of external connectivity redundancy. Yes, this was still a transitional setup, but it was no longer a single external connection for the entire load distributed between the two data centers.

The DDoS service was migrated in parallel and according to the same logic. The DDoS protection provider was also present in QupraDC, so its connection could be transferred to the new location without changing the previous model. The interface with it also consisted of two BGP sessions on two different units, with different VLAN tags on two different Leaf switches. From the point of view of routing and filters, everything remained the same: only the physical connection location changed.

At this stage, the state looked like this: some servers were already in QupraDC, some were still in euNetworks, external transit, including protected transit, worked on both sites, 100G interface to the upstream was already used in QupraDC, both Leaf switches participated in routing, and migration continued without stopping the client service.

After migrating the main part of the client load, it was time to migrate the infrastructure virtual machines.

Route reflector needs to be migrated carefully. If both are lost at the same time, hypervisors will not be able to properly distribute routes to virtual machines, and the network will start losing information about the reachability of VDS prefixes.

Therefore, we migrated them one by one: checked the current state of BGP sessions and the set of reflected routes, took one route reflector out of active operation, migrated or restarted it in the new location, waited for the BGP sessions to recover, checked that the routes from hypervisors were visible on Leaf switches again, and only after that moved on to the second route reflector.

In the network infrastructure, a route reflector is not just another virtual machine. It is an element of the control plane, and it needs to be handled accordingly.

After migrating the RR and the remaining infrastructure, we migrated the second BGP session with the upstream to QupraDC. Recall that external connectivity in the new location was already built on 100G: the target scheme provided for two independent 100G interfaces, one for each Leaf switch.

By this time, all client and infrastructure load was in QupraDC. The old site actually remained only a place where old Leaf and Spine switches were still physically standing, as well as one infrastructure virtualization server, which needed to be removed last.

Final migration of hardware and bringing the scheme to a fault-tolerant form

When there was no more client workload left in euNetworks, we were able to transport the old network equipment and other infrastructure remnants.

At this stage, the old Leaf and Spine were no longer critical for client services in their original role. The primary workload was located in QupraDC, external BGP sessions had also been migrated there, and the temporary inter-datacenter channel continued to maintain connectivity until the work was completed.

We powered down the old Leaf and Spine, dismantled them, transported them to QupraDC, installed them, and powered them back on.

Key point: we did not need to make any configuration changes. The underlay addressing was preserved, device roles were preserved, and BGP sessions came back up after the devices were powered on. From the fabric's perspective, it looked almost as if the patch cords had become shorter, and the devices had moved physically closer to the rest of the workload.

After that, the migration could be considered complete: the client workload was in QupraDC, infrastructure services were also located there, both BGP sessions to the upstream were established from QupraDC via 100G links, the connection to the DDoS provider had been migrated to QupraDC, and the old network equipment had been physically transported to the new site.

We immediately dismantled the temporary inter-datacenter channel, which was only needed as a migration tool.

After the migration, we discovered the expected intermediate situation: most servers were still connected to only one Leaf — the very one we had first installed in QupraDC for the migration.

That was sufficient for the migration itself. For normal operation, it was no longer sufficient. If this entire Leaf were to go down, a significant portion of servers would lose external connectivity.

Therefore, as the next step, we started reworking server connections so that each hypervisor had one active 10G link to each Leaf.

It's important here not to confuse this with classic LACP/ESI-LAG. We did not bundle two physical links into a single L2 aggregate, nor did we build multihoming via LACP. The model remained exactly the same as before. The only difference is that these two L3 paths now go not to the same Leaf, but to two different ones.

This way we got simple and clear access-level fault tolerance. If one link goes down, traffic stays on the second one. If one Leaf fails, for the hypervisor this also looks like the loss of one of the next-hops, and the second path keeps working.

From a diagnostics perspective, this scheme remains very transparent. There is no need to figure out exactly what happened with the LACP state, which of the pair of switches considers itself active, and how the additional EVPN multihoming signaling worked. In this scheme, DF election is tied to Type 4 — Ethernet Segment route, while aliasing and mass-withdraw are tied to Type 1 Ethernet A-D routes: per-ES and per-EVI. This is a working model, but during failures it has more states that need to be able to check and debug.

In our case, there is an interface, a connected route, a next-hop, and a BGP announcement of the VM prefix. Each element can be checked separately.

An important nuance is IPMI. Most servers have only one separate management port. It cannot be connected to two Leafs at the same time without an additional scheme. So we distributed IPMI ports symmetrically: some servers connect to the first Leaf, some to the second. If one Leaf fails, we lose access to the IPMI of part of the servers, but not all of them at once. For this task, this is an acceptable compromise.

After the final switching, the external connectivity looked as follows:

• each Leaf has its own 100G connection to upstream;

• each BGP session is established independently;

• both Leafs can announce the same set of our prefixes;

• failure of one Leaf does not kill the entire external channel;

• failure of one 100G link does not leave the facility without transit;

• incoming and outgoing traffic can be distributed via multipath.

Failure testing and one Spine as a compromise

After migration and reconnection of server links, we conducted a series of failure tests. This is a mandatory stage: fault tolerance cannot be considered present just because it is drawn on the diagram.

We tested several scenarios.

First scenario — rebooting a single Leaf. If it becomes unavailable, servers remove the unavailable next-hop from the ECMP group and continue using the second path, while external traffic is redistributed through the other Leaf. For client services, this should not appear as a full-scale outage: brief loss of individual packets may occur during convergence, but there will be no prolonged downtime.

Second scenario — rebooting the second Leaf. The test is symmetric, but it still needs to be carried out separately. In real operation, it is often found that "identical" devices differ in minor details: the set of connected servers, BGP peering sessions, policies, or physical switching.

Third scenario — disabling server interfaces. In case of physical disconnection, everything works quickly: the hypervisor's network stack detects carrier loss, the route via this interface stops being used, and traffic remains on the other ECMP path. In this case, there is no need to wait for the BGP holdtime to expire, as the issue is visible at the interface level.

BGP timers are important for other cases: for example, if the BGP daemon on the neighboring side hangs, while the physical link remains up. For such scenarios, the worst detection time now depends on BGP/BFD settings.

Fourth scenario — disabling upstream links. When the external interface becomes unavailable, the connected routes associated with it are removed from the routing table, the BGP session with the operator loses transport connectivity and is reset. After this, routes via this link stop being used, and traffic is redistributed through the remaining external path.

Fifth scenario — testing the DDoS segment. Everything here is more or less similar, testing the failure of the link to the DDoS provider.

BFD is used for fast failure detection within the IP fabric itself. Thanks to BFD, we do not wait for long BGP timeouts: if a neighbor stops responding, the path is marked as down faster, and traffic is redistributed to other next-hops within the ECMP group

In the end, we got what we wanted: basic fault tolerance at the server connection level, distribution of external transit across two Leafs, and a multiplefold increase in external bandwidth.

An attentive reader will notice: after all the improvements, there is still only one Spine. Does this mean it remains a single point of failure?

Formally — yes, if we look at the fabric as a canonical Spine-Leaf architecture. In an ideal world, there should also be at least two Spines. Then the failure of any single device at any level does not lead to a loss of fabric connectivity.

But in our current topology, a Spine failure is not equal to a complete stop of client services.

The reason is that the two Leafs after the migration are not tied to Spine as the only exit point to the outside or the only gateway for servers:

• each Leaf has its own BGP session to the upstream;

• each Leaf receives its own local default route from the operator;

• both Leafs announce the same set of our external prefixes;

• hypervisors are connected to both Leafs via separate L3 links;

• routes to virtual machines are distributed via BGP;

• if there is no BGP signaling for a specific VM prefix, the network does not consider this path operational.

That is, the situation "traffic arrived at the Leaf, and there is nowhere to deliver it further" should not occur in normal operation. If a specific host link goes down, the corresponding path disappears. If a Leaf loses external connectivity, traffic can go through the second Leaf, which has its own upstream.

At the same time, one Spine still remains a technical compromise. It is acceptable for the current scale and current topology, but it should not be passed off as an ideal architecture. When the next Leaf is added or the requirements for fabric-underlay fault tolerance increase, I will propose returning to the question of a second Spine.

At some point, one Spine stops being a reasonable compromise and becomes technical debt. Then the fabric will need to be brought to a more canonical Clos topology with redundancy at every level.

Conclusions for the second part and what's next

As a result, we didn't just move from euNetworks to QupraDC. We used the forced migration as an opportunity to improve the architecture.

Before the migration, the scheme was minimal: one Leaf, one Spine, all load in one physical location, external uplink 2×10G, the Leaf simultaneously performs the roles of server leaf and border leaf, fault tolerance at the fabric level is minimal.

After the move, the scheme became noticeably stronger: two Leaf switches in the new location, server connections are distributed across two Leaf switches, external BGP sessions to the upstream are established from different Leaf switches, external physical connectivity in QupraDC was immediately assembled at 2×100G, the DDoS service was migrated to QupraDC, and its connections are also distributed across two Leaf switches.

The key point is that we did not change the architectural model during the migration. The new Leaf was integrated into the already existing fabric, servers continued to operate in routed mode, VM prefixes continued to be distributed via BGP, and EVPN-VXLAN remained the common control and service foundation of the site.

That is exactly why the migration was fully controlled. We did not turn the move into one large risky operation that would require fitting all stages into it, but instead operated in intermediate states for an extended period: first one Leaf in the old location, then the second Leaf in the new one, then part of the servers in the old location and part in the new, then BGP sessions with the upstream active at both sites, then all workloads moved to QupraDC, then the physical relocation of the old equipment.

This is, perhaps, the main conclusion of the entire second part: good migration is not the moment when someone presses a big red button and hopes everything will work out. Good migration is a sequence of states, each of which can be monitored, verified, and maintained for as long as necessary if needed.

The second conclusion — temporary schemes must be designed just as carefully as permanent ones. Even if the channel between data centers is only needed for a few weeks, it must have a clear purpose, clear limitations, observability, and shutdown criteria. Otherwise, a temporary migration construct easily turns into an overlooked part of production that everyone forgets about.

The third conclusion — fault tolerance sometimes emerges not as a separate large-scale project, but as an opportunity that is used correctly. We needed a second Leaf for the move anyway. It could be perceived as a forced purchase, or it could be integrated into the future target scheme. We chose the latter option and gained basic stability at the access level.

The next necessary step is a second independent upstream. Two BGP sessions with a single operator provide redundancy at the level of our peering points, Leaf switches, and physical links. But they do not protect against failures within the ISP's own network: backbone issues, routing errors, provider route server outages, or unfavorable changes to its policy. In such a situation, the fact that we have two active sessions on our end becomes far less relevant: if the issue lies within the operator's network, both peering points can degrade simultaneously.

Therefore, at least a semi-automatic backup path via a different operator is needed. At the initial stage, this does not have to be a full-fledged load-balanced scheme. It is sufficient to have a backup default route from the second upstream, assigned lower priority via BGP policy: the primary default route is used via the current operator, and if it fails, traffic is automatically routed via the backup path.

But there is an important nuance here: such a switchover will only trigger automatically if the BGP session has actually gone down and the route has been withdrawn. Failures within the operator's network do not always present in this way. The session may remain active, the default route may still be present in the RIB, while connectivity quality has already become unacceptable. In such cases, monitoring, clear switchover procedures, and functional OOBM access are required, so that routing policy can be adjusted quickly without relying on the broken network.

And it is entirely possible that at some point it will make sense to migrate external routing to a full pair of carrier-grade border routers with a full view. But this is a later step for us: hundreds of gigabits of traffic, multiple upstreams, more complex traffic engineering, and separate requirements for peering policy.

Another obvious step is adding a second Spine. While a single Spine remains an acceptable compromise for the current topology, it will need to be added as Leaf switches grow further and fault tolerance requirements increase. The fabric will then come closer to the canonical Clos topology: multiple Leafs, multiple Spines, and no single point of failure not only in the underlay, but also in the overlay control plane. In our scheme, the Spine participates in distributing EVPN information, so its redundancy is important not only for the transport layer, but also for the overlay control plane.

The main result has already been achieved: we moved away from the minimal scheme with a single Leaf, migrated the platform online, preserved client addressing, increased external bandwidth, and got a more stable architecture. And it all started with a simple requirement: build a small, low-cost platform with room for future growth.

Article author: Rene, network engineer at FirstVDS