Setting up an Anycast address within a budget test stand

Preface

As a test stand, we will have a Mikrotik hAP lite router and a Proxmox cluster on RockPi mini-computers (analogous to Raspberry Pi) with virtual machines based on the Debian distribution.

Why exactly a Proxmox cluster? Just because :)

It would be possible to manage with virtual machines on a computer/laptop created using VirtualBox or a single mini-computer with Proxmox, but I have this thing for home experiments, so I used it.

Hello everyone! My name is Artyom Druz, I work as a senior engineer at Kontur, where I configure and maintain digital telephony based on Asterisk and OpenSIPS. For digital telephony in the "classic form," UDP transport and, accordingly, Anycast addressing are used. For creating fault-tolerant telephony, this is exactly what the doctor ordered. Some may argue that using UDP is a thing of the past, and now WebSocket/WebRTC or even QUIC are in vogue. But if you turn on WireShark and try to find voice packet exchanges in the vast stream of your traffic in your favorite messenger, you will most likely find UDP traffic. It is for this reason that setting up fault tolerance for services using UDP transport using "old methods and technologies" remains relevant.

Designing the network

Before starting the setup, let's sketch out what we want to see as a result. Within the existing stand, there is a limitation in the form of MikroTik hAP lite: due to the chip on the smips architecture, we cannot use the BGP protocol, which is recommended for such things. Therefore, we will take the OSPF protocol for announcing IP addresses from virtual machines. For harsh production with hundreds of thousands of requests per second, where every data packet is important, this, of course, is not suitable. But within the test stand, such loads are not planned ;)

For the network project, these inputs are sufficient, as the rest of the scheme will be implemented at the software level.

Configuring OSPF on MikroTik

To configure the reception of announcements in MikroTik, we need a combination of three elements:

1. OSPF Instance – here we set the name for our OSPF installation

2. OSPF Area – description of the OSPF zone that is linked to the OSPF Instance

3. OSPF Interface Template – a template for authenticating announcements from virtual machines and creating interfaces linked to the OSPF Area. In it, we specify the interfaces within which we need to work with announcements (in my case, this is the common bridge bridge-lan). Here, timers are also set according to which announcements will be executed and processed, and the absence of announcements within the Dead Interval will be treated as a trigger to remove the announcing server from the routing table. These intervals are interrelated, so you cannot set 1 second everywhere to speed up convergence when the announcer fails.

   Read also:

   • Restoring a non-working keyboard using QMK and RP2040

Below is a screenshot with the settings of my stand.

Configuring address announcement on virtual machines

First, we create a dummy interface on each of the virtual machines and start it.

We add the following structure to the /etc/network/interfaces file:

auto dummy0
iface dummy0 inet static
    address 172.18.0.64/32
    pre-up ip link add dev dummy0 type dummy
    pre-down ip link delete dev dummy0 type dummy

Important. The IP address in the interface configuration must be outside the existing subnets, otherwise the scheme will not work correctly.

Then we start the interface with the command ifup dummy0

For those whose Linux distribution uses NetworkManager, there is a command that essentially does the same thing but in one action:

nmcli connection add type dummy ifname dummy0 ipv4.method manual ipv4.addresses 172.18.0.64/32

Next, install bird

apt install bird2

And make changes to /etc/bird/bird.conf

log syslog all;
protocol device {
}
protocol kernel {
        ipv4 {
              import none;
              export all;
        };
}
protocol static {
        ipv4;
}
protocol ospf myAnycast { # myAnycast - any arbitrary name for the OSPF instance at the virtual machine level
        ipv4 {
                import all;
                export all;
        };
        area 0 {
                stubnet 172.18.0.64/32 { # 172.18.0.64 - Anycast address not included in existing subnets in the local network
                };
                interface "enp0s11" { # enp0s11 - the name of the network interface from which Anycast address announcements should be made
                        cost 1;
                        type broadcast;
                        hello 3; # Hello Interval from the template in MikroTik
                        retransmit 2; # Retransmit Interval from the template in MikroTik
                        wait 2;
                        dead 4; # Dead Interval from the template in MikroTik
                        authentication simple; # Authentication from the template in MikroTik
                        password "myOSPFpassword"; # Auth. Key from the template in MikroTik
                };
                interface "dummy0" { # Name of the dummy interface on which the Anycast address is configured
                        stub;
                };
        };
}

Now restart bird with the command systemctl restart bird, and within a few seconds in the MikroTik interface (provided everything is set up correctly) we should see something like this:

Checking the operation of the Anycast address...

... from the router side

Here we will use a simple ping from the console:

There is contact! This could be the end, but I want a more vivid example :)

... from other virtual machines

In this check, I decided to test the telephony service operation in a scheme where 4 Asterisks from different addresses connect to the Anycast address used by OpenSIPS (one on each virtual machine where the Anycast address is configured).

The reader can use any voice service working through UDP transport instead of SIP servers and software PBXs for verification. In my screenshot of tmux with sngrep running on two virtual machines (I turned off the third one for the duration of the check to get a readable screenshot). It can be seen that:

• Asterisk with address 172.18.0.24 was distributed by MiroTik to the first OpenSIPS

• Asterisks with addresses 172.18.0.22, 172.18.0.23, and 172.18.0.25 were distributed to the second OpenSIPS

How to make another traffic distribution strategy (for example, so that each packet is distributed to the announcers of the Anycast address using the round-robin strategy) on MikroTik is a separate topic, and we will not cover it in this article.

Thus, the goal can be considered achieved and you can start experimenting with your own Anycast address ;)

Conclusion

In the article, I did not mention anything that would tie the reader to Proxmox in general or to its specific implementation on mini-computers with an ARM processor in particular. The article is a guide on setting up an Anycast address on your computer or in a local network to try to implement a service using the features of this type of address. In my example, Proxmox was needed only as a general environment for creating virtual machines.

As a result, we now have our own Anycast address, announced via OSPF, with which we can experiment and test hypotheses. Undoubtedly, this scheme has shortcomings (for example, routing outgoing packets through the interfaces of virtual machines performing the announcement). But to start understanding this topic, this springboard, in my opinion, is enough.

Important caveats that were mentioned above in one form or another:

• The solution described in the article is not for harsh production. It can be used for small businesses or self-development tasks, but definitely not for systems with thousands of RPS. At a minimum, for fast routing convergence in case of failure of one of the nodes, BGP should be used instead of OSPF.

• Anycast addressing is about UDP traffic, which has no concept of guaranteed connection and data delivery (anecdote). If you have TCP traffic (websites, REST APIs, database connections, etc.), then for fault tolerance and/or high availability of services, you need load balancers that handle such traffic or specialize in a specific service (HAProxy, PgBouncer, Galera Cluster, Redis Sentinel, etc.).

That's all. Thank you for reading the article to the end.