Building a Multisourced Infrastructure Using OpenVPN

10.100.100.2   0.0.0.0       255.255.255.255  UH    0 0  0 tun0
10.100.100.0   10.100.100.2  255.255.255.0    UG    0 0  0 tun0

To address this issue, we developed a dynamic routing dæmon called cube-routed (download it from www.cohesiveft.com/multisourced-infra). It shares routing information between vpnsrvA and vpnsrvB and adjusts routing tables depending on which client connects to which server in near real time. Its internal structure is not very complex. One thread connects to a local OpenVPN dæmon process via its management interface (see the management option in the OpenVPN configuration file) and regularly runs the status command to update the list of clients connected locally. Another thread publishes this information for the remote instance of cube-routed. The third thread regularly reads a list of connected clients from the remote instance of cube-routed. Finally, the fourth thread adjusts the local routing table based on the following two rules: 1) adds a host route for every host connected to the remote OpenVPN server and 2) deletes the host route for every host connected to the local OpenVPN server.

cube-routed instances will exchange information over the management subnet we selected earlier. Create a second tunnel tun1 between vpnsrvA and vpnsrvB. vpnsrvA can be a server with IP 10.200.200.1, and vpnsrvB is its client with IP 10.200.200.5. You can use the configuration files from Listings 1 and 2 as a basis, but remember to adjust the IP addresses and select a different port—for example, you could add port 11940 to both the server and client. Start both OpenVPN dæmons, and use ping 10.200.200.1 and ping 10.200.200.5 to verify connectivity between them.

Now, create configuration files for cube-routed on both vpnsrvA and vpnsrvB, as shown in Listings 4a and 4b, and start both instances as root with the path to the configuration file as the only parameter (note that OpenVPN must already be running, and the tun0/tun1 interfaces on both vpnsrvA and vpnsrvB must be up).

mgmt_interface = tun1
data_interface = tun0
remote_mgmt_ip = 10.200.200.1
remote_data_ip = 10.100.100.1
openvpn_mgmt_pass_file = /etc/openvpn/pass
openvpn_mgmt_port = 5656
cube_routed_port = 5657

Once you start everything and after several minutes of initial convergence time, host1 from the example above will be able to communicate with host2, even though they connected to different OpenVPN servers. Thus, you've achieved a fully fault-tolerant virtual LAN connectivity, with data traffic encryption as an added bonus.

This implementation is not without its limitations. First, applications that use broadcast or multicast will not work with OpenVPN's tun device. You can use the same network layout as described here, but instead of tun, experiment with OpenVPN's tap device to work around this. Second, latency of network links over the public Internet is significantly higher than that of Ethernet. If this is an inherent requirement for your application, you probably should leave this part of your infrastructure single-sourced. Third, because we use UDP-based tunnels, OpenVPN links will tend to go up and down more often than Ethernet, especially during times of network congestion. You can implement data caches, avoid long-lived TCP connections, focus on network exception-handling logic and experiment with TCP tunnels to reduce negative impact. Finally, there are exactly two OpenVPN servers in this setup. This generally should be sufficient, as it doesn't affect the number of actual hosts that you have connected to your multisourced infrastructure. If for some reason you need more than two, it becomes much more difficult to implement route sharing among cube-routed instances. In that case, you might want to consider a messaging system instead of raw sockets (for example, RabbitMQ). All in all, in our case, we found that the overall benefits of a multisourced infrastructure far outweighed the problems caused by these limitations, particularly if you design your architecture with these limitations in mind.

Multisourced infrastructure is a logical extension of its single-sourced predecessor, similar to the distributed service-oriented architecture, which came after monolithic applications and enabled greater flexibility, a faster development cycle and higher availability. It can help you design a smarter architecture and avoid a lock-in to a single hosting provider, on top of a standard time-tested open-source OpenVPN.