Developing P2P Protocols across NAT
This problem can be solved by joining the problem, rather than fighting it head on. In order to achieve peer-to-peer traffic across NATs, we have to modify our P2P mesh model slightly to make it a hybrid of a traditional star model and modern mesh model.
So, we introduce the concept of a rendezvous server, or mediator server, which listens on a globally routable IP address. Almost all peer-to-peer protocols have traditionally relied on certain supernodes, or in other words, in P2P, all nodes are equal but some are more equal. Some nodes always have acted as key players in any P2P protocol. If you have heard of a BitTorrent tracker, you know what I mean.
A rendezvous concept is nothing new in the P2P world, nor is the star model totally done away with in P2P.
Coming back to our original NAT problem, private IPs obviously can browse the Internet through NAT devices, and thus they can talk HTTP through port 80 or through a proxy HTTP port over TCP. So private IPs can almost always open TCP connections to global IP addresses. We use this fact to make the private IP connect to a mediator or rendezvous server through TCP.
Our solution relies on the fact that all the P2P nodes are constantly in touch with a rendezvous server, listening on a global IP address through a persistent TCP connection. Remember that P2P nodes are both client and server at the same time, so they can originate connections as well as serve connection requests simultaneously.
It is through this TCP connection that we inform a particular P2P node that another node wants to talk to it. Then, the target node sends a request following which the peer sends the connection request as a response to the request.
Because the private machines behind a NAT device do not have a routable IP address, the only way for us to access them from outside the NAT device is through the mapping that the NAT device maintains for the machine to talk to the external world. For each connection originated from the private IP, a unique port is assigned at the NAT device. For us to talk to the private IP, we have to send our packets to that particular port assigned for the private IP's connection to the external world. Now, we know that there is no notion of connection in the UDP world, so NAT assumes that if a reply doesn't come for a UDP request in about 60 seconds, the connection is deemed non-existent and closed.
So now we have another problem—that of determining the port assigned at the NAT's public interface for the private IP connection. This can be inferred by inspecting the source address of the UDP datagram that reaches any global IP.
So far so good. If we are not behind NAT, we can use the previously mentioned technique to initiate communication with a private IP using the rendezvous server.
However, reality tells us that P2P peers are more likely to be behind a NAT than otherwise. So, this solution is not enough. We want to initiate a P2P connection from behind a NAT device ourselves. So, now we have two NAT devices in the picture, one behind each P2P node.
Now the real fun begins. First, let's redefine our goal in the light of this new twist to the problem and attack it step by step. What we want to do now is use the rendezvous server and inform the target P2P node to send us a request, but we are behind a NAT.
So, for any external party to talk to us, we should have a global IP/port combo that exists at the NAT public interface. First we have to create one for ourselves. Only then we can receive communication requests coming from outside the NAT network.
We can create a mapping for us by sending a packet to a global IP. The global IP can then figure out our mapping by inspecting the from address. But how do we inform our P2P node of this address? For that we can use the TCP connection with the rendezvous machine. But, only the global IP to which we send the packet knows our association, so how do we figure that out? It's simple. The global IP can send that information to us as a reply in the packet payload to us.
Assuming that we somehow obtain a public IP, port pair and figure that out, we tell the mediator that we are listening at that public IP/port pair and request the P2P target node to initiate a request to us. Subsequently, we can connect to it as a reply to that message.
But, then we cannot receive packets from the P2P target node, because NAT is not expecting a reply from that global IP. In fact, some NATs that show full cone behavior allow packets to come from any IP, but most NATs do not—back to square one.
Consider this: if both P2P nodes behind the NAT send packets to each other's public IP/port, the first packet from each party is discarded because it was unsolicited. But subsequent packets are let through because NAT thinks the packets are replies to our original request. And voilà the hole is punched, and UDP traffic can pass through directly between the P2P nodes.
Unfortunately, NATs also differ in their behavior of assigning public ports for different destination IPs. Most NAT devices fortunately do not change public ports between requests to different destination IPs, so we can safely assume that.
So first we send certain probe or discovery packets to two different IPs and figure out the behavior of the NAT. If it is found to be consistent, our approach will work. In the unlikely case that we bump into symmetric NAT behavior that varies the port between requests, we can figure out the delta by which the port number varies. And, using this we can guess the port assigned for a particular request.
The reason we are so particular about this is because the first packet to our P2P destination behind NAT is dropped by NAT. So, all we can do is guess. In practice, however, it works fairly well. This is why it is important that the P2P nodes keep the source and the destination ports the same for communication.
Once this hole punching procedure is performed, the two P2P nodes can communicate with each other without the help of the rendezvous machine. So the rendezvous machine is useful only for informing a P2P node about an incoming connection and informing each of the communicating peers about each other's public addresses. Subsequently, the communication happens directly without the intervention of the rendezvous server.
Now we have to apply some ingenuity and introduce appropriate headers in the packets to inform the peer whether it is sending a reply meant for the P2P client or whether it is sending a request meant for the P2P server. Once we are able to differentiate between the two, we are set. We also need to differentiate between hole punching traffic and regular traffic, because hole punching traffic needs to be bounced, and regular traffic needs to be processed.
Of course, if we stop sending and receiving, the association at the NAT device at both ends will expire. So we either can send keepalive traffic or rerun the hole punching technique. You can choose whichever technique is suitable depending upon your needs.
This technique will not work if both the P2P nodes are behind the same NAT device. So, we also have to figure out whether we can communicate directly using the private IP address itself. Thus, our hole punching has to try the private interface along with the peer's public interface. And, it can happen that our private network has the same private IP as the peer's private IP. So we have to guard against getting spurious responses.
It also can happen that another P2P node in the same private network as ours has the same private IP as the P2P node we want to talk to in another private network. Then we have to do additional validation against the peer's identity to make sure we really are talking to the interested node.
In the unlikely case that you run into brain-damaged NAT devices at both ends, this technique obviously will fail, because we should be able to predict the public address assigned to us. In that situation, the only way is to make the rendezvous server act as a relay for the traffic. So peer-to-peer traffic goes through, but it is no longer peer to peer with the rendezvous machine acting as server. If you run into such situations, you need to think of implementing that as well.
Practical books for the most technical people on the planet. Newly available books include:
- Agile Product Development by Ted Schmidt
- Improve Business Processes with an Enterprise Job Scheduler by Mike Diehl
- Finding Your Way: Mapping Your Network to Improve Manageability by Bill Childers
- DIY Commerce Site by Reven Lerner
Plus many more.
- Download "Linux Management with Red Hat Satellite: Measuring Business Impact and ROI"
- Unikernels, Docker, and Why You Should Care
- Happy GPL Birthday VLC!
- Handheld Emulation: Achievement Unlocked!
- Giving Silos Their Due
- Controversy at the Linux Foundation
- Don't Burn Your Android Yet
- Wine 1.8 Released
- Firefox OS