The IP Security Protocol, Part 2

Encapsulating security payloads, key exchange mechanisms and other components of establishing secure data transfers.

In Part 1 of this article, we talked about security issues related to sending and receiving IP packets and the function of IP Security, or IPSec. We also discussed different levels of security and security protocols. In Part 2, we move on to encapsulating security payloads and key exchange mechanisms.

Encapsulating Security Payloads

IPSec ESP format, specified in RFC 2406, provides confidentiality, authenticity and integrity. The original packet is transparently encrypted by the IPSec layer before being sent and decrypted on the receiving side. An eavesdropper capturing packets in any intermediate node will not be able to recover the original contents of the packet (or, at least, he should not be able to do it in a reasonable amount of time).

Encryption algorithms usually employed for ESP include DES (mandatory for any IPSec implementation), IDEA, Blowfish and RC4. Other encryption algorithms may be adopted as well, maintaining the basic IPSec infrastructure and the ESP packet format.

The ESP header includes the same common IPSec fields found in the AH header (SPI and sequence number), which are used to associate the packet with the proper "IPSec session". Encrypted payload follows the header and is followed by the ESP trailer, which includes an optional variable length padding that helps conceal the original data length and a Next Header field, which is used to determine the upper layer protocol.

The ESP trailer also may be followed by an integrity check value, as in the AH header, in case authentication besides encryption is required. This is usually the case since using encryption without authentication is quite dangerous (see Resources). Encryption protects your data from being eavesdropped, but it does nothing to prevent a malicious user from passing himself off as a legitimate peer and send you fake data. In this case, IPSec encryption even could have a negative effect, as it might cause a false sense of security.

There is a slight difference between the authentication provided by AH and the one provided by ESP. The former also applies to the IP header data, most notably the source IP address, whereas the latter only includes the packet payload.

IPSec Modes

IPSec, both with AH and ESP, can be configured to work in two very different ways: transport mode and tunnel mode. With transport mode, AH or ESP is applied only to the packet payload, while the original IP header remains untouched. The AH or ESP header is inserted after the IP header and before any layer 4 header. In tunnel mode, AH or ESP is applied to the entire original IP packet, which is then encapsulated into a completely new IP packet with a different IP header.

Let's see what this means from a practical point of view. In transport mode the packet maintains its original IP header, thus allowing intermediate routers to recognize it properly (think of quality of service provisioning, for example). On the other hand, an eavesdropper can see the flow of data between the two communicating hosts, and depending on the context, he still can extract a great deal of information from this knowledge. The eavesdropper would not be able to understand the packets' contents, hence the privacy of the communication would be preserved.

Tunnel mode actually provides IP-over-IP transport functionality. In other words, one IP packet is piggybacked on another one, potentially going towards a host that is different from the final destination. In general, using an "outer" IP layer to deliver packets belonging to an "inner" IP layer makes it possible to have two separate IP networks with different topology and addressing schemes, superimposed one over the other.

When in tunnel mode, IPSec offers complete protection from any kind of analysis, because an attacker would be able to learn only the IPSec tunnel endpoints (which are written inside the "outer" IP header), without being able to determine the actual communicating hosts (whose IP addresses are written inside the encrypted "inner" IP header). The drawback is the impossibility for intermediate routers to apply special packet handling techniques to manage the protected traffic, since they too are not able to recognize packet flows properly.

Figure 2


Figure 3

show the difference between an ESP packet in the two modes.

When in transport mode, IPSec must run on each of the communicating peers. In fact, the two must be able to correctly handle the additional headers and manage the IPSec fields. Tunnel mode instead may be used between two routers, acting as the ends of a protected tunnel, which would transparently apply AH or ESP to the packets flowing between them. In this case, the communicating peers would be unaware of the presence of IPSec.

Different networks can be connected by using security gateways running IPSec in tunnel mode, without having to install and configure IPSec software on each host in the LAN. For this reason, tunneling is currently the most widespread IPSec mode, because it allows for the seamless deployment and integration of secure communication services. Needless to say, this is a key component in the building of Layer 3 VPNs.



Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Securing my LAN.

Lord Loh.'s picture

I want to secure my LAN from internal attacks. I primarily want to prevent an external notebook PC from being brought in by an employee and be configured to work on the LAN.

I have certain IP based rules for access and deny of resources. If an external device joins my LAN it could assume any IP and gain access to the otherwise denied resource.

One way I see is to put passphrase on every PC. The other is the use of IKE on the LAN. Can a unauthorized machine be prevented from getting a key via IKE?

Re: The IP Security Protocol, Part 2

Anonymous's picture

As a reference performance note, FreeS/Wan (Linux IPSec implementation) plus AES patches
can almost saturate (95%) a 100Mbps link with a P4 1GHz.