Mastering ATA over Ethernet

At one point in time, when you wanted to attach an external block storage device to a server, you mapped it as a Logical Unit (LU) across a Storage Area Network (SAN). In the early days, you would do this over the Fibre Channel (FC) protocol. More recently, iSCSI (SCSI over IP) has usurped FC in most data centers. Although, they're stable, feature-rich and fully functional, these protocols are built on top of multiple layers and are extremely complex. It requires a certain level of expertise to truly master these technologies. That is why the Brantley Coile Company wrote the ATA over Ethernet (AoE) specification. The standard has been published for a bit longer than a decade. It was Brantley Coile himself who used this technology as the base framework for his then new startup company, Coraid (later rebranded to the Brantley Coile Company). Since then, that same framework has been open-sourced under the General Public License version 2 (GPLv2) and made available to the general public.

What makes AoE attractive is mainly its simplicity. It was written to run in the Data Link Layer (Layer 2) of the networking OSI model. This means that it's not impacted by any of the Internet Protocol (IP) overhead (that is, the Network Layer or Layer 3). Translation: block devices exported via AoE cannot be accessed by IP. Without this additional overhead, network performance does improve when accessing the exported block device(s). This non-routability also adds to the security of the technology. In order to access the volumes, you need to be physically plugged in to the Ethernet switch hosting them.

As for how data is transferred across the line, AoE encapsulates traditional ATA commands inside Ethernet frames and sends them across an Ethernet network as opposed to a SATA or 40-pin ribbon cable.

For the following example, you are required to have at least two computing nodes running any Linux distribution. One of these nodes will export the block devices. This node will be referred to as the Target. The second node will import the block devices and will be referred to as the Initiator.

Configuring the Target

Most modern Linux distributions will provide binary packages to the entire AoE suite in their local repos, but if you prefer to install from source, you can visit the OpenAoE project's new home on GitHub.

To configure the Target, you must install the AoE server dæmon: vblade. On a distribution like Debian or Ubuntu, run the following command:

$ sudo aptitude install vblade

The next step is to identify a block device to export and the Ethernet port to export through. The vblade application service is very flexible and can work with traditional block devices (such as sdb, sdc and so on), loopback devices (loop0, loop1, etc.), and even files on top of a traditional filesystem, the latter of which appears like any other block device to the Initiator.

For instance, if you want to export /dev/sdb through Ethernet port enps0s3, you would run the following command:

$ sudo vblade 1 1 enps0s3 /dev/sdb

Note: the numeric values following the vblade execution binary define the block device's major and minor numbers. This is sort of the equivalent of a SCSI Logical Unit Number (LUN). Each device needs to have a unique major/minor combination to function properly on the Initiator.

Now, create a generic empty file and export it:

$ dd if=/dev/zero of=aoe.block0 bs=1M count=128
$ sudo vblade 1 2 enps0s3 aoe.block0

Note: the vblade binary does not run as a background dæmon. In runs in the foreground. If you interrupt its execution, it will stop exporting the block devices. In order to avoid this, you can either background the service with the traditional ampersand (&) or just run the dæmonized version of the binary (with the same arguments): vbladed.

Configuring the Initiator

Install the AoE initiator utilities. On Debian/Ubuntu run the following command:

$ sudo aptitude install aoetools

Load the kernel module:

$ sudo /sbin/modprobe aoe


Petros Koutoupis is a software developer at IBM for its Cloud Object Storage division (formerly Cleversafe). He is also the creator and maintainer of the RapidDisk Project. Petros has worked in the data storage industry for more than a decade.