Speed up your downloads with Axel

"Nothing travels faster than the speed of light, with the possible exception of
bad news, which obeys its own set of laws" ~ Douglas Adams


Eh, what can I say? I enjoy a good quote, especially when it involves the topics that I discuss in my blog posts.  Have you ever had those moments where you just need a file from a website and you need it now?  Maybe it's the latest patch to some software, or a new version of your favorite distro that doesn't offer bit-torrent downloads.  Nothing is worse than that moment when your heart sinks as you watch your download rate drop to unbearable numbers.  When you realize that it would probably be quicker to write out a bunch of 1s and 0s than it would to finally get the file you're waiting on.

Well, I know I've had this problem many times before.  But about 6 months ago while sitting in #linuxjournal's IRC channel, I was expressing my *ahem* frustrations about my internet provider's download speed while fetching the latest VirtualBox download.  Just about the point where I was going to give up, nerdboy4200 (Josh) recommended the Axel download accelerator application. As I had never heard of Axel, and a tad suspicious of something that claimed to be a 'multithread' download accelerator, I decided to bite the bullet and give it a
try.  I'll tell you what, boy was I surprised!  Now I'm not one to just put my faith in 'oh, that was a quick download' (If you followed my personal blog post on iperf, I'm a stickler for hard data) so I did some comparisons and number crunching using various websites at different points in the day.  Needless to say, I'm still impressed.  So thanks to nerdboy4200 a blog post was born on the Axel download accelerator.

So just what is Axel? The developers call it a download accelerator but if you want to get technical, it can pull in multiple http or ftp streams into a single download location.  In a nutshell, it's wget hopped up on espresso. Wget is a single stream download in that you point it to a single download location and it fetches the file as requested.  With Axel, you can tell the
application to point to multiple ftp mirrors, or a single http download location, but pull multiple streams at the same time, thus increasing your download speed.

Why would you use Axel? Well as I outlined earlier, it's the perfect utility for those connections to wherever you are downloading from that may have a limit on ftp stream speeds, http sessions that burst and slow down, or servers with a slow connection.  As you will see later on in this post, the hard data was actually quite unbelievable, and I was truly happy with the results.

Now of course, for every positive there comes a possible negative.  If you're using a multistream download utility on a network keep in mind that you will be opening x times as many pipes at up to x times the download rate that you would usually get.  For instance, if you are downloading a single stream at 800 Kb/s, and you want to download 2 streams, now you are sucking down the bandwidth on your network connection for 2 simultaneous streams running 800 Kb/s or greater. Keep this in mind if you are on someone else's network as a guest (or employee) as this may cut into their network throughput.  I somehow doubt your significant other will take kindly to buffering the latest episode of their TV show online while you eat up bandwidth from kernel.org.  However, Axel does allow you to set the number of streams and maximum amount of bandwidth to the download stream, which I will cover here shortly.

The other thing to watch out for is that of common courtesy.  If you're downloading from an ftp site where a lot of other people will be downloading from (say kernel.org), the best thing you can do is utilize Axel's ability to multistream from multiple mirrors instead of a single ftp location.  This way you're not using up the bandwidth from a single mirror, but spreading the bandwidth out among multiple ftp mirrors.

Axel Development and Download

Before you go commenting about how Axel is dead, I'll give you that one. Active development on Axel seems to have come to a standstill in 2009. But the beauty of Open Source software means that there's always someone willing to see the diamond in the rough by forking such an awesome project.  And someone has done such a thing with Axel.  Another thing about Axel is that Debian based distros should have Axel in their repo already, so it should be as simple
as:

apt-get install axel

For the rest of us, there's still .tar.gz packages of which I will get into here shortly.

Axel's main website can be found here:
http://axel.alioth.debian.org/

Their Freshmeat.net page can be found here:
http://freshmeat.net/projects/axel/

And the fork for Axel (via GitHub) can be found here:
https://github.com/emiraga/axel

The person that has been updating the Axel codebase via GitHub has done a pretty good job of keeping up-to-date with his releases.  As of right now the latest release is Jan of 2011.  I personally haven't had any problems with the .tar.gz or the Debian package of Axel as of yet.  But if you want to put on your big britches and play with GitHub code, you're more than welcome to it.

First things first, time to get Axel installed.  If you're on a Debian distro (be it Debian, Ubuntu, or anything that uses Debian codebase and apt-get package manager) you should be able to just run:

apt-get install axel

If you're using an RPM package manager (such as Fedora 15) check yum as it should be available to you via: 

yum install axel
As I test using Debian, Fedora, and some custom distros I tend to install packages from source (tar.gz) and go from there.  So for the rest of you, cd to your favorite download directory and let's get to work.

cd ~/Download wget
http://alioth.debian.org/frs/download.php/3015/axel-2.4.tar.gz tar zxvf
axel-2.4.tar.gz && cd axel-2.4 ./configure && make sudo make install


Tada! Now you should be able to type: axel from the commandline. According to Axel's website, it doesn't have any dependencies (granted, you're going to need GCC and Makeutils to actually compile and link the package) such as external libraries.  (So in a perfect world typing Axel should get you the syntax for running the utility).

Command-Line Switches

There really isn't much to Axel when it comes to command-line switches.  If you want more information on the switches you can: man axel, or type axel --help from the command-line to get information on what each switch does. I'll cover the basics here, and also what I have found useful over the short time that I have been using Axel.

-s x    maximum speed (BPS) - Set the maximum speed allowed to download in Bytes per Second, useful when your on a limited network connection, or a guest on someone else's.

-n x    maximum number of connections - Set the maximum number of connections, I tend to max this out around 10 just to be on the safe side

-o f    Output file    - useful if you want to change thisis-my-we-ird.23.2.43.tar.gz file into thisismyweird-232-43.tar.gz

-v    Verbose        - Self explanatory, the more vs you put in here, the more verbose it gets.

-a    alternate indicator    - changes the download indicator I personally enjoy the look of -a over the default. 

Examples:

# Downloads patch with max number of connections set to 10
axel -n 10 http://www.kernel.org/pub/linux/kernel/v2.6/patch-2.6.39.gz

# Downloads patch at 204,800 Bytes/second (200KB/s) and Max 10 connections
axel -s 204800 -n 10 http://www.kernel.org/pub/linux/kernel/v2.6/patch-2.6.39.gz

# Downloads patch from 4 ftp mirrors
axel ftp://ftp.{us,uk,br,fr}.kernel.org/pub/linux/kernel/v2.6/patch-2.6.39.gz

Real life testing and results

No experiment is complete without blowing things up!  Aw, that's what I get for a Mythbusters marathon weekend.  But, alas, no blowing things up with Axel. In this case no experiment is complete without some hard results.  I'm a sucker for fast results, but I also like to see hard data.  For testing purposes I did the following:

1. I downloaded from 2 websites: virtualbox.org and kernel.org

2.virtualbox.org was through http, kernel.org was http and ftp

3. I downloaded a file 5x with wget, 5x with axel -n 10, and 5x with ftp mirrors

4. I ran these tests in the morning, afternoon and evening. 

5. Once the results were in, I took an average from the 5 results for morning, afternoon, and evening and came up with my numbers.

This way, someone can't come back and say 'Well, your cable connection is faster during xx time' or 'maybe you just got lucky and downloaded it when no one else was downloading it'.  I tried to take every possible variable I could and plug it in.  No I didn't stand on my head, drink coffee through a straw and push 'Enter' with my tongue.  Somehow I don't think that would have changed the results.

WGET Test Results:

virtualbox.org: 1.49 Mb/s
kernel.org HTTP: 1.30 Mb/s
kernel.org FTP: 281.5 Kb/s

AXEL Test Results:

virtualbox.org -n 10: 2.57 Mb/s
kernel.org http -n 10: 2.24 Mb/s
kernel.org 4 ftp mirrors: 560.80 Kb/s

Axel comparison:
virtualbox.org: Increase of 1.08 Mb/s = 72% Increase
kernel.org http: Increase of 962.56 Kb/s = 72% Increase
kernel.org FTP: Increase of 279.30 Kb/s = 99% Increase


All I can say is wow. I honestly didn't think I was going to get results like that when nerdboy4200 (Josh) recommended Axel that day 6 months ago. I was seriously impressed. However, I did notice one thing: if your internet speed maxes out at a certain speed (say 2Mb/s) odds are, you're not going to get 4Mb/s on an Axel download, so don't expect miracles.  I did some further testing on other internet connections and noticed that there were times where the results of wget matched that of Axel, only because the speed of the provider I was pulling from matched the max download speed I was allowed to pull on that internet connection.  In that case, both wget and axels results were similar.

Conclusion

It's just amazing when someone recommends an Open Source product through word of mouth, and it becomes something that you end up using on a daily basis. That's just what makes Open Source so powerful, the ability for people to find a product that changes their sys admin life, and pass that knowledge onto the next person they see struggling.  In this case Axel really takes the cake.

Now if you want to 'accelerate' (pun intended) Axel, there are gui utilities out there (axel-kapt comes to mind from debian repos) and a Google search for 'axel debian apt-get' will turn up some interesting results where people have been able to integrate Axel into their apt-get downloads for an increase of package downloading performance.

So the next time you're in #linuxjournal, feel free to drop a line to nerdboy4200 for recommending such an awesome utility.  Without his experimentation with Axel, I'm sure I would still be downloading the latest update to virtualbox today.

 

______________________

www.jaysonbroughton.com

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Bittorent problems

Freda Payne's picture

Well, I have been trying to set up bittorrent for a week now. I read posts and everything and still no luck. I have a WRT54G linksys router and my cable service provider is optimum online. I tried running bittorndao and bittorrent but i would get the yellow circle suggesting i have problems with my ports. I was told to try Argan Oil and that helped a little. My speeds would never top 5kb and it would go very slow. I check on that grc site and I checked if my ports were open. My ports were in stealth. I proceded into port fowarding from portfoward.com. I port fowarded and still the ports were in stealth. I cannot set up a static ip for my router i think because my internet service provider does not offer static ip hosting. I have tried everything.

Try ARIA2

Allison Oil's picture

ARIA2 is great too like the person above me has mentioned. I'd definitely say it's better than the old, jojoba oil coated AXEL!

aria2

Indra's picture

I've been using aria2 during the last 3 years. aria2 more stable than the axel, so please review it.

Indra, Good idea, I'll add

Jayson Broughton's picture

Indra,
Good idea, I'll add that to my list and probably do a comparison between the two products (axel vs. aria2 shootout) in the future.

Error while using

Irfan's picture

Hi, I tried Axel . but i am getting "Too many redirects" error. Pls help

Aria2

Anonymous's picture

You should also try ARIA2 [aria2c after installing].
It is a great piece of software, probobly better than Axel.

Best regards!

Thanks

Bennington's picture

You are a good writer… I am just learning english maybe you could proof read my article for me so I can be good as you.

Source of your problem.

borring's picture

Note that if a multithreaded downloader like Axel is helping you dramatically increase your download speed, the bottle neck is not likely to be your ISP, but the server you're downloading from.

If this is the case, there may be a chance that the server you're downloading from is being hammered at the moment... especially if it's a new download available..

example: Every 6 months, when a new Ubuntu ISO is released, the http download slows to a crawl. Using a program to initiate multiple TCP connections to speed up your own download would benefit only one person while causing grief to everyone else. That's why bittorrent, being so scalable, is often encouraged.

I wish online software (or anything digital) distributors would come to their senses and start using bittorrent to distribute.

I agree on the bittorrent

Jayson Broughton's picture

I agree on the bittorrent side of things. Granted some of the major linux Distro's have started to go with torrent downloads, which have helped out a lot. But at the same time, some businesses have attempted to block torrent downloads because they feel that torrents=evil when their IT Administrators and/or mgmnt don't fully grasp the full understanding how Torrents can help retrieve files at a much faster rate. I've downloaded full ISO images of a linux distro via torrent in less than an hour, when a regular download would take 4-6 hours. I agree, more education would be beneficial to distributors of distributions. I see another post coming ;-)

You should give aria2 and

Kai Lüke's picture

You should give aria2 and proz (prozilla) a try, also! They can continue the download.
Ever heard of apt-proz? I like it more than apt-fast.
Greetings

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix