Speed up your downloads with Axel
"Nothing travels faster than the speed of light, with the possible exception of
bad news, which obeys its own set of laws" ~ Douglas Adams
Eh, what can I say? I enjoy a good quote, especially when it involves the topics that I discuss in my blog posts. Have you ever had those moments where you just need a file from a website and you need it now? Maybe it's the latest patch to some software, or a new version of your favorite distro that doesn't offer bit-torrent downloads. Nothing is worse than that moment when your heart sinks as you watch your download rate drop to unbearable numbers. When you realize that it would probably be quicker to write out a bunch of 1s and 0s than it would to finally get the file you're waiting on.
Well, I know I've had this problem many times before. But about 6 months ago while sitting in #linuxjournal's IRC channel, I was expressing my *ahem* frustrations about my internet provider's download speed while fetching the latest VirtualBox download. Just about the point where I was going to give up, nerdboy4200 (Josh) recommended the Axel download accelerator application. As I had never heard of Axel, and a tad suspicious of something that claimed to be a 'multithread' download accelerator, I decided to bite the bullet and give it a
try. I'll tell you what, boy was I surprised! Now I'm not one to just put my faith in 'oh, that was a quick download' (If you followed my personal blog post on iperf, I'm a stickler for hard data) so I did some comparisons and number crunching using various websites at different points in the day. Needless to say, I'm still impressed. So thanks to nerdboy4200 a blog post was born on the Axel download accelerator.
So just what is Axel? The developers call it a download accelerator but if you want to get technical, it can pull in multiple http or ftp streams into a single download location. In a nutshell, it's wget hopped up on espresso. Wget is a single stream download in that you point it to a single download location and it fetches the file as requested. With Axel, you can tell the
application to point to multiple ftp mirrors, or a single http download location, but pull multiple streams at the same time, thus increasing your download speed.
Why would you use Axel? Well as I outlined earlier, it's the perfect utility for those connections to wherever you are downloading from that may have a limit on ftp stream speeds, http sessions that burst and slow down, or servers with a slow connection. As you will see later on in this post, the hard data was actually quite unbelievable, and I was truly happy with the results.
Now of course, for every positive there comes a possible negative. If you're using a multistream download utility on a network keep in mind that you will be opening x times as many pipes at up to x times the download rate that you would usually get. For instance, if you are downloading a single stream at 800 Kb/s, and you want to download 2 streams, now you are sucking down the bandwidth on your network connection for 2 simultaneous streams running 800 Kb/s or greater. Keep this in mind if you are on someone else's network as a guest (or employee) as this may cut into their network throughput. I somehow doubt your significant other will take kindly to buffering the latest episode of their TV show online while you eat up bandwidth from kernel.org. However, Axel does allow you to set the number of streams and maximum amount of bandwidth to the download stream, which I will cover here shortly.
The other thing to watch out for is that of common courtesy. If you're downloading from an ftp site where a lot of other people will be downloading from (say kernel.org), the best thing you can do is utilize Axel's ability to multistream from multiple mirrors instead of a single ftp location. This way you're not using up the bandwidth from a single mirror, but spreading the bandwidth out among multiple ftp mirrors.
Axel Development and Download
Before you go commenting about how Axel is dead, I'll give you that one. Active development on Axel seems to have come to a standstill in 2009. But the beauty of Open Source software means that there's always someone willing to see the diamond in the rough by forking such an awesome project. And someone has done such a thing with Axel. Another thing about Axel is that Debian based distros should have Axel in their repo already, so it should be as simple
apt-get install axel
For the rest of us, there's still .tar.gz packages of which I will get into here shortly.
Axel's main website can be found here:
Their Freshmeat.net page can be found here:
And the fork for Axel (via GitHub) can be found here:
The person that has been updating the Axel codebase via GitHub has done a pretty good job of keeping up-to-date with his releases. As of right now the latest release is Jan of 2011. I personally haven't had any problems with the .tar.gz or the Debian package of Axel as of yet. But if you want to put on your big britches and play with GitHub code, you're more than welcome to it.
First things first, time to get Axel installed. If you're on a Debian distro (be it Debian, Ubuntu, or anything that uses Debian codebase and apt-get package manager) you should be able to just run:
apt-get install axel
If you're using an RPM package manager (such as Fedora 15) check yum as it should be available to you via:
yum install axel
As I test using Debian, Fedora, and some custom distros I tend to install packages from source (tar.gz) and go from there. So for the rest of you, cd to your favorite download directory and let's get to work.
cd ~/Download wget http://alioth.debian.org/frs/download.php/3015/axel-2.4.tar.gz tar zxvf axel-2.4.tar.gz && cd axel-2.4 ./configure && make sudo make install
Tada! Now you should be able to type: axel from the commandline. According to Axel's website, it doesn't have any dependencies (granted, you're going to need GCC and Makeutils to actually compile and link the package) such as external libraries. (So in a perfect world typing Axel should get you the syntax for running the utility).
There really isn't much to Axel when it comes to command-line switches. If you want more information on the switches you can:
man axel, or type
axel --help from the command-line to get information on what each switch does. I'll cover the basics here, and also what I have found useful over the short time that I have been using Axel.
-s x maximum speed (BPS) - Set the maximum speed allowed to download in Bytes per Second, useful when your on a limited network connection, or a guest on someone else's.
-n x maximum number of connections - Set the maximum number of connections, I tend to max this out around 10 just to be on the safe side
-o f Output file - useful if you want to change thisis-my-we-ird.23.2.43.tar.gz file into thisismyweird-232-43.tar.gz
-v Verbose - Self explanatory, the more vs you put in here, the more verbose it gets.
-a alternate indicator - changes the download indicator I personally enjoy the look of -a over the default.
# Downloads patch with max number of connections set to 10
axel -n 10 http://www.kernel.org/pub/linux/kernel/v2.6/patch-2.6.39.gz
# Downloads patch at 204,800 Bytes/second (200KB/s) and Max 10 connections
axel -s 204800 -n 10 http://www.kernel.org/pub/linux/kernel/v2.6/patch-2.6.39.gz
# Downloads patch from 4 ftp mirrors
Real life testing and results
No experiment is complete without blowing things up! Aw, that's what I get for a Mythbusters marathon weekend. But, alas, no blowing things up with Axel. In this case no experiment is complete without some hard results. I'm a sucker for fast results, but I also like to see hard data. For testing purposes I did the following:
1. I downloaded from 2 websites: virtualbox.org and kernel.org
2.virtualbox.org was through http, kernel.org was http and ftp
3. I downloaded a file 5x with wget, 5x with axel -n 10, and 5x with ftp mirrors
4. I ran these tests in the morning, afternoon and evening.
5. Once the results were in, I took an average from the 5 results for morning, afternoon, and evening and came up with my numbers.
This way, someone can't come back and say 'Well, your cable connection is faster during xx time' or 'maybe you just got lucky and downloaded it when no one else was downloading it'. I tried to take every possible variable I could and plug it in. No I didn't stand on my head, drink coffee through a straw and push 'Enter' with my tongue. Somehow I don't think that would have changed the results.
WGET Test Results:
virtualbox.org: 1.49 Mb/s
kernel.org HTTP: 1.30 Mb/s
kernel.org FTP: 281.5 Kb/s
AXEL Test Results:
virtualbox.org -n 10: 2.57 Mb/s
kernel.org http -n 10: 2.24 Mb/s
kernel.org 4 ftp mirrors: 560.80 Kb/s
virtualbox.org: Increase of 1.08 Mb/s = 72% Increase
kernel.org http: Increase of 962.56 Kb/s = 72% Increase
kernel.org FTP: Increase of 279.30 Kb/s = 99% Increase
All I can say is wow. I honestly didn't think I was going to get results like that when nerdboy4200 (Josh) recommended Axel that day 6 months ago. I was seriously impressed. However, I did notice one thing: if your internet speed maxes out at a certain speed (say 2Mb/s) odds are, you're not going to get 4Mb/s on an Axel download, so don't expect miracles. I did some further testing on other internet connections and noticed that there were times where the results of wget matched that of Axel, only because the speed of the provider I was pulling from matched the max download speed I was allowed to pull on that internet connection. In that case, both wget and axels results were similar.
It's just amazing when someone recommends an Open Source product through word of mouth, and it becomes something that you end up using on a daily basis. That's just what makes Open Source so powerful, the ability for people to find a product that changes their sys admin life, and pass that knowledge onto the next person they see struggling. In this case Axel really takes the cake.
Now if you want to 'accelerate' (pun intended) Axel, there are gui utilities out there (axel-kapt comes to mind from debian repos) and a Google search for 'axel debian apt-get' will turn up some interesting results where people have been able to integrate Axel into their apt-get downloads for an increase of package downloading performance.
So the next time you're in #linuxjournal, feel free to drop a line to nerdboy4200 for recommending such an awesome utility. Without his experimentation with Axel, I'm sure I would still be downloading the latest update to virtualbox today.
Practical Task Scheduling Deployment
July 20, 2016 12:00 pm CDT
One of the best things about the UNIX environment (aside from being stable and efficient) is the vast array of software tools available to help you do your job. Traditionally, a UNIX tool does only one thing, but does that one thing very well. For example, grep is very easy to use and can search vast amounts of data quickly. The find tool can find a particular file or files based on all kinds of criteria. It's pretty easy to string these tools together to build even more powerful tools, such as a tool that finds all of the .log files in the /home directory and searches each one for a particular entry. This erector-set mentality allows UNIX system administrators to seem to always have the right tool for the job.
Cron traditionally has been considered another such a tool for job scheduling, but is it enough? This webinar considers that very question. The first part builds on a previous Geek Guide, Beyond Cron, and briefly describes how to know when it might be time to consider upgrading your job scheduling infrastructure. The second part presents an actual planning and implementation framework.
Join Linux Journal's Mike Diehl and Pat Cameron of Help Systems.
Free to Linux Journal readers.Register Now!
- SUSE LLC's SUSE Manager
- My +1 Sword of Productivity
- Murat Yener and Onur Dundar's Expert Android Studio (Wrox)
- Managing Linux Using Puppet
- Non-Linux FOSS: Caffeine!
- SuperTuxKart 0.9.2 Released
- Doing for User Space What We Did for Kernel Space
- Google's SwiftShader Released
- Parsing an RSS News Feed with a Bash Script
- SourceClear Open
With all the industry talk about the benefits of Linux on Power and all the performance advantages offered by its open architecture, you may be considering a move in that direction. If you are thinking about analytics, big data and cloud computing, you would be right to evaluate Power. The idea of using commodity x86 hardware and replacing it every three years is an outdated cost model. It doesn’t consider the total cost of ownership, and it doesn’t consider the advantage of real processing power, high-availability and multithreading like a demon.
This ebook takes a look at some of the practical applications of the Linux on Power platform and ways you might bring all the performance power of this open architecture to bear for your organization. There are no smoke and mirrors here—just hard, cold, empirical evidence provided by independent sources. I also consider some innovative ways Linux on Power will be used in the future.Get the Guide