Home › Blogs › Bill Childers's blog ›

Presenting squid-deb-proxy! Speed up your update downloads!

Jun 15, 2010  By Bill Childers
 in
Squid-deb-proxy won't eat your submarine.

Are you like me and have multiple Ubuntu machines under one roof? Are you tired of downloading the same update multiple times? Sick of what seems to be duplicate work? Let me introduce you to my little friend... squid-deb-proxy.

Squid-deb-proxy is a new package for Ubuntu 10.04, and it's designed to make your life easier and allow faster updates if you manage more than one Ubuntu machine. Conceptually, squid-deb-proxy consists of two pieces, a client and a server. The server package is "squid-deb-proxy" and the client package is "squid-deb-proxy-client". The "squid-deb-proxy" server package is basically a squid caching server, with an out-of-the box configuration that allows it to cache .deb packages and make them accessible to the local area network. The "squid-deb-proxy-client" package is basically an include file to your standard apt configuration that makes apt aware of the squid-deb-proxy.

To install the server, simply "sudo apt-get install squid-deb-proxy avahi-tools" on the machine that you wish the server to be on. This will install the squid caching server and the avahi (Bonjour) auto-configuration network utilities, and start both servers, so your new caching squid proxy will start broadcasting its availability on your network. Then, a "sudo apt-get install squid-deb-proxy-client" on each Ubuntu 10.04 machine (including the squid-deb-proxy server) will install the apt configuration. You'll want to install the client on the server as well, so whenever the server downloads updates those updates get cached by the squid proxy. This will also allow the server to install already-fetched updates via the proxy.

Once this is done, squid-deb-proxy is transparent to the user. Each machine's apt program will look on the network for a squid-deb-proxy, and if it finds one, it'll pass its requests through that. The proxy will cache any .deb packages that come through it, and make them available for the next update client that needs them. The second client to request these same updates will pull them down from the squid proxy, rather than having to get them from the Internet. You get the benefit of a local repository without the hassle of setting one up!

The beautiful part about the squid-deb-proxy solution is that it is completely transparent. If you have the squid-deb-proxy client installed on your laptop and you choose to download an update while on a business trip, your laptop will grab the updates from the main repository in your sources.list file, since the proxy isn't on that local area network, broadcasting its services via avahi. There's no need to modify your sources.list in any way, because apt becomes proxy-aware automagically. It's really cool stuff.

______________________

Bill Childers is the Virtual Editor for Linux Journal. No one really knows what that means.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Does it work?

rEnr3n's picture
Submitted by rEnr3n (not verified) on Thu, 07/22/2010 - 08:25.

I installed this a couple of weeks ago. But I noticed that it doesn't work. I tried updating the server first before the client. But when I did, the client didn't retrieve the packages from my server. It still retrieves the packages on the internet. I followed the steps but I think I'm still missing something; or the transparency just don't work for me. How do I know that it is working? What should I look for? Or should I try another alternative?

Try AptOnCD

Graeme's picture
Submitted by Graeme (not verified) on Wed, 06/23/2010 - 13:22.

I have been using AptOnCD (install via: apt-get install aptoncd) to distribute all *.deb updates to many other systems. So I only every download updates from one PC. It's simple to use and works like a charm - you can even add your own *.deb files that didn't come via the updates channel.

Answers

Charles's picture
Submitted by Charles (not verified) on Thu, 06/17/2010 - 11:10.

@Jesseca:
Squid is a popular caching http proxy server. It is often used for security and performance reasons on networks for web browsing. The performance principle of a caching proxy is similar to any type of cache: Machine A requests a file (e.g. a web page or package); the proxy fetches it, then passes it back to A. Later, machine B then requests the same file. The proxy now already has the file, so it doesn't have to go over the Internet to fetch it, but can instead return it immediately to B.

@sshvetsov
I use apt-cacher-ng, and like it. The only real advantage I see to this is that while on the "business trip", you will have to temporarily comment out the proxy setting, or apt won't work (or will be really slow if you're vpn'd to your home network!)

@Funtime:
I'm 99.5%+ sure that it won't upload a file back to squid -- I don't think squid even has the capability to accept such an upload.

avahi tools

Gilbert Dion's picture
Submitted by Gilbert Dion (not verified) on Wed, 06/16/2010 - 22:48.

apt get can't find avahi-tools. How come and what's to do?

I think you mean avahi-utils?

Tal's picture
Submitted by Tal (not verified) on Wed, 06/16/2010 - 20:40.

I think you mean avahi-utils?

avahi-tools!

Gilbert Dion's picture
Submitted by Gilbert Dion (not verified) on Thu, 06/17/2010 - 21:27.

I pasted it form the article:
"sudo apt-get install squid-deb-proxy avahi-tools"

apt-get can't find avahi-tools.

Um, why don't we just merge

MLC's picture
Submitted by MLC (not verified) on Tue, 06/15/2010 - 22:49.

Um, why don't we just merge some p2p features into the updates sometime? I'm sure that'd speed things up to a ridiculous extent. XD

Sounds a lot like apt-cacher-ng

sshvetsov's picture
Submitted by sshvetsov (not verified) on Tue, 06/15/2010 - 13:41.

...only instead of installing squid-deb-proxy-client on every client machine and avahi on the server, you have to insert string

Acquire::http::Proxy "http://apt-cacher-ng-host:3142/";

into /etc/apt/apt.conf. Otherwise works exactly the same, no need to change sources.list files and also available on Ubuntu starting version 8.04

Backwards Transparency?

Funtime's picture
Submitted by Funtime (not verified) on Tue, 06/15/2010 - 13:17.

If I am on travel with one of my client systems and I download an update, when I connect it back to my home network, will it advise the squid-deb-proxy server that it has an updated package which is not in the squid-deb-proxy server cache? And will the server then retrieve the package from the client? Or would the update need to be run on the server as well in order to retrieve new packages?

Webcast
More Resources
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers

Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.

Learn More

Sponsored by AMD

White Paper
More Resources
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState