View from the Trenches: Hitchhiker's Guide to a Bug Fix
October 1st, 2003 by Glenn Stone in
Recently, a big flap has arisen over vulnerabilities in OpenSSH and a new bug in OpenSSL, two packages that are cornerstones on which secure access to Linux is built. Of course, these vulnerabilities concern folks; many of us depend on SSH or SSL sessions for our livelihood. We do remote administration, retrieve e-mail and even do such mundane and ordinary tasks as pay bills. If these services aren't secure, we as Linux people could have a hard time keeping peanut butter and jelly on our dinner tables.
So the first advisory comes out, and it's no big deal. Then comes a followup, and a third one, and somewhere in there people start asking, "Is there a replacement for this?" Well, yes, there are several replacements for OpenSSH, some that are compatible, some that aren't as compatible. Most are free, but some are not.
And then there's the question of how to get the published fix working on one's own machine. Do you grab the fix off the developer's Web site, compile it from source and put it in as a local, unpackaged system? Do you wait for your distribution maker to publish a fix? How long is an acceptable wait? Who publishes first? Somehow, the idea of an easy way to roll one's own package gets lost in the shuffle. Admittedly, it's nowhere near as easy as configure-make-make-install or apt-get this or rpm that, but that's what stealing a good script is for, right? Perhaps not.
A lot of energy is given to the idea that it must be absolutely secure all the time, and any delay is bad news. Never mind the fact that, speaking from personal experience, when your system becomes large enough, it's going to take you anywhere from 48 hours to a week to get the new package in the door, test it with your own system peculiarities, schedule the appropriate downtime window and give your customers ample notice. Only then can you install the fixes on your backup system and fail over to put the new code in play. And, never mind the fact that, unless the black hats find the bug first--and so often these days, the white hats or the vendors themselves find things--there is anywhere from a week to two months' delay between when the bug is found and when exploits begin to be used in the wild.
So, when the OpenSSL bug landed in my lap and the first posted comment I saw was "Is there a package that replaces this?", my interest was piqued. I went to OpenSSL's news page and looked for security advisories. I found the current one (30 September), a small group surrounding the timing attack scheme back in March and April of this year, a buffer overflow bug fix from July/August of 2002. Prior to that, the only thing I saw marked as bug fix or security was posted in April of 2000--three years, three sets of bugs.
Amongst all this bug postings, I saw messages stating all the major vendors had published their own fixes, each in its inimitable fashion and in what I considered to be reasonable amounts of time. (As of this writing not everyone has checked in with OpenSSL, but it's been only a few hours since I first heard about it.) I dutifully went and did the little rain dance each of my different machines requires to update their packages and checked in with my boss to make sure the rest of the machines were receiving their appropriate penance. They all were, and nobody had posted any panicked messages of "I got hosed" or "exploit in the wild, look out". Outside of a groaning inbox, it was another quiet week, another routine security update.
Routine, that's a good word. Every good administrator has a routine: get a new machine, subscribe to the appropriate security mailing list, see a security update, read it, see if you have that package, grab the update, apply it, bounce the service if necessary and if the updater doesn't do so already. For what it's worth, Debian and SuSE usually do; Red Hat, Mandrake and Slackware usually do not. Your mileage may vary. I've had this routine since Red Hat 5, when I first had a Linux box on the live Internet, and I've yet to have a single machine I administer get cracked.
I know some people now are going to reach down, press Reply and say I'm not paranoid enough. But, I am reminded here of the philosophy behind PGP. PGP--or its GNU analog, GPG--is something a lot of us use every day. If you have an RPM-based system, you use it without even knowing about it, as most RPMs have a PGP/GPG signature. PGP, remember, stands for pretty good privacy--not Totally Secure Crypto or any other such absolute, but pretty good privacy. Today, PGP and similar systems, including OpenSSL and OpenSSH, are used worldwide to authenticate and protect various bits of data. Certainly, if somebody really wants your data, they can find a way to get enough CPU cycles to brute force the decryption and have your credit card number for lunch. Do we worry about it? No, not really. It's good enough.
So too, I think, is the effort to keep OpenSSL, OpenSSH and all of Linux updated good enough. Every piece of software is going to have bugs. As long as those bugs are reasonably few and squashed in a reasonable amount of time, and as long as the vendors keep up with things and don't keep us waiting weeks on end (at which point we can always resort to grabbing the maintainer's source), I'm content to let the Linux community keep on doing what it's doing--making and distributing the best operating systems in the world. I encourage you as a fellow Linux enthusiast or professional to do likewise. Don't panic. It's open source; it'll get fixed, quickly and well. When the day is done, the bits still flow and life goes on. My inbox thanks you.
Glenn Stone is a Red Hat Certified Engineer, sysadmin, technical writer, cover model and general Linux flunkie. He has been hand-building computers for fun and profit since 1999, and he is a happy denizen of the Pacific Northwest.
Special Magazine Offer -- 2 Free Trial Issues!
Receive 2 free trial issues of Linux Journal as well as instant online access to current and past issues. There's NO RISK and NO OBLIGATION to buy. CLICK HERE for offer
Linux Journal: delivering readers the advice and inspiration they need to get the most out of their Linux systems since 1994.
Sorry, offer available in the US only. International orders, click here.
Subscribe now!
Recently Popular
| The new business of free radio | Jul-24-08 |
| Linux HOWTO: Video Editing Magic with ffmpeg | Jul-23-08 |
| Why We Must React to ACTA | Jul-24-08 |
| Boot with GRUB | May-01-01 |
| Chapter 16: Ubuntu and Your iPod | Aug-30-06 |
| Building a Call Center with LTSP and Soft Phones | Aug-25-05 |
Featured Videos
Non-linear video editing tools are great, but they're not always the best tool for the job. This is where a powerful tool like ffmpeg becomes useful. This tutorial by Elliot Isaacson covers the basics of transcoding video, as well as more advanced tricks like creating animations, screen captures, and slow motion effects.
Shawn Powers reviews the HP Mini-Note portable computer.
Thanks to our sponsor: Silicon Mechanics
Silicon Mechanics is a leading manufacturer of rackmount servers, storage, and high performance computing hardware. The best warranty offerings available are backed by experts dedicated to customer satisfaction.
From the Magazine
August 2008, #172
There's nuttin like a Cool Project to give you some relief from the summer heat, so get out your parka cuz we got a bunch of em. First up is the BUG, not a bug, The BUG. It's got a GPS, camera and more, in a hand-sized package that's user programmable. The BUG does everything. It's both a floor wax and a dessert topping. Get one now. Need a software version of a Swiss Army knife? Take a look at Billix, and don't leave home without it. Then, chew on this one, an X server on a Gumstix device driving an E-Ink display. Need more storage? How about 16 Terabytes? Can do.
And, of course, we have the usual cast of characters: Marcel, Reuven, Dave, Kyle, Doc, plus the new kid on the block Shawn Powers. But it doesn't stop there: build a MythTV box on a budget, build your own GIS system, set up the tools to monitor your enterprise and more. Finally, remember The War of the Worlds? Now you can play too.
Delicious
Digg
Reddit
Newsvine
Technorati







Re: View from the Trenches: Hitchhiker's Guide to a Bug Fix
On October 2nd, 2003 Anonymous says:
Exactly. Anyone who uses Linux for any length of time learns the importance of a high quality package management system. Debian provides dpkg and apt. Life is good. Once that functionality is in place, patching is no worse than any other update and dependancy management.
My pet peave is spinmeisters who point at security advisories and turn logic on its head by saying that a high patch rate is a BadThing(tm). When I first got into IT someone told me there are two promises that a salesman can make with confidence, "Software has bugs, and hardware breaks". Well, seeing as how bugs are inevitable, then what really counts is the timeliness of discovery, and speed with which a patch comes out. After that, it's just business (package management) as usual.
Not True
On October 2nd, 2003 Anonymous says:
Once Linux get's BSD like randome position independent compiled code (RedHat is doing this I think) then 99% of all exploits will be gone.
And if the distros would just run all their code through Valgrind they'd save themselves a lot of bug fixes later as well.
If those two things happened we'd see OpenBSD level security in Linux.
i sort of disagree
On October 2nd, 2003 florin (not verified) says:
99% of all exploits will be gone
I wish i could agree. Yes, the remote root exploits will be made much more difficult. But it will be a statistical thing: the correct entry points in certain code segments will be randomized. That doesn't mean it will be impossible to exploit a code and get remote root. It just means it will be a lot harder (orders of magnitude).
This will not prevent classic DoS, i mean remotely crashing the service. The remote root will not work, but the attacker may still be able to crash that particular service.
But anyway, i'm not saying it's worthless. It's still a valuable protection, it's just not panacea.
P.S.: Yes, Red Hat appears to be the first distro to deliver this thing. Their next release (scheduled for launch this month) will include this kind of protection. I can barely sit still while waiting for it... :-)