Usenix Tutorials Answer Deep Questions

As a Linux administrator, you have a lot of choices about where to go for training. For a good mixture of deep and practical knowledge that doesn't duplicate what you can get from books, man pages and experimentation, it's hard to beat the tutorial section of the Usenix Technical Conference.

I attended three tutorials: "Inside the Linux Kernel", conducted by Ted Ts'o and intended for people considering getting into kernel development; "Topics in UNIX and Linux Administration", a selection of new items system administrators should be concerned about and conducted by gurus Trent Hein, Ned McClain and Evi Nemeth; and "Exploring the Potential of LDAP", taught by Samba contributor Gerald Carter.

Ts'o said the kernel tutorial was "enough background material that you'll be able to dive into the sources", and he did give a useful road map for people exploring the kernel without slowing down and going over actual source code. He made the useful assumption that if he gives you a map of how things work, you'll be able to figure out the details when you get there.

He did cover some items of historic interest from the 2.2 and previous kernels, but mainly he covered 2.4 with an occasional peek at or speculation about what's happening in 2.5. Instead of simply presenting how things work, Ts'o gave the audience many of the performance considerations behind the design decisions. For example, he covered the older interrupt bottom halves and the new (with 2.4) tasklets in the context of the free performance testing that Mindcraft thoughtfully gave the 2.2.6 kernel in 1999. Only one bottom half can be run on any CPU, while different CPUs can run different tasklets.

How effective was the tasklet innovation? The fact that Mindcraft's home page still links to the 2.2.6 tests should tell you something. Performance wars aside, a long-time kernel contributor's view of this anecdote helps wannabe kernel hackers see what's important to the kernel team.

Other considerations when writing kernel code include the fact that every process has its own 8Kb kernel stack, which is not enough to do serious recursion or big arrays on the stack. Also, interrupts can borrow space on the kernel stack of the currently running process.

Finally, although the session was mainly an overview for kernel developers, Ts'o also laid out some good tips for people using an unmodified kernel. For example, the old advice about having a swap partition equal to at least the size of physical RAM is now obsolete. 2.4 and later kernels will discard, without hitting disk, any page already present on the swap partition, so it is better to have a swap partition twice the size of physical RAM to increase the likelihood of this happening.

Adding more RAM still is your best performance bargain. In the "advantages of working for IBM" department, Ts'o's laptop is equipped with 512MB, and he mentioned that the IBM ThinkPad T23 will accept up to 1GB. A final performance tip is that when making a new ext2 or ext3 filesystem, you ordinarily should make the block size the same as the kernel page size.

All in all, this was a worthwhile tutorial for people starting kernel development or doing serious performance tuning and interested in developing an accurate mental model of how things work.

In "Topics in UNIX and Linux Administration, Part I", Trent Hein, Ned McClain and Evi Nemeth, three of the coauthors of Linux Administration Handbook, covered some new topics that aren't in any book yet, but that professional system administrators will need to learn about.

Most relevant in today's world of bigger and bigger Linux servers, with bigger and bigger responsibilities, was the Linux Logical Volume Manager section. With LVM, you can grow and shrink filesystems without repartitioning the disk and, in the case of ext3 or ReiserFS, without even remounting the filesystem. Another advantage of LVM, perhaps more relevant for day-to-day-use, is the ability to snapshot a filesystem--to create quickly what looks like a complete copy, frozen in time.

The snapshot doesn't actually contain a whole copy of all the data from the original, of course. It only stores unchanged versions of any blocks that get changed in the original. At the time you create the snapshot, you specify how much space to allow. As you might guess, in most cases you should take the snapshot, dump it to tape and then remove it.

While the LVM portion of the tutorial was Linux-specific, the rest was relevant to almost any OS. BIND 9 is a total rewrite and includes some features you can start using now and others for which some important decisions still need to be made. You can start using TSIG/TKEY to authenticate zone transfers among your name servers now, because you create and manage the necessary keys, Nemeth said. How the TLDs, such as .com, will handle key management for the more complex DNSSEC is still being worked out.

No section on security would be complete without covering the human element. In the matter of making users choose good passwords, it's better to educate them than simply to reject easy-to-guess passwords. Make sure that your passwd utility gives a reason, such as "too short" or "based on a dictionary word", that helps the user pick a good password.

Educating users about security really helps. "You need to get your users to buy in a little", Nemeth said, and educating them about security helps this happen. Also, security measures can end up reducing security when users work around them. "I hate password aging programs", Nemeth said. They make users rotate among two or three passwords--however few they can get away with--and keep the password aging tool happy.

Among the nastiest of the system administrator's legal problems is discovering unauthorized copies of proprietary software. In many cases, your immediate manager may be among the offenders and might tell you to leave it alone. But this is not an option. "Sometimes you have to go further up the management chain and make your complaint in writing. Sometimes you have to quit", Nemeth said.

Hiring and firing is another difficult problem. Nemeth recommends that you give a questionnaire to job applicants and add a few bogus technologies to measure the applicant's "bullshit level". If someone says he or she can "wire up an MX connector" without supervision, be wary. When checking references, remember that some of them won't be willing to say explicitly anything negative. "Listen really carefully to the previous boss", she said.

In a large company or the public sector, it may be significantly easier to let someone go in the first few months of employment, so don't delay. In any case, "You'll lose good people if you don't deal with the folks who are not doing their job."

One final piece of common sense is that when you have several administrators working on the same group of machines, keep a "diary" for each one that covers hardware and software changes and any unusual events. And label everything.