Constructing Red Hat Enterprise Linux v. 3

 in
Putting together a Linux distribution gets a lot more complicated when stacks of requirements start arriving from hardware vendors and other partners.

Countless external requests ask for either proprietary additions or hooks. In the open-source tradition, this is something we consistently have to refuse.

For Enterprise Linux v. 3, the initial set of requested features was about 500 items. Each of these was entered into our feature-tracking tool called Featurezilla, which actually is a derivative of the Bugzilla bug-tracking tool. Figure 1 shows several items in Featurezilla. To show that the jabs go both ways, one partner has a business manager whose name is Paul, name changed to protect the innocent. He manages a list as a text file, but back at his company, the list is referred to as Paulzilla.

Figure 1. The feature-tracking tool Featurezilla lists feature requests and their status.

We had countless tortuous internal meetings to prioritize and slog through the full set of 500 items. From that prioritized list, the engineering managers went off to try to get the list down to a manageable set that is humanly achievable by their team members. Ultimately, the list is negotiated down to something that vastly exceeds anything reasonably achievable by the development team. But, that's the Red Hat way, and it's only the beginning of the story.

Red Hat Enterprise Linux v. 2.1 Maintenance Pulls

To keep things interesting, in addition to having to develop to an aggressive schedule for Red Hat Enterprise Linux v. 3 development, a large amount of effort also is required to support the Enterprise Linux v. 2.1 maintenance stream. By the time Enterprise Linux v. 3 shipped, v. 2.1 had been out for 15 months. Because we provide a five-year life cycle for releases in the Enterprise Linux family, we simply cannot dump Enterprise Linux v. 2.1 support and maintenance work. The requirements demanded by our partners in the maintenance stream form an interesting paradox. When it comes to the objectives of the maintenance stream, it seems that all partners and high-profile customers speak out of both sides of their mouth. “Don't change anything! I'm using this release in production. Don't upset the apple cart”, is followed by “I really need this feature immediately. Hurry up and give it to me, but don't put anything else in.”

Ultimately, it all boils down to a careful case-by-case risk/benefit assessment. Internally, we review features and bug-fix proposals among the engineering ranks. It is well beyond my literary abilities to convey the strong-willed and passionate debates conducted over incorporating features or bug fixes that straddle the risk/benefit fence.

Red Hat Enterprise Linux v. 3 Kernel Development

Some people have the mistaken impression that Red Hat simply pulls together a random collection of bits and pieces from the upstream Open Source community, slaps the Red Hat name on it and calls it a product. Truth be told, Red Hat engineers contribute a substantial percentage of upstream (for example, 2.4 and 2.6) kernel development. The productivity, breadth of knowledge and ability to be on top of a torrent of internal and external communication demonstrated by these kernel developers is stunning. They truly are world class and humbling to be among. But don't tell them that, lest it goes to their heads.

Figure 2. Kernel developers at Red Hat, left to right: Larry Woodman, Dave Anderson and Rik van Riel.

In addition to the sizable upstream enhancements that get pulled back into the Enterprise Linux kernel, a large set of enhancements is developed in-house to meet the product requirements. As an open-source company, all of the kernel enhancements are available to the community at large. The vast majority of these changes do end up being incorporated upstream.

The kernel in Enterprise Linux v. 3 primarily is based on the 2.4.21 kernel, but it has a huge number of features backported from the more recent 2.6 kernel. In recognition that the 2.6 kernel had not yet been stabilized, only key features deemed commercially ready were candidates for the backport into Enterprise Linux v. 3. Here's a little tip on how to really annoy a Red Hat kernel developer: say something like, “So, you guys just ship the stock 2.4.21 kernel; right?”

The most daunting challenges in constructing the Enterprise Linux v. 3 kernel were the requirements that support be provided for seven different architectures and that the kernels for these architectures all must be built from a common source tree. The seven architectures include: x86 (and Athlon), AMD64, Itanium 2, IBM mainframe (both s390 and s390x) and IBM's iSeries and pSeries PPC systems. Although each of these architectures is supported by the upstream kernels in varying degrees, the reality is many of these architectures are developed primarily in isolation. This inevitably leads to integration nightmares.

Another interesting twist to the story is Red Hat's kernel development team literally is spread all over the world. Out of necessity, this has led to virtually all interaction being done on-line. For example, we rarely have team meetings, as time-zone challenges get in the way. Sometimes this on-line communication goes to extremes; it's quite common to use IRC chat sessions to speak with someone sitting at the next desk. Yes, our mothers were right; we are a bunch of geeks.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

This is a comment addressed to RedHat ->
RH9 converted me from Windows because of its polished finish. Thank you.

Some advice: That was a confusing article title as I didn

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Actually, there is a cheap version sold basically at media cost for academic users:

http://www.redhat.com/solutions/industries/education/products/

WHAT about FEDORA?

Anonymous's picture

This article completley negates and ignores the fact that RHEL had RH9 (and in future Fedora Cora) to use as a base for testing as well.

The REAL open source community uses the community version - that's where it's proven. The enterprise hacks put it in a nice box.

Nice try to pull the wool over our eye LJ but we know better we're not WINDOZE users after all...

Re: WHAT about FEDORA?

dmarti's picture

How many Fedora users have 8-way SMP systems and 128 SCSI disks?

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Wow, time travel! The article was posted on 4/1/2004 which is 8 days in the future as I enter this comment. I never knew Enterprise Linux was so powerful. :-)

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Great marketing. A lot of blather for collecting other people's work.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Please look at a recent changelog at kernel.org and count the redhat.com addresses.

http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.4

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Thats because RH doesn't go way out of its way to advertise about all the work we do upstream. For example, here's just a few of the big ticket items that come to mind:

- the whole new NPTL threading model
- rmap VM
- lvm2/device mapper
- O(1) scheduler
- ext3
- SATA subsystem
- ipv6, ipsec
- LSM
- PIE
- lots of network drivers

Plus a ton of misc fixes. This list is by no means comprehensive, as its just kernel stuff.

RH's gcc team (formerly Cygnus) is the heart of gcc.

A substantial portion of the GTK development crew is here.

Substantial contribution among the 1500 rpm packages.

Tally it up yourself. To me that looks like a lot more than just collecting up other people's work.

-Tim Burke

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Nice troll. It succinctly demonstrates your lack of knowledge about software development, and product development.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Hmmm... I must have missed the part where removing support for ISA soundcards was a requested feature... They were supported in 2.1! Or adding a new, broken bindconf that still hasn't been fixed, though it's been out and broken for five months. Or even leaving out many, many *-devel rpm packages, such as sendmail-devel, which enables one to actually add and use milters with the milter-enabled base rpm.

If only the core bits got extensive testing, then that would explain why RHEL3 is worse than RHL9 from a "working-features" perspective.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Thats because the amount of possible work is almost infinite while the staff we have at RH is finite. Ultimately we do have to make some hard tradeoffs. We can't please all of the people all of the time. For example, we pour a lot of effort into new hardware support. Things like AMD64, sata, graphics cards, new storage and networking adapters, etc. So due to new stuff, the pile just gets bigger. Something has to give. This is why sometimes some things have to fall out the other end. ISA sound cards are 1 such example.

-Tim Burke

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

This is because the RH AS 3 is for enterprise use.
The only thing that need to work perfect is the core, and they need to work fast to compete with Solaris, Aix and HP-UX.
What is necessary is a stable and fast kernel with support for lots of mem, lots of disks, low latency, better threading to use in database and ERP softwares.
Sound cards and video cards is a secondary target.
In most part of datacenters, these machines are controled via console, and don't have video, mouse and keyboard.
In this scenario Red Hat AS 3 really do the job.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

A truly herculean effort. It makes you wonder how long the RH internal developers can run at this pace?

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Enjoyed reading it....
Well done...

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Very nice article, indeed.

That gave me some really good arguments in the distro wars.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Whew! Thats alot of work.

Maybe now some people can understand why the RHL was not a worthwhile product for Redhat. Selling for peanuts somthing that is so friggin complicated.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

"Issue 120: Constructing Red Hat Enterprise Linux v. 3
Posted on Thursday, April 01, 2004 by Tim Burke"

Linux is cooler than I thought. Today is March 09, 2004 but this article wont be posted until April 01, 2004, which proves that Linux must have some kind of time machine built in.

Cool!

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix