Constructing Red Hat Enterprise Linux v. 3

 in
Putting together a Linux distribution gets a lot more complicated when stacks of requirements start arriving from hardware vendors and other partners.

Many people have little understanding of the behind-the-scenes efforts required to construct an enterprise Linux distribution. The fact that this process largely can be taken for granted is actually a compliment. This article offers a glimpse into the methodology we used to deliver Red Hat Enterprise Linux v. 3 (Enterprise Linux v. 3). As you will see, we faced numerous challenges along the way. Then again, if it would have been easy, it wouldn't have been so much fun.

Throughout this article, the focus is on how the release was put together. This article primarily discusses the development of the kernel used in Enterprise Linux v. 3. The kernel is only a fraction of an overall distribution, the portion that controls the underlying hardware and system resources. The challenges faced by the other teams, with projects such as compiler tools, the installer, hundreds of application packages, documentation and testing, are equally daunting. Each of these items was developed by gifted individuals.

Allow me to start by noting that we here at Red Hat have established strong relationships with our key partners in the industry. Although partners' anonymity is guarded here, I'm sure they can identify where they fit in to this story and recognize that it's all in good humor.

What Is an Enterprise Distribution?

High customer expectations have been set by the proprietary UNIX operating systems, and customers planning to migrate from UNIX to Linux do not want to adopt technology that cannot deliver the same level of robustness, quality, support and compatibility. Business users demand stability and reliability. In some cases this means bleeding-edge technology is not appropriate for inclusion in a product. Users also want the ability to run across a wide range of architectures and hardware components, thereby realizing the Linux goals of avoiding proprietary vendor lock-in. Support needs to be in the form of ongoing maintenance for several years, including security and bug fixes, as well as incremental hardware support and valuable, but not destabilizing, feature enhancements.

Requirement Gathering

The ball gets rolling on a new release by identifying the targeted feature set. Next to strong opinions, the second most plentiful commodity at Red Hat is feature requests. They come from all directions, such as insatiable independent hardware/software vendors, demanding customers, both large and small, and Red Hat's worldwide sales and service support organizations. Additionally, as many ideas are generated within Red Hat engineering, ranging from performance and usability enhancements to marketing proposals on how to organize the product set.

Obviously, we don't have infinite developer resources, so the major challenge is to choose the best of these ideas. Another key feature acceptance criteria is conformance with upstream Linux direction, which is necessary for compatibility and to remain true to the spirit of the kernel.org tree governed by Linus Torvalds and containing contributions from around the world.

Here's a few examples of the challenging scenarios we face in requirements gathering:

  • We ask each of our partners to submit a reasonably sized top-ten list. One partner's requirements came in the form of a binder that was two inches thick. This became affectionately known as the bible. My first exposure to the bible came when it was heaved onto my desk with a resounding thud. When I saw that, I swear that my heart went thud, too.

  • The more mathematically inclined partners do internalize the concept of a top-ten list. However, most ended up using a tactic that those of you familiar with TCP/IP networking should recognize—the sliding window protocol. The way it works is as soon as any of your features have been accepted, those pop off the top of the stack, freeing up space for features 11, 12 and 13 to all of a sudden become cataclysmic issues.

The following is a representative example feature list from a hardware vendor:

  1. Support more than 32GB of memory on x86.

  2. Support more than eight CPUs.

  3. Support for our new XYZ100 series computers. (This is a thinly veiled multiple feature request; behind it is a series of required device drivers, PCI IDs and installer hooks.)

  4. Updated I/O adapter driver. (This ultimately turns into differences of opinions regarding to which newer version this refers).

  5. Integrated support for the vendor's proprietary baseboard management software. (A perennial list item, which consistently gets rejected to the amazement of the requester.)

  6. Compiler optimizations to match the latest chipsets.

  7. USB support for our CD-ROM drive. (Needed because the drive is brain-dead and doesn't conform to the spec—of course that subtlety is absent in the initial feature request.)

  8. Support for more than 128 disks.

Then, there's typically the following implied requirements:

  • All of these feature requests apply to multiple architectures, including x86, AMD64 and Itanium 2.

  • Oh, by the way, we also want this feature backported to the prior Enterprise Linux v. 2.1 release.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

This is a comment addressed to RedHat ->
RH9 converted me from Windows because of its polished finish. Thank you.

Some advice: That was a confusing article title as I didn

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Actually, there is a cheap version sold basically at media cost for academic users:

http://www.redhat.com/solutions/industries/education/products/

WHAT about FEDORA?

Anonymous's picture

This article completley negates and ignores the fact that RHEL had RH9 (and in future Fedora Cora) to use as a base for testing as well.

The REAL open source community uses the community version - that's where it's proven. The enterprise hacks put it in a nice box.

Nice try to pull the wool over our eye LJ but we know better we're not WINDOZE users after all...

Re: WHAT about FEDORA?

dmarti's picture

How many Fedora users have 8-way SMP systems and 128 SCSI disks?

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Wow, time travel! The article was posted on 4/1/2004 which is 8 days in the future as I enter this comment. I never knew Enterprise Linux was so powerful. :-)

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Great marketing. A lot of blather for collecting other people's work.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Please look at a recent changelog at kernel.org and count the redhat.com addresses.

http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.4

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Thats because RH doesn't go way out of its way to advertise about all the work we do upstream. For example, here's just a few of the big ticket items that come to mind:

- the whole new NPTL threading model
- rmap VM
- lvm2/device mapper
- O(1) scheduler
- ext3
- SATA subsystem
- ipv6, ipsec
- LSM
- PIE
- lots of network drivers

Plus a ton of misc fixes. This list is by no means comprehensive, as its just kernel stuff.

RH's gcc team (formerly Cygnus) is the heart of gcc.

A substantial portion of the GTK development crew is here.

Substantial contribution among the 1500 rpm packages.

Tally it up yourself. To me that looks like a lot more than just collecting up other people's work.

-Tim Burke

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Nice troll. It succinctly demonstrates your lack of knowledge about software development, and product development.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Hmmm... I must have missed the part where removing support for ISA soundcards was a requested feature... They were supported in 2.1! Or adding a new, broken bindconf that still hasn't been fixed, though it's been out and broken for five months. Or even leaving out many, many *-devel rpm packages, such as sendmail-devel, which enables one to actually add and use milters with the milter-enabled base rpm.

If only the core bits got extensive testing, then that would explain why RHEL3 is worse than RHL9 from a "working-features" perspective.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Thats because the amount of possible work is almost infinite while the staff we have at RH is finite. Ultimately we do have to make some hard tradeoffs. We can't please all of the people all of the time. For example, we pour a lot of effort into new hardware support. Things like AMD64, sata, graphics cards, new storage and networking adapters, etc. So due to new stuff, the pile just gets bigger. Something has to give. This is why sometimes some things have to fall out the other end. ISA sound cards are 1 such example.

-Tim Burke

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

This is because the RH AS 3 is for enterprise use.
The only thing that need to work perfect is the core, and they need to work fast to compete with Solaris, Aix and HP-UX.
What is necessary is a stable and fast kernel with support for lots of mem, lots of disks, low latency, better threading to use in database and ERP softwares.
Sound cards and video cards is a secondary target.
In most part of datacenters, these machines are controled via console, and don't have video, mouse and keyboard.
In this scenario Red Hat AS 3 really do the job.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

A truly herculean effort. It makes you wonder how long the RH internal developers can run at this pace?

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Enjoyed reading it....
Well done...

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Very nice article, indeed.

That gave me some really good arguments in the distro wars.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Whew! Thats alot of work.

Maybe now some people can understand why the RHL was not a worthwhile product for Redhat. Selling for peanuts somthing that is so friggin complicated.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

"Issue 120: Constructing Red Hat Enterprise Linux v. 3
Posted on Thursday, April 01, 2004 by Tim Burke"

Linux is cooler than I thought. Today is March 09, 2004 but this article wont be posted until April 01, 2004, which proves that Linux must have some kind of time machine built in.

Cool!

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState