Constructing Red Hat Enterprise Linux v. 3

 in
Putting together a Linux distribution gets a lot more complicated when stacks of requirements start arriving from hardware vendors and other partners.
Late-Breaking Features

Just as the sun never sets on kernel development, our near-and-dear partners and customers don't sit still either. One of the few constants in this dynamic environment is that throughout the course of the release development, there is a steady influx of must-have crisis feature additions. We endeavor to make it a trade-off and kick out a lower priority feature in order to keep the workload sane. In the end, however, it never seems to work out to such a sweet balance. The ability of the team to persevere through all these demands is remarkable.

As productive as the Red Hat developers are, some late-breaking features always have to be deferred to a later release or update. These situations cause no end of trauma for our partner managers who have to be the bearers of bad news. It's improbable that our partners ever heard the saying “don't shoot the messenger”. There was one incident when we were two hours from shipping the release and a delivery arrived on the loading dock. It was a new computer platform we needed in order to be able to develop and test support for it. The partner was incredulous that we were unable to accommodate.

Testing

Several different levels of testing are performed throughout the development process. It all begins with the developers performing unit testing. This consists of manual, hands-on testing as well as development of automated testing programs. The set of automated tests constantly is being augmented as new problems are addressed. These automated test programs then are incorporated into a test grid that is managed by the QA department. In addition to the internally written unit tests, our test grid includes a wide range of regression tests. Examples include POSIX, LSB conformance, LTP, crashme, gcc suite and diabolical tests provided by our partners. Stress tests also are performed for a range of system functions using such tests as cerberus, lmbench, bonnie, spec and other micro-benchmarks, to name a few. These automated tests are run nightly in order to detect regressions quickly. This is critical in order to isolate the offending code. In contrast, if we waited to perform monthly base-level testing, it would be much more difficult to identify the culprit.

On top of the nightly tests, more time-consuming test scenarios are run less frequently. Examples include installation testing using a wide range of configuration options across the many different languages we support. Hands-on testing rounds out the internal QA coverage. Doing justice to the staggering range of testing done by the hardworking QA team here substantially exceeds the scope of a single article.

After the kernels have passed internal units tests and QA scrutiny, we make them available to our development partners. This vastly broadens the coverage to include external QA and development teams in other companies. Our partners focus testing on their hardware platforms as well as the typical server capabilities their enterprise customers demand.

The last layer of testing includes external beta testers. These beta testers include high-profile customers, as well as the many people in the Linux community who respond to our open invitation to help with testing.

The combination of all these different testing activities yields an extremely stable and well-tested product. It also points out the huge value of the open-source model. It is the combination of testing resources from many different companies and dedicated individuals that scales well beyond the resources a single company could bring to bear to tackle an infinite testing matrix.

Figure 3, composed by Nick Carr, summarizes the development model from requirements gathering, development and testing, culminating in production.

Figure 3. The Development Process, from Requirements Gathering to Release

Conclusion

In the end, the Red Hat Enterprise Linux v. 3 distribution did ship on schedule in mid-October 2003. Having worked for many years for a major IHV on a large-scale operating system engineering team, I have been exposed to both the proprietary operating system development model and the open-source development model. The successful outcome of the Enterprise Linux v. 3 release clearly demonstrates the power of cooperative open-source development. The quantity of features, rapid development time and caliber of participants, both internal and external, is remarkable and stands as a strong testament to the Linux community.

When the release shipped, we all were proud to have contributed to such a tremendous accomplishment. But the time window to rejoice was short indeed. As soon as Enterprise Linux v. 3 finished, it seemed we already were late with the development of Enterprise Linux v. 4. This reminds me of the quote, “Congratulations for winning this battle, that earns you the privilege to come back and fight another day.” Gotta go, more battles to fight.

Tim Burke is the director of Server Development at Red Hat. This team is responsible for the kernel and the set of core applications included in Red Hat Enterprise Linux. Prior to becoming a manager, Tim earned an honest living developing Linux highly available cluster solutions and UNIX kernel technology.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

This is a comment addressed to RedHat ->
RH9 converted me from Windows because of its polished finish. Thank you.

Some advice: That was a confusing article title as I didn

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Actually, there is a cheap version sold basically at media cost for academic users:

http://www.redhat.com/solutions/industries/education/products/

WHAT about FEDORA?

Anonymous's picture

This article completley negates and ignores the fact that RHEL had RH9 (and in future Fedora Cora) to use as a base for testing as well.

The REAL open source community uses the community version - that's where it's proven. The enterprise hacks put it in a nice box.

Nice try to pull the wool over our eye LJ but we know better we're not WINDOZE users after all...

Re: WHAT about FEDORA?

dmarti's picture

How many Fedora users have 8-way SMP systems and 128 SCSI disks?

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Wow, time travel! The article was posted on 4/1/2004 which is 8 days in the future as I enter this comment. I never knew Enterprise Linux was so powerful. :-)

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Great marketing. A lot of blather for collecting other people's work.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Please look at a recent changelog at kernel.org and count the redhat.com addresses.

http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.4

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Thats because RH doesn't go way out of its way to advertise about all the work we do upstream. For example, here's just a few of the big ticket items that come to mind:

- the whole new NPTL threading model
- rmap VM
- lvm2/device mapper
- O(1) scheduler
- ext3
- SATA subsystem
- ipv6, ipsec
- LSM
- PIE
- lots of network drivers

Plus a ton of misc fixes. This list is by no means comprehensive, as its just kernel stuff.

RH's gcc team (formerly Cygnus) is the heart of gcc.

A substantial portion of the GTK development crew is here.

Substantial contribution among the 1500 rpm packages.

Tally it up yourself. To me that looks like a lot more than just collecting up other people's work.

-Tim Burke

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Nice troll. It succinctly demonstrates your lack of knowledge about software development, and product development.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Hmmm... I must have missed the part where removing support for ISA soundcards was a requested feature... They were supported in 2.1! Or adding a new, broken bindconf that still hasn't been fixed, though it's been out and broken for five months. Or even leaving out many, many *-devel rpm packages, such as sendmail-devel, which enables one to actually add and use milters with the milter-enabled base rpm.

If only the core bits got extensive testing, then that would explain why RHEL3 is worse than RHL9 from a "working-features" perspective.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Thats because the amount of possible work is almost infinite while the staff we have at RH is finite. Ultimately we do have to make some hard tradeoffs. We can't please all of the people all of the time. For example, we pour a lot of effort into new hardware support. Things like AMD64, sata, graphics cards, new storage and networking adapters, etc. So due to new stuff, the pile just gets bigger. Something has to give. This is why sometimes some things have to fall out the other end. ISA sound cards are 1 such example.

-Tim Burke

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

This is because the RH AS 3 is for enterprise use.
The only thing that need to work perfect is the core, and they need to work fast to compete with Solaris, Aix and HP-UX.
What is necessary is a stable and fast kernel with support for lots of mem, lots of disks, low latency, better threading to use in database and ERP softwares.
Sound cards and video cards is a secondary target.
In most part of datacenters, these machines are controled via console, and don't have video, mouse and keyboard.
In this scenario Red Hat AS 3 really do the job.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

A truly herculean effort. It makes you wonder how long the RH internal developers can run at this pace?

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Enjoyed reading it....
Well done...

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Very nice article, indeed.

That gave me some really good arguments in the distro wars.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

Whew! Thats alot of work.

Maybe now some people can understand why the RHL was not a worthwhile product for Redhat. Selling for peanuts somthing that is so friggin complicated.

Re: Constructing Red Hat Enterprise Linux v. 3

Anonymous's picture

"Issue 120: Constructing Red Hat Enterprise Linux v. 3
Posted on Thursday, April 01, 2004 by Tim Burke"

Linux is cooler than I thought. Today is March 09, 2004 but this article wont be posted until April 01, 2004, which proves that Linux must have some kind of time machine built in.

Cool!

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState