Wow, time sure flies when you are having fun! Seems like only yesterday I was sitting here writing “Constructing Red Hat Enterprise Linux v.3” (see the on-line Resources). Hard to believe that 16 months have flown by so quickly, resulting in the launch of Red Hat Enterprise Linux v.4 in February 2005. The last article on v.3 provided a behind-the-scenes glimpse of the challenges we face here at Red Hat in order to deliver a robust enterprise-caliber Linux distribution. Although we still face many of the same challenges with the new release, there were many changes in how we conduct business. In this article, I cover the new challenges we faced and how we adapted to address them.

Out of practical necessity, I cover only a small fraction of the hundreds of features and issues we address in a new Red Hat release. Also for this reason, I am unable to identify all of the literally hundreds of contributors, both internal and external. Allow me to apologize up front to my Red Hat friends who escape mention here (it's not that you too aren't awesome).

Truly the most remarkable trend in the computing industry is the dramatic rise in Linux adoption. Seemingly, every day, there are media alerts, on-line articles, notifications from our peers in local Linux User Groups (LUGs) and sales announcements reporting large new user communities migrating to Red Hat Enterprise Linux. For example:

It is an immensely gratifying phenomenon to have the work you do benefit a huge and swiftly climbing user community. The collective user base of both Red Hat Enterprise Linux and the Fedora community version is well above a million users. In fact, due to the proliferation of our software, it is impossible to derive exact numbers to characterize the popularity. Given this scope, all our developers have a strong sense that their contributions truly have impact. There is a betterment of humanity aspect that is inherent with the spread of open-source software.

Given the great diversity of our user base, it becomes increasingly challenging to meet its needs with a finite set of internal developers and testers. In order to keep pace with the growing user base, we needed to find a better way to scale our effectiveness. To accomplish this, we had to look no further than the open-source model that is the core of Red Hat's philosophy. That is, to involve a broader community of participants in an inclusive “early and often” approach. This was the genesis of Fedora.

Fedora is one of the main differences in the Red Hat Enterprise Linux v.4 development as compared to Red Hat Enterprise Linux v.3. There are several objectives of the Fedora Project, including:

The experiences gleaned from Fedora are invaluable in the productisation of Red Hat Enterprise Linux. The Fedora community consists of tens of thousands of users. This volume is larger than the Red Hat Enterprise Linux beta-testing audience. Through the experiences of Fedora, we are able to get a solid understanding of which package revisions and new technologies are mature enough for inclusion in Red Hat Enterprise Linux. The Fedora community members were involved actively in many aspects of development.

A perfect example of community involvement in Fedora development consisted of an external contributor developing an awesome Java application that output diagrams illustrating where time was spent in the boot process. This highlighted slow-starting system services. One such offending service identified by this application subsequently had its starting time corrected to take half a second rather than 20 seconds.

Portions of Fedora are even developed and maintained entirely outside of Red Hat. A key example of this is the yum package delivery and update technology. This shows how Fedora is free to grow in many dimensions, unrestricted from Red Hat's agenda.

For those who demand the latest bleeding-edge technology, Fedora is a perfect, free distribution. For those who demand a more stable supported product, Red Hat Enterprise Linux is the right choice. The Fedora Project has moved ahead in the new technology curve from Red Hat Enterprise Linux v.4. In this manner, it forms a glimpse of promising new features that may appear in future Red Hat Enterprise Linux releases.

The success of the Fedora Project truly has been win-win. Community contributors and users receive a free vehicle to mature open-source technology. Enterprise customers benefit from an increasingly feature-rich and mature product after completion of the stabilization phase.

With this increasingly diverse user base comes a correspondingly large set of requirements. Example requirements include bug-fix requests, software feature addition and hardware enablement. By far, our biggest challenge is to strive to prioritize customer bugs and feature requests to identify the set that yields broadest general usefulness.

In the initial planning phases of Red Hat Enterprise Linux v.4, we carefully reviewed more than 500 feature requests. This was accomplished in numerous marathon sessions of feature reviews interspersed with countless hours of follow-up scoping of the viability and developer time required to deliver. Below are some of the main themes we tried to focus on in Red Hat Enterprise Linux v.4:

Highlights of each of these main themes appear in upcoming sections.

In addition to an increased user base since the introduction of Red Hat Enterprise Linux v.3, we also have fostered closer working relationships with a growing set of hardware and software partners. We recognize that the operating system itself is only one layer in an overall solution stack that end customers need in order to make Linux practical for them in solving their computing needs. For this reason, we work closely with our partners in terms of identifying our priorities, aligning schedules and addressing issues critical in enabling their hardware and software.

Our hardware and software partners increasingly are seeing value in working closely with Red Hat. Historically, it has been highly challenging for us to accommodate the insatiable and diverse requirements from our partners. As much as we would like to satisfy everyone, ultimately we do have a finite staff and time frame in which to do this work. In response, we have invited many of our partners to join us inside Red Hat to work along side our developers to augment our staff to achieve mutually beneficial objectives. For example, we currently have multiple on-site staff members from IBM, Intel, SGI, HP, Fujitsu and NEC. Here are some of the benefits:

The on-site partner contribution benefits the product set beyond the parochial interests of the sponsoring company. For example, although the SGI team's primary mission was support of their large CPU count Altix platform, a side effect was overall improvement in scalability in generic layers, which benefits all architectures. Another example is the work the Fujitsu team accomplished by adding diskdump support. Other hardware partners have augmented this support in Red Hat Enterprise Linux to yield improved problem analysis capability by our collective support organizations.

Numerous on-site partners are here from Japan. We invited them to join us at Boulder Morty's indoor rock climbing gym. It's amazing how much trust it fosters to be hung 40 feet up on a rope with your new-found friends. Given that English isn't their primary language, I often wonder how much of the introductory rock climbing instruction they understood before we gave them the “Go!” thumbs up. Figure 1 shows the Red Hat and partner crew out for our weekly climbing session.

Figure 1. The Red Hat rock stars out for a night of climbing.

One of the major themes of Red Hat Enterprise Linux v.4 was security. Security considerations prevail throughout the entire distribution. For example:

SELinux refers to Security Enhanced Linux. Details of SELinux have been presented in prior Linux Journal articles (see Resources).

At its core, SELinux consists of a set of low-level primitives that provide fine-grained access control. Prior to the advent of SELinux, the Linux security model has been a rather all-or-nothing approach, in that the two common cases were general unprivileged user applications and privileged applications. The privileged applications typically consisted of system services such as bind, Apache, MySQL, Postgres, ntpd, syslogd, snmpd and squid. The historical down-side to having all-powerful system services is that if they were compromised by a virus attack or other exploit, the entire system could then become compromised.

SELinux provides a means of tightly restricting the capabilities of user applications and system services to a strict need-to-know authorization. For example, it sets access control on the Apache Web server (httpd) to limit the set of files and directories it is able to modify. Additionally, Apache is strictly limited to what other applications it is capable of executing. In this manner, if Apache is attacked, the set of damage that can occur is well contained. In fact, SELinux is so well contained that one of Red Hat's developers, Russell Coker has set up a Fedora system where he provides the root password and invites anyone to see if they can inflict damage to the system.

What is most monumental about Red Hat Enterprise Linux v.4's SELinux implementation is that it is the first widely adopted commercial operating system to provide such fine-grained security integrated in the newest release. Historically, it has been the case that such fully featured secure operating systems have been relegated to obscure forks of mainstream products, which typically have lagged a year or two behind the respective new releases.

The implementation of SELinux got its tentacles into virtually all areas of the distribution. This included:

There were many challenges in the implementation of SELinux. On the kernel front, the core SELinux primitives were highly at risk of being accepted into the upstream 2.6 Linux kernel. James Morris valiantly completed the implementation and garnered the required upstream consensus. On the user-level package front, the introduction of SELinux required a specific or default policy to be constructed for each package. Naturally, this at times was a bumpy process as we sorted out which files should be writable and other details.

Minor implementation glitches would wreak havoc across the entire distribution. However, it also resulted in SELinux being the initial scapegoat for virtually all problems. Dan Walsh was a true workhorse in pouring through this onslaught of issues.

“Upstream, Upstream, Upstream”—this became the mantra among our kernel team throughout the entire duration of Red Hat Enterprise Linux v.4 construction. The reason for this is that every change in which Red Hat's kernel diverges from the upstream Linux community kernel.org becomes a liability for the following reasons:

These principles are core to the value of true community open-source development. As testament to Red Hat's active participation in the upstream Linux kernel community, through the course of 2.6 development more patches were accepted from Red Hat kernel developers than from any other company. During the past year, more than 4,100 patches from Red Hat employees were integrated into the upstream 2.6 kernel. In contrast, other companies boast that their offering contains the most patches on top of the community kernel. An interesting statistic is that currently, more than 80% of all kernel patches originate from kernel developers employed explicitly to do such development. The kernel has become mostly a professional employment endeavor, not a hobbyist project.

Red Hat's developers were highly active in upstream 2.6 development. Some of the areas of involvement included:

Arjan van de Ven and Dave Jones, Red Hat Enterprise Linux v.4 kernel pool maintainers, integrated kernel contributions from our collective internal kernel development team.

They frequently rebased our trees against the latest upstream kernels as well as integrated additional bug fixes, performance tunings, hardware platform support and feature additions. This is truly a monumental effort given that we simultaneously support seven different architectures (x86, x86_64—AMD64 and Intel(r) EM64T, Itanium2, IBM Power (31- and 64-bit), mainframe in 31- and 64-bit variants) from a single codebase.

Initially, it was painful for Arjan to be beating everyone over the head to ensure that all patches were accepted upstream prior to incorporating them into our pool. Through his vigilance, the entire team became conditioned to working upstream first. In the short term, it involves more effort on the part of the developer to work both internal to Red Hat as well as upstream. However, in the long term, as described above, the benefits are considerable.

A large class of new Linux deployments consists of proprietary UNIX migrations. These users represent a set of enterprise customers who have high expectations (a euphemism for highly demanding). Traditional functionality gaps in Linux consist of robust software volume management capabilities. In response to these needs, over the course of Red Hat Enterprise Linux v.4, Red Hat acquired a strong team of storage-centric experts when Red Hat purchased Sistina. In this manner, Red Hat now employs the major upstream developers of the Logical Volume Manager (LVM) technology.

Overall ease of use has been improved in the installer, where it now enables the user to create LVM volumes. Through the use of a graphical interface in Disk Druid, usage of LVM is much more approachable to the end user. Another example of ease-of-use improvements are the capabilities to grow both LVM volumes and ext3 filesystems that are on-line. This obviates the need to unmount the filesystem, back up, grow the volume, reformat the filesystem and restore the data.

We also wanted to take open-source storage management to the next level to provide a cluster filesystem. The industry trends have been toward distributed computing among large sets of commodity computers. Although that yields cost savings in hardware, it increases costs of managing storage and filesystems among a distributed pool of servers. To address this need, Red Hat has augmented the LVM layer to operate in a clustered environment by layering a robust cluster filesystem called GFS.

In keeping with Red Hat's core values of being an open-source player, the complete source code base for LVM and GFS is now freely available to the Linux community at large. Ongoing development has rekindled industry-wide contributions. Cluster Suite is the name of the productised version of GFS and LVM, which is layered on top of Red Hat Enterprise Linux.

One of Red Hat's largest areas of increased investment is in what we refer to as the desktop space. Under the guidance of Havoc Pennington, we have formed an extensive close-knit team of developers. The primary mantra of the desktop team has been ease of use. If you look closely at the new adoptions of Linux you will see an increasing trend of usage in less computer-savvy scenarios. Examples include kiosks, call centers, government agencies and earlier grade-school levels.

The desktop team worked with our application developers to identify the most useful application set. Although there are more than 80,000 projects on Sourceforge.net, for example, it is impractical to include all of them in our distribution. One of our main roles as a system integrator is selecting and organizing the most useful applications. In v.4 we have reorganized how the applications are listed in the menus so as to be grouped logically by function.

Inside the walls of Red Hat, we take the open-source model to an extreme, where product decisions are debated by anyone who has a nearby soapbox. Given that everyone is a self-proclaimed authority on “what the users want” and what “usability” means, this provided ample fodder for highly emotionally charged debates. This all came to a head in the selection of the default browser. The main contenders were Firefox and Epiphany. The on-line e-mail debates raged on. In the end, Havoc pulled all interested parties together for a raucous conference call to hash things out. The result was the selection of Firefox. Given the huge amount of attention that Firefox has been garnering, both in the media and practical deployments, we think we made the right choice.

These debates are a core part of being at Red Hat. They become so volatile because the crew sincerely cares about what they are doing. Most people here feel part of something bigger than a small company. This high level of energy, creativity and enthusiasm found at Red Hat makes it extremely challenging to be a manager. Sometimes it seems like I'm a referee to a crew of prize fighters, who in addition to sparing with each other, often share a punch to the head with me too. Perhaps I should have strived to find a more constructive example. It's really not combative here, just highly stimulating and challenging. After living in this world for 3.5 years now, I can't imagine what its like to work at a place that would be “just a job”.

One of the key usability technologies that our developers (including Havoc Pennington and John Palmieri) were involved with is D-BUS (see Resources). D-BUS is a communication and event mechanism that enables a range of desktop applications to compliment each other in a coordinated manner. For example, the insertion of a CD results in launching of a corresponding application depending on media format type. Similarly, D-BUS is used for USB device hot plug, for example, to initiate configuration and startup of network services or mounting filesystems from USB pendrives.

Ease of use was further enhanced through the bundled collection of third-party proprietary applications. This is done for the convenience of the end user, so that it doesn't become an egg hunt for them to find commonly used applications. This resulted in the bundling of RealPlayer, Helix Player, Adobe Acrobat Reader, Citrix, Macromedia Flash and a Java runtime environment (JRE).

In April 2004, Red Hat conducted a global company meeting in Raleigh, North Carolina. The entire company was invited. One of the strongest impressions I took from this meeting was how truly worldwide Red Hat is. It seemed as though there were as many non-US team members as US members. In addition to the US, development is conducted in Australia, Canada, Germany, Czech Republic, UK, Japan, India and Brazil.

Not all development is conducted within the offices of Red Hat. Through the worldwide legions of contributors to Fedora we invite broader participation. We actively contribute and draw from a great diversity of community open-source projects. Again, this substantially broadens the circle of participation. In many ways, this inclusive process makes Red Hat feel like a trusted steward of the community, forming a distribution representing the best and brightest technology. This is a privilege we do not take for granted as we know it needs to be continuously earned every day. This makes both Red Hat Enterprise Linux and Fedora truly distributions “by the people, for the people”.

Red Hat Enterprise Linux v.4 is supported in 15 different languages. These translations are all performed as an integral part of the development cycle. Consequently, the translation process doesn't lag the release or introduce forks in the development trees. We have a team of “translation elves” located in Australia who magically do their work at an opposite phase of the clock from headquarters. This results in a nearly real-time translation that tracks development changes. Additionally, there are many contributors to Fedora who are actively involved in internationalization activities.

Figure 2. The Red Hat Crew from the Westford, Massachusetts Office

There are several ways in which Red Hat has improved upon our development methodology over the course of Red Hat Enterprise Linux v.4's construction. Interestingly, the main theme of these improvements has been to stick to core proven Linux open-source development practices. Although we did subscribe to these practices previously, we paid increased focus this time around to the following:

We are all extremely grateful for the steady guiding influences of Donald Fischer who did an outstanding job as overall product manager and release manager. He was at once a diplomat, innovator, book-keeper and go-to guy. Hats off to “the Donald”.

Red Hat is truly a restless place to be. It seems that no sooner have we shipped one release, than we are already behind on the next one. This is due to the fact that in addition to new release development, we also support prior releases for a seven-year interval. So, for example, here's the list of releases concurrently in development now:

Never a dull moment, and we wouldn't have it any other way!

Resources for this article: /article/8204.