Ultimate Linux Box 2005
Let's just call 2005 the year of power management. Processor vendors made a big deal out of whitepapers about saving watts, and we heard a lot about power management at LinuxWorld Conference and Expo in February.
Did the industry start caring about global warming? Do IT CEOs want to eat swordfish more often, so they have to reduce the mercury emissions of power plants? Not quite. Today's server systems are packing more and hotter processors closer together, and customers' air-conditioning systems aren't ready for the strain. NASA had to install water cooling for its 10,240-processor Columbia cluster, as we showed in our January issue.
Every watt-hour you can save is heat that the customer doesn't have to deal with—3.6kJ, or 3.4BTU to be precise. With data centers full of blade servers, and 1U systems sporting as many as four processors, all that heat really adds up.
The Linux desktop greedily devours the scraps from the multibillion dollar Linux server market, and power consumption matters to us on the desktop too. Fans are loud. If you have better power management on your processors, they produce less heat, and you can run fewer fans or run the fans you do have more quietly. We took a different approach to fans, as you'll see later on.
Finally, of course, power matters on the laptop and on portable devices because of battery life. We'll leave the specifics of tweaking for maximum off-AC time to future articles.
We like Tyan motherboards, and companies that build custom Linux systems do too. The four-Opteron Tyan Thunder K8QS Pro came out just a little too late to make it into last year's Ultimate Linux Box. It's based on an AMD 8000 series chipset. When we say “chipset”, we mean a slightly different combination of hardware from an Intel-based system, though. The AMD64 way is to have an onboard memory controller per processor, give each processor its own bank of memory and link them with HyperTransport. Your AMD64 “SMP” box is really a mini-NUMA, and the “chipset” doesn't include the memory controller.
Last year, we used a Celestica A8440 bare-bones rackmount system as the basis for the Ultimate Linux box. Although starting with pre-integrated chassis and power supplies can be a great time saver, we realized that last year's box was on the loud side. This year, going back to our usual plan lets us pick everything else just the way we want it.
The K8QS Pro has two PCI-X busses, A and B. B is dedicated to two 133MHz-capable PCI-X slots, and A offers two PCI-X slots maxing out at 66MHz and one regular PCI slot. Onboard networking is two Broadcom BCM5704C Gigabit Ethernet interfaces, also on bus A.
There are all the regular PC ports, of which we're using the USB. SCSI and serial ATA are options, which you might want to keep in mind if you're planning to move this board into a more conventional server role when you're building your next Ultimate Linux Box.
Into this mighty board we plugged four of the best of the Opteron processors available at the time—the 846 HE, clocked at 2.0GHz and offering 1MB of L2 cache. See the sidebar for what became available while we were testing the system. We maxed out the system's main memory at 32GB.
Unfortunately for case shoppers, this board is SSI MEB size—13"x16" or 330.2x406.4mm. Not a problem for us because we're using a custom case this year, but the size does limit your case options.
When we're picking out a case for any custom-built system, Ultimate or otherwise, we usually get one that's quite a bit larger than what a big vendor would use for a comparable system. Smaller cases require less material and they're cheaper for vendors to ship, but since we like to tweak things, we get a case with more room to add devices and more room to work inside.
In order to have a completely silent system, you need to move storage outside the box. Options for doing this have changed a lot since the days when you had a choice between NFS and external SCSI enclosures connected by a 3-meter cable.
Today, you can make your drives go away using USB, FireWire, SCSI of course, Fibre Channel or the new ATA over Ethernet, which we covered in the June 2005 issue. A separate storage enclosure is no longer only an enterprise server-room thing.
Another option is simply to boot over the network and mount your storage via NFS. Since Penguin works with enterprise server-room hardware, and Fibre Channel does deliver impressive benchmark results, we went with it; an nStor 4320F Fibre Channel RAID enclosure, with Hitachi 18GB drives for the OS and larger Seagate drives for more storage.
Listing 1. Partition scheme as seen in /etc/fstab.
LABEL=/nstor-OS / ext3 defaults 1 1 LABEL=/cfboot /boot ext3 defaults 1 2 LABEL=/nstor-DATA /u1 ext2 defaults 1 2 none /dev/pts devpts gid=5,mode=620 0 0 none /dev/shm tmpfs defaults 0 0 none /proc proc defaults 0 0 none /sys sysfs defaults 0 0
Because we wanted the system to be self-contained and not depend on another server to boot, we installed a Sandisk 256MB CompactFlash card to boot from. This device looks exactly like another ATA drive to the system, so any PC motherboard will boot from it.
We considered using a USB thumbdrive, but that would have required some initrd drive juggling and GRUB wizardry. There are advantages to being able to pull your boot device out of the system and store it separately, but we didn't anticipate shipping the system through airports with drives loaded with encrypted confidential data.
If you plan to leave your silent Linux system on your network, you'll be a little more flexible in booting, and you can set up PXE booting. But if you want to take your Ultimate Linux Box over to a friend's house to play some music, you'll want to be able to boot independently. The Penguin crew plans to take this system to LinuxWorld Conference and Expo, and when you're wrangling hardware for a tradeshow one fewer thing to set up is good.
If you do build and install a silent Linux box, you'll probably end up doing a mix of both: NFS for user home directories, the company /usr/local/bin/ and other items that need to be in sync but aren't performance-critical. You can save your machine's own filesystems for big working files, like all the audio data you'll get from this system's high-end sound hardware.
Finally, to take even the keyboard clicking out of the silent system, Penguin founder Sam Ockman suggested a TouchStream LP keyboard, which works like a touchpad and requires no moving parts. It's also a pointing device and lets you map gestures to interface actions.
For the first time, we put professional audio hardware into the Ultimate Linux Box. What better place for a silent machine than the recording studio?
The RME Hammerfall HDSP9652 card we chose for this system is capable of up to 52 channels, and we matched it with an external box called the Multiface that brings out 8 1/4" jacks, as well as optical, coax and MIDI.
This card is as close as you can get to a “studio in a box”, because it's built around an internal mixer and allows you to route signals around inside the card with low latencies and low load on the CPU. Other features include the ability to “punch in” and “punch out” like a conventional tape deck.
Best of all, RME has been supporting the Advanced Linux Sound Architecture (ALSA) Project since 2000, so Linux users aren't second-class citizens. RME's site says, “ALSA support for the Hammerfall breaks the annoying chicken/egg principle—no professional hardware/driver, no professional software.”
Peter Todd covered the necessary tools for working with the Hammerfall HDSP cards in our October 2003 issue.
For video, we used a relatively low-end card (see the on-line Resources). We'd really like to start putting interesting and innovative video on Ultimate Linux Boxes, but there are still some issues with the drivers (see sidebar).
So how do we keep this thing cool? First of all, it's important not to start tweaking with hardware combinations unless you know how to measure the effects that your changes have on the system's temperature. Don't change anything unless you know how to measure the effect of the change.
The good news is that the processor and motherboard vendors thoughtfully give us temperature sensors right on the key parts. And we can keep track of them using an all-important tool, lm_sensors.
We didn't have to measure drive temperature because we moved the drives to a separate enclosure, but smartmontools (see Resources) gives you an easy way to do that.
We ordered up some parts from Zalman, which offers a beautiful set of water-cooling hardware. The most visible part is the Reserator 1, a combined water reservoir and radiator that stands half a meter tall and holds 2.5 liters of water. Besides the Reserator, we also ordered one CPU waterblock per processor and matching tubing.
Thermal estimates showed that we wouldn't need a full Reserator per processor, so we used one Reserator per two processors and one for the power supply.
The Reserator comes with a 5W pump, which would break our beautiful silence, so it was time to convert it to operate purely by convection. In its stock configuration, the Reserator's inlet and outlet are close to each other, so we installed a tube inside each Reserator, running from the hot inlet to near the top.
What's that in your cubicle, Justin? We tested convective cooling with a scratch system and lm_sensors.
Did it work? The processor temperature climbed to about 50° C, then the tubes leading up from the processors to the Reserators warmed enough to start the convection. Temperature fell to 47° or 48° C in normal use, and running full-out, the system holds out below 50° C.
Cooling the power supply was a little harder. Zalman's beefiest fanless power supply is only 400W, and a big four-way board needs more. We decided to use the PC Power and Cooling Turbo-Cool 510 ATX.
We decided not to design and build a power supply for the project, since it's important to apply power to components in the right order, and we know PC Power and Cooling solved that problem for us. The cooling problem remained.
Enter the magic of metalworking. Phil brought the problem to a machine shop called Global Precision, and we had them do three pieces of work. They machined down the original fins of the power supply's heat sinks to create flat areas for attaching waterblocks. They made the waterblocks themselves—using blue anodized aluminum to match the Zalman parts. And they made two custom Y-connectors to split the water flow between the two heat sinks.
We removed the fan control board from the power supply. We didn't need it any more.
Cases capable of accommodating and doing justice to Ultimate Linux Boxes are rare. This year, only one alternative would work: going full custom. This year's case has acrylic windows to show off the cooling system, integrated supports for the three Reserators and a mounting place for the RME Multiface.
Difficult as it might be for us to believe right now, many real-world systems don't need both 52-channel audio and Fibre Channel. But unusual combinations of hardware are what enable creative projects, and we're happy that Linux stays out of our way and lets us hook up what we want.
When you start with what's possible and take out what you don't need, you'll be confident that you can build a machine for your needs. We hope that whatever class of system you decide to build, you'll get some ideas out of this year's Ultimate Linux Box.
Ultimate Linux Box 2005 Parts List
Motherboard: Tyan Thunder K8QS Pro (S4882)
CPUs: 4 x AMD 846HE Opterons
RAM: 8 x 4GB Registered ECC Samsung DDR PC2700 CL 2.5 DIMMs
Power supply: 510W Custom harness PC Power and Cooling Turbo-Cool 510 ATX (modified)
Case: Custom, designed by Matt Fulvio, constructed by Trevor Sherard
Fibre Channel: Qlogic 2342 dual-port, 133MHz, PCI-X, 2Gb Fibre Channel adapter
Boot device: Sandisk 256MB CompactFlash card, DCFB-256-A10
Storage: nStor 4320F Fibre Channel RAID enclosure
Hard disks: 2 x 18Gb Hitachi DK32DJ-18FC 10KRPM Fibre Channel drives in a RAID 1 array (OS install) and 6 x 73Gb Seagate ST373405FC Cheetah 73LP FC 10KRPM Fibre Channel drives in a RAID 10 array
Graphics card: PNY NVIDIA Quadro NVS 280 PCI
Displays: 2 x ViewSonic VX2000 20" 1600x1200 LCD displays
Audio card: RME HDSP9652 PCI audio card
Audio I/O: RME Multiface 36-channel 24-bit 96-kHz I/O box
Cooling system: 3 x Zalman Reserator 1s.
CPU waterblocks: 4 x Zalman ZM-WB2 Gold waterblocks
P/S Cooling HW: Custom-designed and machined by Global Precision.
Hardware of the Future, Lawyers Stuck in the Past
It never fails. New products that we'd like to try in the Ultimate Linux Box come out right when we're in the middle of building this year's.
Too late to make it through our thermal testing, AMD introduced dual-core Opteron processors, which let you build an eight-way system on an existing four-socket motherboard with a BIOS upgrade. Today, that means spending $10,000 on processors, but (all together now) we expect prices to come down.
We're watching the progress of the LinuxBIOS Project (see page XX) and are planning to get a supported motherboard for next year. We know patience is a virtue, but booting in mere seconds is cool for its own sake.
This year's system sounded so nice that we'd like to do another quiet machine next year. That means we have to pick a storage technology, and added to next year's list of alternatives will be ATA over Ethernet, as covered in Ed Cashin's article in the June 2005 issue.
Video is still a weak spot, not because of hardware problems, but because of the vendors' lawyers. Everybody doing 3-D is infringing everyone else's patents, and burying the driver code behind a proprietary EULA with a no-reverse-engineering clause is only slowing the industry down. When the normal kernel development process frequently breaks the driver for commonly used hardware, that hardware needs to get with the program.
Graphics vendors, please get together, cross-license the patents for hardware, and come up with a license for software and documentation that lets developers release the new code that makes people want graphics hardware in the first place. It'll help everyone in the long run—NVIDIA maintains an entire parallel software distribution system just because of its licensing decision. Why not get that cost center out of the budget?
Pessimists will say to be “realistic” and accept the proprietary drivers. But realistically, the UNIX vendors of the mid-1990s weren't about to support Linux, either. Today, every UNIX vendor is out of business or supporting Linux. I'll put reality up against “realistic” any day.
dbench with 100 simulated clients:
%dbench 100 Throughput 1234.57 MB/sec (NB=1543.21 MB/sec 12345.7 MBit/sec)
Bonnie++ 1.03—a more accurate disk benchmark:
Sequential output by character: 58,577Kb/s, 98% CPU
Sequential output by block: 281,032Kb/s, 50% CPU
Sequential output, rewrite: 52,603Kb/s, 18% CPU
Sequential input by character: 34,717Kb/s, 58% CPU
Sequential input by block: 90,097Kb/s, 11% CPU
Random seeks: 257.5/s
Sequential create: 5,924 files/s
Random create: 6,056 files/s
Postmark benchmark—Postmark simulates the operations of a busy mail server. For 20,000 base files and 100,000 transactions, we obtained the following results.
46 seconds total
40 seconds of transactions (2,500/s)
70,128 created (1,524/s); Creation alone: 20,000 files (5,000/s); Mixed with transactions: 50,128 files (1,253/s)
49,656 read (1,241/s)
50,199 appended (1,254/s)
70,128 deleted (1,524/s)
Deletion alone: 20,256 files (10,128/s); mixed with transactions: 49,872 files (1,246/s)
303.46MB read (6.60MB/s)
436.18MB written (9.48MB/s)
Kernel compile: 50s
Resources for this article: /article/8330.
Justin Thiessen is a Linux Engineer at Penguin Computing. As head of this year's Ultimate Linux Box Project, he was responsible for system design, construction and testing, and was involved in component selection. When not busy with the Ultimate Linux Box, he works on new product development and improving Linux support for Penguin hardware by contributing to the lm_sensors Project.
Matt Fulvio is a freelance industrial and architectural designer in the Bay Area. He can be found teaching mathematics at the San Francisco Institute of Architecture or at www.mattfulvio.com.
Philip Pokorny is the Director of Engineering for Penguin Computing. He worked with the power supply vendor and machine shop to get the power supply modified for water cooling. When he wasn't doing that, he was standing around watching and asking silly questions like a typical pointy-haired-boss.
Trevor Sherard, the craftsman of the case for the ULB, is a San Francisco Bay area freelance sculptor and woodworker. He can be contacted at www.woodentemple.com.
Don Marti is editor in chief of Linux Journal and wrote the text of the article.