Ultimate Linux Box 2005

Some people wanted us to build a big powerful SMP system. Some people wanted us to build a silent machine that would be good for audio. So we did both.

Let's just call 2005 the year of power management. Processor vendors made a big deal out of whitepapers about saving watts, and we heard a lot about power management at LinuxWorld Conference and Expo in February.

Did the industry start caring about global warming? Do IT CEOs want to eat swordfish more often, so they have to reduce the mercury emissions of power plants? Not quite. Today's server systems are packing more and hotter processors closer together, and customers' air-conditioning systems aren't ready for the strain. NASA had to install water cooling for its 10,240-processor Columbia cluster, as we showed in our January issue.

Every watt-hour you can save is heat that the customer doesn't have to deal with—3.6kJ, or 3.4BTU to be precise. With data centers full of blade servers, and 1U systems sporting as many as four processors, all that heat really adds up.

The Linux desktop greedily devours the scraps from the multibillion dollar Linux server market, and power consumption matters to us on the desktop too. Fans are loud. If you have better power management on your processors, they produce less heat, and you can run fewer fans or run the fans you do have more quietly. We took a different approach to fans, as you'll see later on.

Finally, of course, power matters on the laptop and on portable devices because of battery life. We'll leave the specifics of tweaking for maximum off-AC time to future articles.

Motherboard: the Heart of the System

We like Tyan motherboards, and companies that build custom Linux systems do too. The four-Opteron Tyan Thunder K8QS Pro came out just a little too late to make it into last year's Ultimate Linux Box. It's based on an AMD 8000 series chipset. When we say “chipset”, we mean a slightly different combination of hardware from an Intel-based system, though. The AMD64 way is to have an onboard memory controller per processor, give each processor its own bank of memory and link them with HyperTransport. Your AMD64 “SMP” box is really a mini-NUMA, and the “chipset” doesn't include the memory controller.

Last year, we used a Celestica A8440 bare-bones rackmount system as the basis for the Ultimate Linux box. Although starting with pre-integrated chassis and power supplies can be a great time saver, we realized that last year's box was on the loud side. This year, going back to our usual plan lets us pick everything else just the way we want it.

The K8QS Pro has two PCI-X busses, A and B. B is dedicated to two 133MHz-capable PCI-X slots, and A offers two PCI-X slots maxing out at 66MHz and one regular PCI slot. Onboard networking is two Broadcom BCM5704C Gigabit Ethernet interfaces, also on bus A.

There are all the regular PC ports, of which we're using the USB. SCSI and serial ATA are options, which you might want to keep in mind if you're planning to move this board into a more conventional server role when you're building your next Ultimate Linux Box.

Into this mighty board we plugged four of the best of the Opteron processors available at the time—the 846 HE, clocked at 2.0GHz and offering 1MB of L2 cache. See the sidebar for what became available while we were testing the system. We maxed out the system's main memory at 32GB.

Unfortunately for case shoppers, this board is SSI MEB size—13"x16" or 330.2x406.4mm. Not a problem for us because we're using a custom case this year, but the size does limit your case options.

When we're picking out a case for any custom-built system, Ultimate or otherwise, we usually get one that's quite a bit larger than what a big vendor would use for a comparable system. Smaller cases require less material and they're cheaper for vendors to ship, but since we like to tweak things, we get a case with more room to add devices and more room to work inside.

Storage

In order to have a completely silent system, you need to move storage outside the box. Options for doing this have changed a lot since the days when you had a choice between NFS and external SCSI enclosures connected by a 3-meter cable.

Today, you can make your drives go away using USB, FireWire, SCSI of course, Fibre Channel or the new ATA over Ethernet, which we covered in the June 2005 issue. A separate storage enclosure is no longer only an enterprise server-room thing.

Another option is simply to boot over the network and mount your storage via NFS. Since Penguin works with enterprise server-room hardware, and Fibre Channel does deliver impressive benchmark results, we went with it; an nStor 4320F Fibre Channel RAID enclosure, with Hitachi 18GB drives for the OS and larger Seagate drives for more storage.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Case with 2 - Motherboards?

Anonymous's picture

Anyone know where I can get a case that will hold at least two motherboards having expansion room for each, external cooling, and hot swap PSU, drives, CF support, etc? I see many doing case mods but nobody consolidating multiple systems in one CPU. I have seen a manufacturer with a commercial cabinet for as many as eight motherboards but it was not tailored to anything other than the commercial environment.

Each motherboard in the dual system could run VMware Workstation with one motherboard hosting failover firewall(dual), radius, LDAP, etc, and the second running multiple LAN clients. The option to one CPU with multiple boards is one high end CPU running VMware ESX or GSX which cost quite a bit more so the trade off is software vs hardware...but a box developed for ESX or GSX is going to cost far more than a dual motherboard system tucked into one CPU cabinet given the high end hardware solution will require 8-12GB of RAM on one board. The boards could each be populated with 4GB of RAM and dual core CPU. Your cost for _one_user_ with such a setup would be the cost for the systems and two copies of VMware workstation. I would think you could share the components of cooling and power at a minimum and there may be more. Why do this? So many cabinets...not enough processing power in one cabinet in residential systems but at this time they are cheaper to build.

Great modding article.

Step one?

Porges's picture

You guys seem to have skipped something on the way to building your own Ultimate Linux Box... where to get the money!

To build a comparable system would likely need the backing of a few (or many) sponsors, would it not?

Boot drive options

philiph's picture

When I built my custom wan router (August 2004 Linux Journal article 'Point to Point Linux'), I went through a similar decision about the boot drive. While that system is quite a bit different, the goals are the same: reliability and minimized heat output. I chose a 256mb flash ide drive from MagicRAM because we had some problems with the CF-IDE adapters we tried. The MagicRAM device plugs right in to an IDE connector as a small dongle. On the downside, it is more expensive than CF/IDE solution.

I also was able to shrink a stock Fedora Core 1 distro down to a size that ran comfortably off the 256mb disk. That of course requires throwing out all graphical components and other extra stuff, like man pages and localization. Still, it is completely possible to get a good working Fedora environment in that space. That may be a more robust solution since it's more 'standalone' than having to depend on network booting, etc.

One thing you have to be careful about in that situation is moving all your log files and other temporary data to a tmpfs. Flash devices are great but individual memory bits can only handle a few hundred thousand writes before dying. Thus its important to minimize continuous writes like log files.

Ultimate Linux Box

Romuald's picture

This is a wonderful, insightful and uplifting case of a revealing infrastructure implementation of Open-Source & Free Software.
Keep up these useful pieces.
Romuald

I was wondering about the Res

Anonymous's picture

I was wondering about the Reserator 1, how it was modified to allow cooling by natural convection. I understand they took out the pump and attached a pipe to the inlet port, which then went up through the reservoir. Makes perfect sense, thermodynamically, and I wonder why Zalman didn't think of this in the first place (even without removing the pump). All you get now, without the pipe up, is a lot of turbulence in the reservoir (or is that good?)

Anyway, anybody have any idea how hard it is to make that modification? Did it require some serious custom metalworking just like they did to the power supply, or is this something an average DIY can do in his garage?

Also, Zalman has the Reserator 2 out, which has coolant hose connectors that seal when you take them apart, so you don't have to mess with clamps anymore. Anybody knows if this restricts the flow of coolant seriously enough to prevent effective natural convection?

Very nice case

jsroy's picture

What really caugth my eye when I looked at the Linux Jounal is the incredibly cool case for this machine. It then made me think of my lame desk, and that my machine deserved better. Finally, it made me want to build one (or maybe even commission one) for myself.

So I was woundering if anybody would know how or where I could get plans to build a cool custom case?

Also, I have tryed to look into the "www.woodentemple.com" site to look at the craftmans works, but can not seem to get there. Is the site not built yet?

Thanks

More nice cases

Vaino Husgafvel's picture

Go to www.mini-itx.com and at right You see several projects.
Click any of them and enjoy and be INSPIRED !

Best Regards

Vaino

Very nice case

jsroy's picture

What really caugth my eye when I looked at the Linux Jounal is the incredibly cool case for this machine. It then made me think of my lame desk, and that my machine deserved better. Finally, it made me want to build one (or maybe even commission one) for myself.

So I was woundering if anybody would know how or where I could get plans to build a cool custom case?

Also, I have tryed to look into the "www.woodentemple.com" site to look at the craftmans works, but can not seem to get there. Is the site not built yet?

Thanks

Touchstream LP Keyboard

chrisg's picture

Curious about this device, I looked it up on Google (since you didn't include it in the parts list, I didn't know the manufacturer). I found this interesting note on the home page at www.fingerworks.com:

Important note!

FingerWorks has ceased operations as a business.

I was actually hoping to try them out...

I have one...

Anonymous's picture

While the interface is kind of novel, and gesturing to replace the mouse is nice, it wasn't the best $400 that we ever spent. I can touch type fine but my speed never improved on it to the point that I'd consider shelving the regular keyboard. After a couple weeks of having both a normal keyboard and the gestureboard on the desk at the same time, it moved up to the shelf, where it stayed, except when I try and fob it off on friends who invariably return it.

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState