How Many Disks Are Too Many for a Linux System?

The answer relies much more on the hardware than on the operating system.

With recent advances in network bandwidth, resources and facilities are becoming centralized once again. The trend of departmental computers and services is fading as larger, more centralized services appear. In regards to this centralization, one of the questions most IT managers ask is “What size server is really needed?” In this article, we analyze how large a server is needed for file services and how many disks a typical Linux system can support. We assume we are dealing with a typical Intel architecture solution and that one or more high speed network connections is available to handle the load and requests from the client systems. If we assume two or more Gigabit network adaptors, this should provide enough bandwidth for most applications and data centers.

When attempting to see how many disks can fit into a system, we must analyze two parts of the problem. The first is the operating system itself. Linux has some limitations that prohibit it from hosting a million disk drives. Some of these limitations can be worked around, but others are hard limits. The second problem we face is hardware. The backplane, power requirements and physical construction of the computer physically restrict the number of disks that can be attached to a system.

Limitations of Commercially Available Computers

We start by looking at the hardware limitations of three systems commercially available today. We chose low-end, mid-range and high-end systems: an HP Presario 4400US, a Dell PowerEdge and an IBM zSeries, respectively.

Low-end Computer: The HP Presario lists for $549 and has one extra IDE drive slot for expansion and two additional PCI slots for disk controller expansion. Because the extra disks are not able to pull enough power from the system power supply, the disks need to be attached to a separate chassis and power supply. If we continue with the HP option of the HP Surestore Disk System 2300, we can get 14 disks per disk controller. This will give us a total of 30 disks, about 2TB of storage (assuming 73GB per disk). Each of these disks will yield about 11MB/sec across the 160MB/sec communication channel. We could go with a more robust fibre channel storage solution, the HP Surestore Disk Array XP1024. It allows for 1,024 disks or about 74TB storage per controller. Unfortunately, the system bus on our Presario 4400 only goes to 100MHz, while the Surestore XP1024 runs at 2GB/sec. The system we selected would not be able to handle such a high-end disk system, thus limiting us to the Ultra160 technology and 30 disks.

Mid-range Computer: If we decide to raise our expectations and look at a Dell PowerEdge 2650, which lists for $2,199, we can attach significantly more storage. The internal controller supports five SCSI disks and has the option of three expansion slots, one at 133Hz and two at 100MHz. The motherboard backplane runs at 400MHz, so the embedded SCSI controller can operate much faster than the expansion controllers. By using the PowerVault 220S/221S SCSI Disk Storage Unit, we can attach 47 disks, three 14-unit PowerVaults and five internal disks, for a total storage size of 3.4TB. We also can expand the memory of this system to 6GB of RAM, which would handle the disk operations much better than the 512MB limitation of the Presario 4400. On this system, we also could go for a more robust fibre channel storage solution, the Dell/EMC FC4700-2. This system allows 120 drives per fibre channel, or 360 disks yielding 63TB of storage on a single system. Since this system transfers data at 360MB/sec, the backplane running at 133MHz could easily handle the operations, but we are reaching the limits for the 100MHz backplane.

High-end Computer: At the very high end, we could use the IBM zSeries 900, which supports up to 64GB of memory and multiple fibre channel connectors. There really isn't a limit to the number of disks that can be supported through this fibre channel configuration, but connection to a single disk subsystem is limited to 224 drives on a single instance (16TB per subsystem). Direct SCSI is not supported, but a fibre to SCSI interchange is available.

Hardware Summary

In summary, on the low end, the hardware limitations begin with the backplane support of multiple controllers limiting the system to 30 disks. The mid-range system tops out at 360 disks, while the high-end machine does not have a practical upper limit. All of these systems could utilize storage area networks or network attached storage to increase their limits, but initially we wanted to analyze the hardware limitations imposed by the architecture for direct attached storage.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Re: Please consider a rewrite

Anonymous's picture

This could have been a very interesting article had it not been implemented so poorly. Frankly the article should be rewritten.

I often found it difficult to follow the article, particularly in the paragraph regarding the low end system.

I think the judicious use of charts, tables, diagrams, clearly explained and stated math, and a thorough confirmation of facts and numbers could make an article of this nature extremely valuable.

As it stands this article is a chore to read and does not offer nearly the value it could. This is very interesting subject matter and I would definitely like to read this authors thoughts again though once the article has been rewritten.

Re: Please consider a rewrite

PShuff's picture

excellent recommendations. I wanted to use some charts and diagrams but could not figure out how to include these with my article. I will work with the publisher and learn how to include graphics with future submissions. Chalk this up to inexperience on how to submit articles.

pat

One inode per 512B... huh??

Olive's picture

Each inode describes a 512-byte of data on the disk. If we have a 20GB disk, we will need about 40,000 inodes to describe the disk.

I'm sorry, where did this assumption come from? First of all, a disk block in Linux systems is usually 4KiB, and second it has always been possible to specify the ratio between inodes and disk blocks when you create a filesystem. Newer (as to stable kernels) filesystems like ReiserFS and XFS don't even have static inode pre-allocation upon filesystem creation, so this kind of information is really misleading.

Re: How Many Disks Are Too Many for a Linux System?

Anonymous's picture

Linux Journal really should be embarrassed to publish such an incompetently-written article. Next time, try to find someone who knows something about the material they intend to discuss (in this case, file system and storage).

Re: How Many Disks Are Too Many for a Linux System?

PShuff's picture

Being a graduate student, my intent is to publish articles to improve my ability to write. I am encouraged to do this by my advisors and faculty that I work with. Linux Journal has created a web publication area so that professionals can exchange ideas and improve their ability to communicate. I realize that my article was not worthy of print in their magazine for a variety of reasons. I will continue to write and share my ideas with other professionals. If anyone has constructive criticism on how I can improve my style, please publish it here for everyone to read or send it to my university email shuff@tamu.edu.

pat

Re: How Many Disks Are Too Many for a Linux System?

Anonymous's picture

Excellent reply!

A gentleman reply indeed!

Those who don't do, don't err ...

Re: How Many Disks Are Too Many for a Linux System?

etbe's picture

One issue that seems to be missed is the PCI bandwidth. Most machines on the market have only 32bit PCI and only a single PCI bus. If you have a 33MHz card in a PCI bus then the entire bus runs at 33MHz. This limits machines with a single PCI bus to ~130MB/s of total bandwith (for disk, network, etc)!

Becase of this most companies that make disk controllers don't have much interest in 66MHz or 64bit hardware, so if you get a machine with multiple PCI busses that supports 66MHz 64bit PCI, then you will have a limited range of hardware to choose from (which means you may be more likely to encounter bugs).

Also where did the number 11MB/s per disk come from? It seems that any modern drive can sustain 30MB/s or more for linear reads, but if you do random seeks of small data blocks then the transfer rate can go below 1MB/s. I think that when comparing transfer rates the speed for linear reads is the best number to use (also it's interesting to note that most hardware RAID solutions are the bottleneck for bulk IO - in many cases it's possible to prove that both the system bus and the drives are capable of greater speed than the RAID hardware delivers).

Also I feel that omitting any mention of IDE RAID products such as those from 3ware was a mistake. My experience with 3ware products and that of friends and colleagues has been very positive. IDE drives are cheaper than SCSI drives, and using the 3ware products you can cheaply and easily get 8 drives in a machine.

3ware has announced plans to support 12 serial-ATA drives on a single controller. 12 * 320G ATA drives will give 3.5TB of RAID-5 storage (or 3.2TB or RAID-5 with a hot spare disk), which will compete very well against more expensive SCSI based solutions.

Russell Coker

Re: How Many Disks Are Too Many for a Linux System?

PShuff's picture

good point about the 3ware device. We have one in out lab that is based on SCSI and it works very well. I should have addressed the IDE connectivity as well but mistakenly did not. Thanks for the input.

The 11 MB/s number comes from the average latency of a disk. If a disk can do an average seek of 10-15ms, this correlates to about 10-12 MB/sec. Most disks can perform burst rates substantially higher but not sustained for the general case. I will attempt to clearly state this in future articles.

pat

Wrong block size

Anonymous's picture

From mke2fs man page:

-b block-size

Specify the size of blocks in bytes. Valid block size vales are 1024, 2048 and 4096 bytes per block.

In practice -b is is not used and ext2/3 uses 4 KB block size. Reiserfs has always 4 KB block size.

So assumption of 512 byte block size is incorrect.

Re: Wrong block size

PShuff's picture

I stand corrected. The block size implementation is vastly different from system to system. All of the text books that I did my research from used a base of 512 as the file system foundation. After further research I have found that the many file systems allocate at a larger chunk and do fragment management as small as 512 bytes within these larger allocations.

pat

Re: How Many Disks Are Too Many for a Linux System?

Anonymous's picture

Why didn't you address IDE disks? They're far cheaper than SCSI and with controllers like 3Ware available, you can get a LOT of bang for your buck. There are 160GB disks available (that's more than twice the size of the 72GB disks you dealt with).

Re: How Many Disks Are Too Many for a Linux System?

Anonymous's picture

size doesnt always matter....

Re: How Many Disks Are Too Many for a Linux System?

PShuff's picture

The main reason I only addressed SCSI and not IDE is that you can get more SCSI disks per controller than you can with IDE. The goal was to get more disks on a system. I should have made this assumption clear at the beginning of the article. Thanks for the feedback.

pat

Re: How Many Disks Are Too Many for a Linux System?

Anonymous's picture

IDE disks are not very reliable. It is very common to have many disks fail at the same time, especially if they are large. You would need about 5 times as many disks as with SCSI to get reasonable reliability

Re: How Many Disks Are Too Many for a Linux System?

Anonymous's picture

They were addressed where should have been... entry level system.

The major problems with IDE systems

1) Low drive count on IDE RAID systems (Promise's ATA133 solution tops out at 8 drives)

2) Slow drives (those 160 GB drives you mention spin at a whopping 5400 RPM vs the 15000 RPM that's common in enterprise SCSI or FC systems)

3) rack density... offshoot of 1) none of these systems is designed for the datacenter where a couple of square feet makes an enormous cost/benefit difference. It may actually be cheaper to use more expensive SCSI or fibre solutions than use IDE simply because of that.

IDE is great for the desktop and entry level server but frankly sucks with serious use. Maybe serial ATA will solve that, we'll have to see.

Re: How Many Disks Are Too Many for a Linux System?

etbe's picture

For anything up to 1TB IDE is the best option at the moment. IDE drives are getting bigger all the time and IDE RAID systems are handling more drives (expect support for 16+ S-ATA drives in the near future), so soon IDE will be good for >4TB.

If you read the mailing lists about IDE RAID then you'll notice that the three main problems are PCI bus bandwidth (running software RAID-0 across two hardware RAID cards in different buses is one solution to this), the 2TB limit for block devices, and the number of drives that can physically fit into a case.

All three of these problems apply to SCSI in the same was as to IDE.

Also 15K rpm drives run very hot, this compounds the problems in finding a suitable case for a large number of drives. I generally purchase the drives with the slowest rotational speed to gain extra reliability from cooler drives. Reliability is more important than a small amount of speed (even the slowest drives today are very fast).

Russell Coker

Re: How Many Disks Are Too Many for a Linux System?

Anonymous's picture

1. 3Ware has just introduced a 12-channel IDE RAID. And multiple people have started selling FC storage boxes with IDE drives inside, making high-disk-count IDE configurations eminently feasible.

2. 7200 rpm IDE drives are only about half as fast as the fastest FC/SCSI drives (15Krpm) for small, random accesses - but they cost less than 1/7th as much per GB (under $1.50/GB for an 80 GB IDE drive vs. well over $10/GB for a 73 GB SCSI/FC drive). So using twice as many IDE drives to obtain performance comparable to SCSI/FC drives still costs less than 1/3 as much (and likely doesn't require noticeably more power or cooling: fast drives run hotter). If access patterns are more sequential in nature the IDE drives look even better, as their sequential transfer rates are considerably better than half that of the high-end drives. And if the prime requirement is simple storage space, the IDE drives enjoy the full factor-of-7 cost advantage.

3. Worst case (small random access performance) is that you need twice as many IDE disks as you would high-end disks. In some cases this could be significant (even though that larger disk farm would cost less than 1/3 as much), but since rack-mounted hot-swap 2U boxes exist that can hold 9 IDE disks apiece (i.e., 4.5 disks/U) you can still get quite reasonable rack densities.

And the one performance advantage that FC/SCSI disks used to have in write-intensive or heavily-queued environments is in danger of disappearing: IBM's new Deskstar IDE series has implemented tagged command queuing.

IDE doesn't suck for serious use: it's just not yet well known in that area. Don't expect that situation to exist for much longer.

- bill

Re: How Many Disks Are Too Many for a Linux System?

Anonymous's picture

Splendid !!!

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState