BLOGS: My COW Blog Adobe Blog Editing Technology After Effects Final Cut Entertainment

Disk I/O Speeds, RAID levels and Spindle Count

COW Blogs : Tim Jones' Archive Bits Blog : Disk I/O Speeds, RAID levels and Spindle Count
In the world of storage, many claims are made about performance that may or may not be defensible. In this adventure, I'm going to visit the technical side of disk performance - how it relates to data types, disk RPM speed, RAID levels and spindle count (number of disks). Note that the information presented here is related to spinning disks, not SSD or PCIe memory disks. However, some of these thoughts apply to SSD-based arrays as well.

When you look at the performance claims for a spinning disk drive, there are three principle numbers that you need to be aware of:

RPM Speed -
This defines how fast the platters spin within the drive and affect how long it takes the heads to locate a block of data. This is referred to as the "seek time". It also affects the actual read and write speeds (versus cached read and write speeds). Modern drives are available in 4800, 5400, 7200, 10K and 15K speeds. As you would expect, the higher RPM drives are both faster and more expensive than the lower RPM drives. You will also find that as RPM values go up, maximum drive capacities come down, power requirements go up, heat produced increases and noise levels go up.

Cache Size -
This is an area of memory built into the disk drive that receives the data from your system. The larger this cache memory is, the more data that can be written to the drive at your systems I/O bus speed. The actual performance when reading from or writing to the disk's physical platters is much lower than the cache performance.

Interface Speed -
Modern drives are usually 3Gb SATA, 3Gb SAS, 6GB SAS, 8Gb FC, and the newest 12Gb SAS and 16Gb FC. Do not confuse these speeds with secondary connectivity interface speeds like USB, FW, or Thunderbolt.

In a perfect world, we would all be able to use 15K RPM drives that have 128MB Caches and attach via the 16Gb Fibre Channel layer, but the reality is that the cost, capacity points and availability place these drives into the realm of very high end IT environments rather than in our desktop systems.

On the other hand, when you look at an average 4TB, 3.5" drive, it will be 7200 RPM, offer a 64MB cache, and connect via either SATA-2 or SAS-2 connections. On average, one of these drives can burst read and write at around 400-500MB/sec. However, burst numbers are not going to apply when you are working with media files that are 256MB+ in size. These files exceed the capacity of the disk's cache and the result is much lower real read and write speeds of around 110MB/sec. This number is most often called the "sustained throughput rate" and that's the number that you need to take into consideration when you are working with video and audio files.

How do we make the best of what we've got (or can afford)?

When you buy a sailboat, more sails means more speed. When you buy a car, more horses means more speed. When you buy a computer, more processor cores means more speed. When you buy or build disk storage, the same adage applies - more disks mean more speed to a point.

Above, I mentioned that an average modern drive can sustain around 110MB/sec. When we add disks and configure them as an array of disks, we quickly discover that this speed is nearly additive up to the performance of the primary I/O interface being used.

For example - using 4, 7200 RPM drives in a 6Gb SAS chassis connected directly or via a Thunderbolt to PCIe chassis with the disks configured as a striped, or RAID 0 array you could expect an aggregate, or additive throughput of around 550MB/sec. However, that same 4 drive array attached via USB 3 would not deliver anywhere near that throughput because of the limitations of USB 3. In that case, you would be limited to USB 3's limit of 250MB/sec. In that case, you would weigh the value of 4 spindles from a capacity standpoint rather than a performance standpoint.

If you take the SAS example further and expand that array to 16 drives, you can safely expect the sustained read and write performance to approach 1.6GB/sec. In fact, using proper disk chassis with 6Gb SAS 2 connectivity, you could expand that array out to a total of 64 drives using 4 SAS channels and achieve a potential of 2.4GB/sec or using 8, 6Gb SAS 2 channels for almost 4.8GB/sec. Note that I'm discussing big-B Gigabytes per second here, not little-b Gigabits.

While there are specialized systems that can be had, there aren't any normal OS X or Windows systems that have the bus bandwidth to support that higher number in a sustained manner because of the current limitations of bus transfer speeds.

What about buying or building for capacity?

As I mentioned above, when you consider a multi-spindle disk solution, sometimes it's more about the capacity than the performance. When you start adding disks in an array, in addition to possible performance increases, the secondary gain is in the ultimate capacity of the array. This capacity increase is valid regardless of the secondary I/O connection. As I mentioned, that 4 drive array attached via USB 3 would be limited to 250MB/sec, but you would still be able to create a single volume of 16TB if you were using 4TB disks. Therefore, with the proper disk connectivity, you could connect a 64 disk array via USB 3 for a potential capacity point of 256TB of storage capacity, but your throughput would still be limited to 250MB/sec.

How do the other RAID levels play into this?

RAID (Redundant Array of Independent Disks) levels define how the separate disks of an array are interconnected, or assembled in the resulting single volume. The predominantly used RAID levels are:

RAID 0 - Striping
Used for performance and capacity. In this configuration, the data is segmented and striped across all of the disks in the array. This spreads the block writes and reads across all of the disks resulting in an additive capacity of n x single drive capacity and performance of n x single drive performance up to the performance ability of the physical connection of the array. This configuration offers no redundancy of data.

RAID 1 - Mirroring
This configuration is used for the ultimate redundancy. Since all data is written to both drives, the loss of either drive has no effect on the availability of the data on the array nor the performance of the array when reading. There can be a small write performance penalty depending on the RAID controller being used. The capacity nor performance of a single drive is not otherwise affected.

RAID 10 - A stripe of Mirrored drives
This configuration provides both redundancy and performance equivalent to n/2 x capacity and performance of a single disk. It is the most expensive mechanism for providing both capacity and performance since the you must use 2 times the number of disks required to achieve your required capacity point.

RAID 5 - Parity using 1 drive
This configuration allows for a single drive failure using a striped parity algorithm that uses the equivalent of one drive's capacity for ECC and recovery code. RAID 5 will allow for any single disk in the array to fail with no loss of data. However, the loss of a drive will result in a dramatically reduced performance when writing new data to the array volume. This configurations capacity and performance are both based on (n - 1) x single disk capacity or performance. There is a slight overhead on writes depending on the RAID controller being used.

RAID 6 - Parity using 2 drives
This configuration allows for 2 drives to fail using a striped parity algorithm that uses the equivalent of two drive's capacity for ECC and recovery code. RAID 6 will allow for any two disks in the array to fail with no loss of data. However, as with RAID 5, the loss of either one or two drives will result in a dramatically reduced performance when writing new data to the array volume. This configurations capacity and performance are both based on (n - 2) x single disk capacity or performance. There is a slight overhead on writes depending on the RAID controller being used.

RAID 50 or RAID 60
These are combinations of RAID 5/6 with RAID 0. They should be referred to as RAID five zero or RAID six zero rather than RAID fifty or RAID sixty. These configurations are usually implemented when you are working with a very large number of disks and a high capacity requirement. For example, you can connect 4, 8 drive units that are each configured as RAID 5 and then stripe these volumes together to create a much larger volume. Since each individual 8 drive array is RAID 5, you can lose 1 drive in each 8 drive set with no data loss.

How should data types play into your decision?

Since you are reading this here, you are most likely creating and accessing data that is very large - in the 100's of MB per file to multiple GBs per file range as video and audio files are prone to be. As mentioned above, this type of data very quickly eliminates the majority of the caching that is offered by a single disk. When you look at the vendors who sell video-oriented array solutions - vendors like Facilis, Maxx Digital, Small Tree and others - they generally take this fact into consideration and their performance numbers are dead-on for what you are doing. It's when you start looking into non-specialized arrays that are sold more to IT installations that you need to be wary. When these vendors are claiming I/O performance numbers that are 3 to 4 times the capabilities of a single disk, they are basing the numbers on smaller file I/O such as database read and writes or email file access. While these claims aren't wrong, they don't apply for your data.

My recommendations for configuration are as follows:

Capacity and Performance with an alternate backup solution:
Use RAID 0 (striping) as you will get the highest capacity and performance for any given drive type.

Capacity and Performance with long term storage in mind:
Use RAID 5 if you're using enterprise-class drives or RAID-6 if you're using consumer-class drives.

Protection of data only:
Use RAID 1 (mirroring) as this allows you to lose a drive with no affect on data availability.

Combination of performance / capacity with expandability:
Use RAID 50 or RAID 60. As you add new RAID 5 or RAID 6 chassis, you can add them to your volume stripe (RAID 0) and extend the filesystem.

Last Comments

If you are using RAID configurations 0, 5 or 6, I urge you to have a backup plan in place - whether to another, similarly sized array or to tape. While RAID is good for disk failure, it does nothing to safeguard your data from deletions or overwrite.

Don't waste money on a Fusion drive for an OS X system unless you're using it as your OS drive. It won't improve throughput for your media files.

Posted by: Tim Jones on Feb 4, 2014 at 12:01:17 pmComments (5) DISK, RAID


Re: Disk I/O Speeds, RAID levels and Spindle Count
by Erik Lindahl
It could be that some USB-3 raids have cheap controllers or something cause you can easily push more than 450 MB/s over the interface.
@Erik Lindahl
by Tim Jones
Good info, Erik, but that appears to be a very specific Windows-based test environment. Additionally, they don't list what their testing methodology was. In the Bare Feats article, they're using SSD, not HDD, so the results are going to be different (as I mention, this article was aimed at HDDs).

The article also references the FirmTek miniSwap/U3 which is using a hybrid USB chipset as being the big winner - FirmTek even touts:
"The miniSwap/U3 is a revolutionary new product designed to meet the needs of computer professionals on the go. This new, bus-powered, data storage device offers performance as high as 440MB/s, which is more than twice the speed of typical USB 3.0 enclosures."

My suspicion is that they are using a huge cache which would explain why their USB-3 empty enclosure is more than 3x the price of most empty USB-3 drive enclosures (ref: USB-3 2.5" enclosures at Amazon). However, it appears that the miniSwap/U3 enclosure is not easy to find. Even ordering it from their site told me that it was backordered.

If you examine the performance listings in that same article for the other USB-3 devices, they were all in the 250MB/sec range.

All I can tell you is that our current USB 3 samples using HDDs don't achieve performance above ~250MB/sec under OS X (10.8 or 10.9) as RAID 0 or under Windows 7 Pro as a Dynamic Striped volume.

Please don't misunderstand - 250MB/sec is still quite useable for portable, media storage, my point in the article is to describe the most usual considerations with HDD-based devices.

As with any technology, YMMV.

Tim Jones
CTO - TOLIS Group, Inc.
BRU ... because it's the RESTORE that matters!
Re: Blog: Disk I/O Speeds, RAID levels and Spindle Count
by Erik Lindahl
Great read. I'm however wondering what you base your "USB-3 has a limit of 250 MB/s" quote on. Modern USB-3 drives (be it SSD's or multiple HDD's) can easily run faster than 500 MB/s.
@Erik Lindahl
by Tim Jones
While the USB-3 interface spec claims 5Gb/sec, the reality is that every USB-3 RAID device that is currently available claims somewhere around 250MB/sec. Our lab tests on 4 different 4 drive units using both 7200RPM 3.5" HDDs and 1TB Samsung SSDs agree with those numbers. The best we saw was when reading the SSDs in RAID-0 mode and that was only 267MB/sec.

Three of the tested units also offered eSATA and we were able to achieve much more speed - with the SSD's we pegged 500MB/sec consistently.

Since the units and the drives and formatting were unchanged between the USB and eSATA tests, the bottleneck is definitely somewhere in the USB subsystem.

Tim Jones
CTO - TOLIS Group, Inc.
BRU ... because it's the RESTORE that matters!
Re: Disk I/O Speeds, RAID levels and Spindle Count
by Tim Jones
Update -

I've gotten email on this asking about RAID levels 3 and 4. I'll just state that because of the huh speed of modern disks (compared to 1993 performance), these dedicated parity layouts really don't bring anything to the mix so far as modern RAID 5 or 6 striped parity solutions are compared.

Tim Jones
CTO - TOLIS Group, Inc.
BRU ... because it's the RESTORE that matters!

Holy COW! It's a blog dedicated to the world of storage, backup and archiving of M&E data.
© 2020 All Rights Reserved