Creative COW SIGN IN :: SPONSORS :: ADVERTISING :: ABOUT US :: CONTACT US :: FAQ
Creative COW's LinkedIn GroupCreative COW's Facebook PageCreative COW on TwitterCreative COW's Google+ PageCreative COW on YouTube
BLOGS:My COW BlogAdobe BlogMacWorldEditingTechnologyAfter EffectsFinal CutEntertainment

10Gb is Coming

COW Blogs : Steve Modica's Blog : 10Gb is Coming
Share on Facebook
You've all read that 10GbaseT is on the way. It's true. Very soon, you will be able to plug standard RJ45 connectors (just like on your Mac Book Pro) into your 10Gb Ethernet cards and switches. You'll be able to run CAT6A cable 100m (assuming the runs are clean runs) and have tons and tons of bandwidth between servers and clients. Who needs Fibre Channel anymore?!

But with the widespread migration to 10Gb, you may have a plumbing problem my friend.

Many years ago, I had the privilege of supporting three of the large animation studios in LA that were trying to use their new RAID5 arrays and run OC-3 and OC-12 right to their desktops. These two ATM standards were capable of 155Mbits and 622Mbits, respectively (this was before the days of Gigabit Ethernet). Everyone expected nirvana.

They didn't get nirvana. In fact, they found out right away that three clients ingesting media could very quickly "hang" their server. Within about 30 minutes it would slow to a crawl and sit there. They could not shut it down. Shutdown would hang. What was really happening? The machine had used all of its RAM collecting data and was unable to flush it quickly enough to their RAID. The machine was out of IO buffers and almost completely out of kernel memory. The "hang" was simply the machine doing everything it could to finish flushing all this unwritten data. We had to wait (and wait and wait).

Further, we discovered that with only three clients we could quickly start generating dropped packets. ATM had no flow control and so too many packets at once would result in dropped packets. Since the clients were very fast relative to the server, it didn't take more than a few to overwhelm it.

Similarly, as we all start to salivate over 10Gb to our Mac Books, iMacs and refrigerators, we should consider how we're going to deal with this massive plumbing problem.

First, you *will* need some form of back pressure. The server must be able to pause clients (and vice versa) or these new 300MB/sec flows are going to overwhelm all sorts of resources on the destination system.

Second, just because the network got faster, doesn't mean the disks did. In fact, now your users will have ample opportunity to do simple things like "drag and drop copies" that will use up a great deal of the resources on the server. A simple file copy over 10Gb at 300MB/sec bidirectional could overwhelm the real-time capabilities of a normal RAID. The solution lies in faster raids, SSDs and perhaps even 40Gb FCOE raids for the servers. (That's right, 40Gb FCOE raids)

So as you consider your 10Gb infrastructure upgrades, make sure you're working with an experienced vendor that knows about the pitfalls of "plumbing problems" and gets you setup with something that will work reliably and efficiently.

Posted by: Steve Modica on Oct 4, 2012 at 11:04:51 amComments (6) networking, storage
Reply   Like  
+2
Share on Facebook

Comments

Re: Blog: 10Gb is Coming
by Drew Lahat
Steve, I'm confused. You talk about 10GbE in future tense - "it's coming", "very soon"... yet 10GbE has been a reality for several years, and your company is one of the key players.

There are a number of installations out there and they seem to be working fine. And for both SAN and NAS setups, the bandwidth to the storage headend is rarely a multiple of the clients' aggregate bandwidth. You know where your bottlenecks are

The main problem with 10GbE is that it's so darn expensive. It was standardized 10 (!) years ago (and 10Gbase-T released in 2006), yet it's still an emerging market. Adoption rates of GigE and Fast Ethernet were much shorter in comparison.
Re: Blog: 10Gb is Coming
by Steve Modica
10Gb was around in 2003 (we wrote the first driver for Mac OS X at the time). The card was huge and it used single mode fiber.

"10Gb Ethernet" in the context most customers use is 10GbaseT. They want RJ45 connectors everywhere. You can just get that today with a few caveats (and yes, it's very expensive).

When I say "it's coming", I mean the day is coming when 10GbaseT will be part of the base IO for just about any server or desktop you buy. Along with that, the associated back end plumbing will be required like high port count cards and 40Gb uplinks.

Steve

Steve Modica
CTO, Small Tree Communications
Re: Blog: 10Gb is Coming
by Sareesh Sudhakaran
Doesn't it already exist?

http://www.wolfcrow.com - Workflow information and support for filmmakers, photographers, audiographers and videographers.
Re: Blog: 10Gb is Coming
by Steve Modica
[Sareesh Sudhakaran] "Doesn't it already exist?"

Yes and no. You can buy 10Gb cards and switches, but large port count cards for servers (4 or 6 port) are still SFP+ only. So there are certain elements that are still missing.

The biggest problem is that people will want to run 10GbaseT to the desktop and this will require either 40Gb cards in the server or large port count cards in the server.

Steve

Steve Modica
CTO, Small Tree Communications
Re: Blog: 10Gb is Coming
by Sareesh Sudhakaran
Thanks, Steve. I was planning to use a couple of 10 gigabit cards to transfer data from a private NAS for uncompressed streaming.

Won't two cards and an optical cable be enough? It's basically a media server on RAID.

http://www.wolfcrow.com - Workflow information and support for filmmakers, photographers, audiographers and videographers.
Re: Blog: 10Gb is Coming
by Steve Modica
[Sareesh Sudhakaran] "Won't two cards and an optical cable be enough? It's basically a media server on RAID."

That should work fine assuming the storage is fast enough. I've seen many people set that up only to discover that the destination storage (the internal drive on the second station) only goes 40MB/sec. Quite a disappointment!

Steve

Steve Modica
CTO, Small Tree Communications


Steve Modica's place to pontificate.
Blog Feed  RSS


FORUMSTUTORIALSFEATURESVIDEOSPODCASTSEVENTSSERVICESNEWSLETTERNEWSBLOGS

Creative COW LinkedIn Group Creative COW Facebook Page Creative COW on Twitter
© 2014 CreativeCOW.net All rights are reserved. - Privacy Policy

[Top]