Ok, I got a bunch of emails yesterday asking to explain exactly how an Ethernet SAN works. I've done a full article on this in the past, but here's a quick primer on how it works.
1 - The SAN controller computer. In our case, a Mac Pro 12 Core machine with 32GB RAM.
2 - An Ethernet controller inside the SAN controller computer. In our case it's a Small Tree Communications Card.
3 - A RAID Host controller inside the SAN controller computer. In our case it's the Atto R680 SAS Host controller.
4 - A high speed ethernet switch. In our case, a Small Tree 24 port ethernet switch.
5 - A high speed media array. In our case, a brand spankin' new Small Tree 48TB ST RAID II. 16 drive chassis with 3TB drives.
6 - Client computers connected to the ethernet switch via Cat 5 / 6 cable. Mac Pro, iMac, Mac Mini, Laptops, anything with an ethernet port.
So in a nutshell, the high speed RAID connects directly to the SAN Controller computer.
The SAN Controller computer connects to the Ethernet Switch.
The Client computers connect to the Ethernet Switch.
Set up the Network settings correctly on the SAN Controller Computer and all Client computers.
Set up the File Sharing correctly on the SAN Controller Computer and all the Client computers.
Go into the Mac OS on the SAN Controller Computer and all the Client Computers and tune the heck out of them.
That's it in a nutshell. Mount the SAN to each client and start working.
There is zero control software needed to run the SAN. Just set it up, tweak it and start editing. Of course I'm not going to get 500MB/s to each client workstation like you might with Fibre Channel, but we get well over 100MB/s to each workstation allowing us to cut and view Apple ProRes HD files all day long across 14 workstations simultaneously.
Here's a diagram of what the system looks like in our shop. In our case, we've decided to keep one of the original 16TB Expansion chassis from our original SAN so we can use it as a direct connect to our Resolve system for RED / Alexa playback and to use as a "dump drive" for the big SAN when needed to clean up the RAID.
Most of you know that we're running an ethernet based SAN here at BCM. It's the Maxx Digital Final Share SAN which runs a combination of their drive arrays with Small Tree ethernet wizardry and some stout Atto Host Bus Adapters. We generally get around 100 - 120MB/s to the 7 workstations that are connected to the SAN. More than enough speed to cut using Apple's ProRes codec all day long.
At the moment we're cutting two feature documentaries (over 300 hours of material), 3 PBS series and a multitude of other projects all on the SAN simultaneously. It's been a very stout performer and when we need to do Uncompressed or 2k work, we have two local 8TB RAIDs directly connected to two workstations giving us 500 to 650MB/s. So for our needs, we've got everything set up to handle whatever comes in the door and need to allocate the uncompressed workstations as necessary.
Well at NAB 2010, Bob Zelin brought me over to the Maxx Digital Booth to show me 350MB/s coming off a single 8TB RAID connected via ethernet.
Now we're getting into Uncompressed HD territory. Albeit a single stream of Uncompressed HD, but that's perfectly fine for color grading or sound mixing and even editing. Not to mention serious multi-stream ProRes capabilities. Via Ethernet!
I knew this technology was going to get better as we moved along but I didn't expect a 250MB/s jump in just one year. In fact, the system could go faster but we need to wait for the drive manufacturers to catch up! Can you say Multi-stream Uncompressed HD and 2k via Ethernet on the horizon? All I can say is Wow.
Really looking forward to working with these new speeds once everything is ready for shipping. I'll update you all as details become available.
In December of '08 we installed a Maxx Digital Final Share SAN system consisting of a 16TB array that is shared to 6 workstations via high speed ethernet connectivity. You can read a full article I wrote on the installations at the Creative Cow website.
The primary purpose for this installation was to allow a shared editing environment for three feature length documentaries. We have in the neighborhood of 450 hours of footage (and growing) for all three and our first doc, Foul Water, Fiery Serpent is using around 100 or so hours. All footage is digitized at Apple ProRes 720p/60 via Apple's Final Cut Pro so we're using a very low bandwidth format for the edit.
We started really cutting on the project in March of '09 and as I have reported both on the Creative Cow website and here on my blog, it has for the most part, been a thing of beauty. I'm cutting on a Mac Pro / AJA Kona 3 workstation while my edit assist is cutting on a 21" iMac. We broke the doc into 9 segments to make for easier project management and to allow each of us to work on different segments simultaneously.
To give you an idea of the size of this project, we have between 2,800 and 3,600 raw video clips, over 100 music cuts, animations, graphics and voice tracks. So we're in the neighborhood of 4,000 to 4,500 media files for this project. That's as big a project as I've worked on yet. As I said, the SAN has worked great during the editing process.
However, in the past few months I believe we have found the limits of ethernet based SANs; playback of a large project timeline. In November we finally had a full 98 minute timeline cut of the entire documentary. I could not play the entire timeline without dropping frames. And not just once, it would drop frames multiple times, every time during playback of the timeline. Plenty of speed on the RAID (about 600MB/s or more), plenty of speed on the network (about 100MB/s) but for whatever reason, I was dropping frames throughout the timeline.
Now I don't believe the length of the timeline is an issue. In my testing I played a 30 minute episode of "Good Eats" in a continuous loop for 3 hours on multiple systems simultaneously. So the system can easily sustain a long playback cycle across multiple systems, let alone on a single Mac Pro workstation. No there has to be something else other than pure speed.
If you've read my blog entries you also know that we've been dealing for months with an ethernet port issue introduced by Apple with the latest Mac Pros that caused the network disconnect from my Mac. So this problem of dropped frames was thought to be part of the same issue. Over the Christmas Holidays, with help from Small Tree Electronics, we dealt with that issue finally by moving the SAN to the Snow Leopard operating system because Apple finally created a fix to the disconnect issue in the latest updates.
But our dropped frames remain. The SAN is running as fast as ever, but we're still dropping frames during playback of our 90 minute timelines. From what I can gather, we are the only facility running this large of a project off this type of ethernet SAN. All the other facilities are doing 30 minute or shorter programming and a lot of :30 to :60 spots. In our own shop, we have multiple workstations doing projects of 20 minutes or less with no problems.
How we're dealing with these dropped frames right now is to export a self contained movie to a local 8TB array that is connected directly to my Mac Pro. This is how I screen the film for the client or folks who come in for reviews. It's the only way I can play the film without it stopping.
So for whatever reason, it appears to be the sheer project size and the amount of files that the project has to access during playback of the timeline that seems to be the issue. It really shouldn't matter, but it does appear in our real world application, the system simply does not support playback of an extended timeline from a project with this many files.
It's a real shame because the system is performing incredibly well overall, we have two series being cut on it and I've been able to work with a 2nd editor simultaneously on editing the documentary. But if you can't play back your main timeline without dropping frames on a large project, well then the system is not made for all editing applications as I originally thought and was led to believe when I made the purchase.
So if you're working on shorter projects, episodic television, 1 hour projects and need to share media across multiple workstations, this is still a killer deal. Fibre Channel is still the only alternative and you can't come close to what this system does for the money. We will continue to use this system moving forward on most of our projects.
But for the long form stuff like these documentaries, I'm going to invest in a few more local 8 and 16TB arrays. The primary workstation for each documentary will have its own dedicated local storage and anything that needs to move across to other workstations, we'll push to the SAN. It'll make things a little less efficient for the really REALLY big projects, but I'll have the best of both worlds. Low cost SAN for 90% of our projects. High speed local RAID for the documentaries.
So would I still recommend an Ethernet SAN for you? Absolutely, but go in understanding the limits and make sure it's right for your application before you buy. I won't say that this system was a $20,000 mistake, but I would have spent my money a little differently 12 months ago had I known this system would be limited by the documentary. And we all know technology improves almost daily so with any luck, future improvements will allow this type of SAN to even support the really REALLY big projects in the future.
After some suggestions from colleagues, I mixed down the audio tracks (we had a total of 24) in the timeline and attempted a full timeline playback. We got 38 minutes through the timeline before it dropped frames, but it did not drop frames again. It was a 1 hour 18 minute timeline. So that's progress. Not exactly efficient since it took a while for the computer to do the mix, but it's an improvement.
Today's just been one of those days where the Final Share SAN isn't working right, the AJA Kona 3 isn't working right and FCP isn't working right either. Just as we're about to finish our first feature documentary.
It's one of those days where you wonder what the hell you invested all of this money for and if you should just switch over to something else....
To paraphrase a famous author, "I have seen the future of shared storage and his name is Ethernet."
Last month we invested in the new Final Share
system from MaxxDigital
and after some tweaking, we now have 16TB of shared storage supported a high definition workflow with 6 workstations all running Apple's ProRes HQ in high definition, both 720p and 1080i. And actually it's not "workstations" in the traditional sense of the word, since we're running ethernet, we can connect any Mac computer to the array.
So in our case, we have three Final Cut Pro desktop workstations and three iMac's all connected. In our testing today we configured the three FCP workstations to capture approx. 3 hours of 720 and 1080i ProRes HQ material each. As that was happening, all three iMacs were playing back 20+ minute clips in Quicktime Player in a loop. After all the capturing, we had all three FCP workstations set up with 90 minute timeline playing in a loop while the iMacs kept playing their clips. We left it all alone for several hours and all way still playing. 6 streams of high definition from one storage array and all via simple ethernet cable!
We plan to use the iMacs both to allow Producers to review footage immediately upon capture and also for Assistant Editors working on upcoming series. Once the footage is in the system, anybody can access it at any time and since it's not Fibre Channel, I don't have to invest in top of the line desktop editing systems for the assists.
Watch for a full article on this system coming up shortly, but wow, this thing really works and it's really affordable!
Ok, late news flash as this information is actually from MacWorld (what two months ago now?!?) but the folks at ATTO were able to crank up two MaxxDigital EVO HD SAS/SATA units to 1200MB/s. 1200!!! Good Golly Miss Molly that's some serious horsepower.
Ok, ok, ok, how did they pull this off since you can't connect two systems to the same card? Easy, two ATTO R-380 SAS/SATA cards in a Mac pro. Then connect a MaxxDigital EVO HD unit to each card. Stripe both EVO HD units as one large storage array and presto, 1200MB/s.
Hmmmmm, how many streams of video would THAT be? You could do the Brady Bunch Open times 5 at least I suppose. Would be fun to test that out with Uncompressed HD and see if there's any way to do multiple streams of HD in realtime with filters.
Here's a photo only a mama could love. A bunch of tech geeks standing next to the unit with the speed tests displayed on the screen.
Yet another reason I love these guys. They're products are really fast, they work and you get this incredibly fine looking support staff to help you out. Ok, maybe not fine looking.... sorta geeky looking support staff...... maybe in a creepy sort of way......... but they know their stuff......so just pretend you don't know what they look like.........maybe I should just shut up now.
So I just have to pass along this wonderful experience we've had and it really underscores the tremendous community we have here at Creative COW. For about a year I've been contemplating an upgrade to our facility to make it more efficient. When I started in 2001 I had one edit suite and hoped to add a 2nd one at a later date. In 2003 we expanded to that 2nd suite and I was eventually able to get a full sized rack unit to hold all my tape decks, my computer and some storage.
It worked pretty well though a bit combersome because there were no patch panels so to run anything to anywhere meant crawling behind the racks, pulling cables, figuring what went where, etc.. Then at the end of 2006 we suddenly needed a third suite which we quickly threw together. Now suddenly we were pulling decks out of the rack, walking them to the third suite, walking drives around, etc... It was starting to get really messy.
So I contacted a few local companies about assisting me in re-engineering the shop to be much more efficient. I can figure out what I need, but really wanted an engineer to come up with the best way to make everything here more efficient both in layout and operation. Didn't really find what I was looking for in terms of personnel and price. Enter Creative COW and fellow COW Leader, Bob Zelin.
It started with a simple email to Bob. Through reading his posts it was obvious that facility design and installation was one of his specialities. I simply asked if he would be interested in assisting me at all. To my surprise he said yes and immediately asked for a full equipment list and a general layout of our facility. What I loved about this process was I kept downplaying what we needed here and he kept insisting on some additional cables and connections that would make our lives easier.
Within about two weeks we had the first drawings and plans from Bob. Again, I was questioning some of his ideas because I thought it was overkill. Three video patch panels? Three audio patch panels? We're a simple, small shop, aren't we going too far here? I'm thinking one patch panel for each should do fine. But Bob, in his own gentle fashion, reminded me that it's easier to run cables once, than to install something and then come back in a few months and say " you know we should add some more cables runs here, there and the other room." It started to make sense, especially after the full drawings for the patch panels showed up and I could visually see what he was talking about.
Two months after we started discussions, we finalized the plans which included: two new full sized rack units; three video patch panels; three audio patch panels; new reference DA; a Gefen DVI/USB extender to allow one computer to move 55 feet from the suite to the rack unit; and I'm guessing about 1,500 feet of new audio / video / control cables. At the same time we decided to upgrade our storage with two new MaxxDigitial SAS/SATA 8TB arrays for the two main suites adding 16TB of new storage. I have to say, Bob's insight and advice during this entire process was invaluable and he really made the plan much better than anything I could have created on my own. In addition, he probably saved me a lot of money from the inevitable mistakes I would have made designing all of this on my own.
Almost immediately our new racks and shelves showed up in about 6 huge boxes. We replaced the small box that was to the left of the original rack with one of the new ones and already the place started looking better.
A few weeks later, the storage and patch panels showed up so we pulled out the original rack and set up the 2nd new rack into it's final position.
Now it's really starting to look good and I get a real surprise from Bob. He's going to personally come to Atlanta to do the patch panel installation. The original plan was for Bob to make up all the cables and ship them to us with instructions. But due to an opening in his schedule, he was able to make the trip himself. I've never met him personally so I'm thrilled, can't wait to see if he's as angry in person as he is on the forums!
He went ahead and shipped up the long run cables to run through the walls which my assistant, Aaron, and I ran prior to his arrival. A messy job to be sure, but I had designed drop wire chases into the walls when we moved into the office thinking they would come in handy one day.
I killed the power before allowing Aaron to play with live stuff. Never let kids play with wire cutters!!!
So today Bob Zelin in the flesh shows up and to my surprise, he's a heckuva nice guy! Well, it wasn't really a surprise, I had spoken to him several times on the phone and he's actually quite funny. A straight shooter to be sure, but a very funny guy. He's not here 5 minutes and he jumps right into work.
After watching Bob for about an hour I was really happy he had the time in his schedule to come up here because while I could have done the cableing myself, it would have probably taken me three days to do what he did in less than 8 hours. As with any installation, there were some last minute issues and questions that needed to be addressed. Moving some audio patch points, re-routing some cables, adding some new cables, etc... were all last minute issues we took care of as the day progressed. It really boggled my mind to see how much cable three edit suites and about 5 VTR's would require to make this shop more efficient.
Before we knew it, lunch time was upon us. It was time to initiate Bob Zelin to the wonders of the Nintendo Wii. It took a few frames, but before we knew it, Bob was throwing Strikes and Spares with the best of them. Check out the professional form on this throw!
It's looking good Ted.... staying away from the gutters.... a little spin to the left..... and it's a Strike!
About the only real tough discussion was when Bob tried to convince Aaron and I to abandon all component and SDI cabling to run everything via Composite video. After all the VHS look is in these days so why spend all this extra money to run these extra cables. Ok, no he really didn't do that, he accidentally forgot to make up the final component / SDI cables for our Kona boxes, but of course, being prepared for everything, he had plenty of video cables and connectors to make up these last few runs on the spot.
Less than 8 hours after Bob started, we had a fully re-engineered facility. Every audio and video output from every device in the shop, from the Konas to the VTRs, passes through the patch panels. We can route any signal, anywhere. All the RS-422 controls pass through their own patch panel allowing all three suites to take control of any VTR in the racks. We have almost 20TB of fast storage for all projects. It's absolutely awesome and something both Aaron and I are not used to. You just get so used to the "moving cable ballet" behind the rack that to have something so efficient is weird.
Look what I made!
I cannot believe what we started in a single room in 2001 has grown to something the really feels like professional Post House. The growth of my business and the work with Bob Zelin can all be traced back to the Creative Cow. Folks we have a tremendous resource at our disposal with thousands of incredible working professionals in all aspects of the creative production field. You know you get answers when you post questions on the forums. Just remember that those same people who answer your questions are also some great people to call upon and hire when you need high quality services. Just look at our transformation in just three months from a decent working facility to a real professional facility.
So Thank You Bob Zelin for all your help in re-designing our shop and Thank You to Ron and Kathlyn for starting such an awesome creative resource.
Another Satisfied Zelin Customer!
One question we see a lot on the forums, in fact there was another one today, is "How much storage will I need for X amount of video?" Our friends at Digital Heaven have created a wonderful free widget for those of you on Mac OS 10 computer systems for this very thing.
Call VideoSpace, all you do is enter the video format and the amount of footage you have (in hours and minutes) and voilá! You get exactly how much storage it will take to hold that amount of footage. And keep in mind, this is just the raw camera footage, you also need to factor in space for graphics, renders and audio files. It's a great little tool and I highly recommend it!
Here's the link:
So a recent thread in the Creative Cow Final Cut Pro forum about Internal vs. External RAID's got me thinking more about the subject. I'm a firm believer in external RAID's and really don't have any desire to install 3 or 4 drives inside my Mac Pro tower.
Let's think about this. Companies like LaCie, Ciprico, Medéa, CalDigit, Facilis and others spend a lot of time in research and development to produce storage solutions that are not only fast, but reliable. Every board, every drive, every switch, everything about that storage solution has been drawn up, assembled and tested by engineers who know a heckuva lot more than me about the inner workings of computer storage.
The hard drive units themselves are rigorously tested and "beat up" to figure out which manufacturer and model should be installed in a particular RAID unit. Drive reliability is different among manufacturers and even among individual models of the same manufacturer. Company X may have a killer 500GB model but the 750GB model may have issues. RAID companies will discover this issue much faster than I ever will and adjust their products accordingly.
RAID companies that I've dealt with have had very good tech support teams both for generaly questions and in the event of a failure. This is invaluable because when something goes wrong and a deadline is looming, the last thing I want to do is become a hardware engineer and try to figure out exactly where something might be wrong. I have enough trouble just keeping up with all the formats and software we run, I really don't care to become a computer engineer too.
Expansion is so much easier with an external RAID than internal. I mean, what do you do if you have maxed out your internal RAID and now you need another 2TB of storage? I guess you're going to connect an external RAID device? So now you're working with internal AND external media storage, at the same time? That doesn't sound like a very stable solution in my book. Start external, expand external. Keep everything coming down one or two pipes from the outside.
Speed is very limiting when it comes to internal RAID. Four drives striped together are going to be slower than say a 10 drive Ciprico Fibre Array, or maybe two arrays striped together. Heck stripe four of the new CalDigit 4:4:4 arrays together and you're pushing upwards of 900MB/s. You're not going to touch that with and internal RAID and you're certainly not going to stripe and internal AND external RAID together to gain more speed.
Zero protection is had by current internal RAID's as RAID 0 and 1 are the only supported formats right now. With External RAID's you have a multitude of protection options which gives you real control over speed vs. protection. Granted, right now I run an external SATA Array in RAID 0, but I also run another external backup device. But that will change in the next few months and we migrate potentially back to Fibre Channel.
I figured I'd ask an engineer I know about this debate and he said external is always the preferred way to go when working with high volume media storage. It's generally faster, more stable and a much more proven technology.
There you go, my reasoning for working with and recommending external RAID storage. You're free to do what you want, but this is where I stand on all this.