|I have to admit, as an old time UNIX guy that's been around inodes, fsck and corrupted filesystems all my life, snapshots sounded a little too good to be true. |
The word was long known to me. Customers would say, "I took a snapshot of that disk so I could upgrade it and revert if I screwed something up." It's just that imaging a disk would take hours. You'd start the copy and go home for the night.
These new snapshots (like those supported by ZFS) were instantaneous. One click and you would “instantly” have a new copy of your data. How? That's not even possible. To make it even weirder, the new copy takes up no space!? Now it's starting to sound like perpetual motion.
The actual explanation is a lot simpler. Every filesystem is composed of data (your file data) and metadata (the name of the file, permissions, location of blocks, inode number, etc.). All this metadata is what organizes your data. You have what's called an "inode table" where all that stuff lives, and it "points to" the actual data you wrote. It might be video data, or your mom's apple pie recipe.
When you create a snapshot, you are instantly making a copy of that inode table. You now have two. All these inodes point to the same data. So the data was not copied.
Now the magic happens. When a user deletes a file from the original data, the inode for that file is removed, but the snapshot inode remains. ZFS will keep the data around as long as there's an inode in some snapshot somewhere pointing to it. The same is true if you edit a file. The old data is saved, but the new data gets written.
All this old stuff (old data) essentially becomes part of the snapshot. As more things change, the snapshot grows larger. If you were to delete “all” the data on the original filesystem, the snapshot would essentially grow to the size of the original filesystem. (The original filesystem would drop to 0.)
In some ways, it's a little like a trashcan. When you delete something, it doesn't really go away. It goes into the trash. If you wanted to, you could drag it out of the trash.
There's a similar way of recovering snapshots. You simply "clone" (or mount) them. When you do this, the snapshot inode table is mounted and it still points to all the old data. That file you deleted yesterday? If you mount yesterday's snapshot, it's right back where it was. Simply drag it back out.
Obviously, while snapshots make for a great method of saving previous images of a set of data, they are not a backup solution. If your RAID dies and can't be recovered, your snapshots die too! So for true backup protection, consider rsync or some other method of moving your data to another system.
Small Tree's TitaniumZ servers support snapshots and rsync and we have a very nice graphical interface so you can manage it all yourself. If you have any questions about snapshots or a backup solution that’s right for your editing team, don’t hesitate to contact me at firstname.lastname@example.org.
|Yesterday, I had to say goodbye to my dog Pete. He's been with me for almost 10 years and helped me raise my kids, chase all the deer away, and keep my house safe from squirrels and mailmen.|
He started feeling ill last week (while I was away at NAB) and this Friday, after several visits to the vet, we figured out it was a serious case of liver cancer, which was now causing significant pain.
I thought I'd include my favorite picture of Pete, after falling asleep in the back of my truck on the way home from up north. He'll always be missed. He was a good boy.
The entire family was with him at the end.
|1. Shared storage is becoming the norm. It's not a "hack" anymore that's used to skirt licenses or the need for more disks. Vendors are beginning to embrace it more and more, and the storage software and protocols are adapting. There's never been a better time to implement a shared storage solution.|
2. 10Gb is being adopted very quickly. Small Tree has 10Gb ports built into its TitaniumZ systems and vendors are releasing inexpensive 10GbaseT Thunderbolt PODS now. So it's time to get up to speed with 4K codecs and start using 10Gb Ethernet.
3. Don't skimp on storage space. The storage you use for every day editing needs to be kept below 80% full to avoid fragmentation. Over-provision your editing space and plan on having some sort of archive space as well. Small Tree has TitaniumZ archive options that are very inexpensive that let you store twice as much stuff for half the price.
4. Small Tree's new TitaniumZ operating system (ZenOS 10) uses a balanced storage allocation strategy so your performance remains constant as the disk begins to fill up. So you get performance and efficiency across the entire array, which also helps to mitigate any fragmentation issues.
5. Shared NAS storage like Small Tree's is easy to setup and manage. You don't need meta-data servers, licenses, or expensive fibre channel networks. You just rack it up, plug it in and go!
|We’ve been working pretty hard on Thunderbolt products over the last few weeks and I thought I’d write up some of the interesting things we’ve implemented.|
I’m sure most of you are aware that Thunderbolt is an external, hotplug/unplug version of PCIE. Thunderbolt 1 provided a 4X PCIE bus along with an equivalent bus for graphics only. Thunderbolt 2 allows you to trunk those two busses for 8X PCIE performance.
This is a new feature of Thunderbolt designed to deal with the uncertainty of what a user may plug in.
Normally, when a system boots up, all of the PCIE cards are in place. The system sorts out address space for each card and each driver is then able to map its hardware and initialize everything.
In the Thunderbolt world, we can never be sure what’s going to be there. At any time, a user could plug in not just one device, but maybe five! They could all be sitting on the users desk, daisy-chained, simply waiting for a single cable to install.
When this happens, the operating system needs the capability to reassign some of the address space and lanes so other devices can initialize and begin working.
This is where PCIE Pause comes into play. PCIE Pause allows the system to put Thunderbolt devices into a pseudo sleep mode (no driver activity) while bus address space is reassigned. Then devices are re-awakened and can restart operations. What’s important to note is that the hardware is not reset. So barring the odd timing issue causing a dropped frame, a PCIE Pause shouldn’t even reset a network mount on a Small Tree device.
Wake On Lan
We’ve been working hard on a Wake On Lan feature. This allows us to wake a machine from a sleep state in order to continue offering a service (like File sharing, ssh remote login or Screen sharing). This may be important for customers wanting to use a Mac Pro as a server via Thunderbolt RAID and Network devices.
The way it works is that you send a “magic” packet via a tool like “WakeonMac” from another system. This tells the port to power up the system far enough to start responding to services like AFP.
What’s interesting about the chip Small Tree uses (Intel x540) is that it requires power in order to watch for the “magic” wake up packet. Thunderbolt wants all power cut to the bus when the machine goes to sleep. So there’s a bit of a conflict here. Does a manufacturer violate the spec by continuing to power the device, or do they not support WOL?
This is most definitely true for the early Thunderbolt/PCIE card cage devices. They were all very careful to follow the Thunderbolt specification (required for certification and branding) and this leaves them missing this “powered while sleeping” capability.
Interested in learning more about how you could be using Thunderbolt? Contact me at
|1. The non-linear editing market (FCP, Avid etc) is changing rapidly. Avid was delisted, FCP supports NFS natively, Adobe is adding tons of new features (and a subscription model). More than ever, editors need to see what's out there and how people are using it.|
2. Storage is changing rapidly. SSDs are becoming more and more common (and less and less pricy) and spinning disk vendors are consolidating.
3. Thunderbolt is here (and it appears that it's here to stay) and it offers new methods for connecting high bandwidth IO and video devices. Should you go big and buy a Mac Pro with 6 Tbolt ports? Or can you go small and buy an iMac with 2 Tbolt ports and just hot plug? Are the devices too loud to be in your edit suite? Now's the time to come and see.
4. There are many new cameras and codecs. They are have different methods of access to systems. It's good to hear from each storage and/or camera vendors how that will work.
5. New technology announcements. With all these changes coming, vendors are constantly looking for new ways to make better, faster and cheaper. Many of these revolutionary ideas are announced at NAB. I think it's helpful to be there and see “in person” the sort of reaction different products get.
6. Who's living and who's dying? Every vendor paints a happy face on their business and their products. It's always good to see that translated into booth traffic. It should be interesting to see which edit software vendors are getting visited this year.
|Small Tree has been working closely with Adobe to make sure our shared editing storage and networking products work reliably and smoothly with Adobe’s suite of content creation software.|
Since NAB 2013, we’ve worked closely with Adobe to improve interoperability and performance, and test new features to give our customers a better experience.
Most recently, I had the chance to test out Adobe Anywhere in our shop in Minnesota.
Adobe Anywhere is designed to let users edit content that might be stored in a high bandwidth codec, over a much slower connection link. Imagine having HD or 4K footage back at the ranch, while you’re in the field accessing the media via your LTE phone and a VPN connection.
The way it works is that there’s an Adobe Anywhere server sitting on your network that you connect to with Adobe Premiere and this server compresses and shrinks the data “on the fly” so it can be fed to your machine much like a YouTube video. Except you are scrubbing, editing, cutting, dubbing and all of the other things you might need to do during an edit session.
This real-time compression/transcoding happens because the Adobe Anywhere system is taking advantage of the amazing power of GPUs. Except rather than displaying the video to a screen, the video is being pushed into a network stream that’s fed to your client.
I tested my system out with some Pro Res promotional videos we’ve used at trade shows in the past, and did my editing over Wi-Fi.
What I found was that the system worked very well. I could see that the Adobe Anywhere system was reading the video from Small Tree’s shared storage at full rate, then pushing it to my system at a greatly reduced rate. I had no trouble playing, editing and managing the video over my Wi-Fi connection (although Adobe recommends 1Gb Ethernet as the minimum connectivity for clients today).
This type of architecture is very new and there are caveats. For example, if you are very far from the server system or running over a very slow link (like a vpn connection), latency can make certain actions take a very long time (like loading an entire project, or using Adobe’s Titler app which requires interactivity). Adobe cautions that latencies of 200msecs or more will lead to a very poor customer experience.
Additionally, just because the feed to the clients is much lower bandwidth (to accommodate slower links), the original video data still needs to be read in real-time at full speed. So there are no shortcuts there. You still need high quality, low latency storage to allow people to edit video from it. You just have a new tool to push that data via real-time proxies over longer and slower links.
All in all, I found the technology to be very smooth and it worked well with Small Tree’s shared network storage. I’m excited to see the reach of Small Tree shared storage extended out to a much larger group of potential users.
For a demonstration of Adobe Anywhere over Small Tree shared storage, visit us at the
NAB Show in Las Vegas this April (Booth SL11105).
|One day, when we're sitting in our rocking chairs recounting our past IT glories ("Why, when I was a young man, computers had ‘wires’”), we'll invariably start talking about our storage war stories. There will be so many. We'll talk of frisbee tossing stuck disks or putting bad drives in the freezer. We'll recount how we saved a company’s entire financial history by recovering an alternate superblock or fixing a byte swapping error on a tape with the "dd" command. I'm sure our children will be transfixed.|
No…no, they won't be transfixed, any more than we would be listening to someone telling us about how their grandpa's secret pot roast recipe starts with "Get a woodchuck...skin it." You simply have to be in an anthropological state of mind to listen to something like that. More likely, they walked into the room to ask you your wifi password (Of course, only us old folk will have wifi. Your kids are just visiting. At home they use something far more modern and futuristic. It'll probably be called iXifi or something).
Unfortunately for us, many of these war story issues remain serious problems today. Disks “do” get stuck and they “do” often get better and work for a while if you freeze them. It's a great way to get your data back when you've been a little lazy with backups.
Another problem is fragmentation. This is what I wanted to focus on today.
Disks today are still spinning platters with rings of "blocks" on them, where each block is typically 512 bytes. Ideally, as you write files to your disk, those bytes are written around the rings so you can read and write the blocks in sequence. The head doesn't have to move. Each new block spins underneath it.
Fragmentation occurs because we don't just leave files sitting on our disk forever. We delete them. We delete emails, log files, temp files, render files, and old projects we don't care about anymore. When we do this, those files leave "holes" in our filesystems. The OS wants to use these holes. (Indeed, SGI used to have a real-time filesystem that never left holes. All data was written at the end. I had to handle a few cases where people called asking why they never got their free space back when they deleted files. The answer was "we don't ever use old holes in the filesystem. That would slow us down!")
To use these holes, most operating systems use a "best fit" algorithm. They look at what you are trying to write, and try to find a hole where that write will fit. In this way, they can use old space. When you're writing something extremely large, the OS just sticks it into the free space at the end.
The problem occurs when you let things start to fill up. Now the OS can't always find a place to put your large writes. If it can't, it may have to break that large block of data into several smaller ones. A file that may have been written in one contiguous chunk may get broken into 11 or 12 pieces. This not only slows down your write performance, it will also slow down your reads when you go to read the file back.
To make matters worse, this file will remain fragmented even if you free more space up later. The OS does not go back and clean it up. So it's a good idea not to let your filesystems drop below 20% free space. If this happens and performance suffers, you're going to need to look into a defragmentation tool.
Soon, this issue won't matter to many of us. SSDs (Solid State Disks) fragment just like spinning disks, but it doesn't matter near as much. SSDs are more like Random Access Memory in that data blocks can be read in any order, equally as fast. So even though your OS might have to issue a few more reads to pull in a file (and there will be a slight performance hit), it won't be near as bad as what a spinning disk would experience. Hence, we'll tell our fragmentation war stories one day and get blank looks from our grandkids (What do you mean "spinning disk?" The disk was “moving??”).
Personally, I long for the days when disk drives were so large, they would vibrate the floor. I liked discovering that the night time tape drive operator was getting hand lotion on the reel to reel tape heads when she put the next backup tape on for the overnight runs. It was like CSI. I'm going to miss those days. Soon, everything will be like an iPhone and we'll just throw it away, get a new one, and sync it with the cloud. Man that sucks.
Follow Steve Modica and Small Tree on Twitter @smalltreecomm. Have a question? Contact Small Tree at 1-866-782-4622.
|Now that we’re several months removed from Apple’s introduction of Mavericks for OSX and we've all tested the waters a little, I wanted to talk about video editing software and how the various versions play with NAS storage like we use at Small Tree.|
Avid has long since released Media Composer 7, and from what I've seen, their AMA support (support for non-Avid shared storage), continues to improve. There are certainly complaints about the performance not matching native MXF workflows, but now that they've added read/write support, it's clear they are moving in a more NAS friendly direction. With some of the confusion going on in the edit system space, we're seeing more and more interested in MC 7.
Adobe has moved to their Creative Cloud model and I've noticed that it made it much easier to keep my system up to date. All of my test systems are either up to date, or telling me they need and update, so I can be fairly certainly I'm working with the latest release. That's really important when dealing with a product as large and integrated as the Adobe Suite of products. You certainly don't want to mix and match product revisions when trying to move data between After Effects and Premiere.
Another thing I've really grown to like about Adobe is their willingness to work with third party vendors (like Small Tree) to help correct problems that impact all of our customers. One great example is that Adobe worked around serious file size limitations present in Apple's QuickTime libraries. Basically, any time an application would attempt to generate a large QuickTime file (larger than 2GB), there was a chance the file would stop encoding at the 2GB mark. Adobe dived into the problem, understood it, and worked around it in their applications. This makes them one of the first to avoid this problem and certainly the most NAS friendly of all the video editing applications out there.
Lastly, I've seen some great things come out of FCP X in recent days. One workflow I'm very excited about involves using "Add SAN Location" (the built in support for SAN Volumes) and NFS (Network File Sharing). It turns out, if you mount your storage as NFS and create "Final Cut Projects" and "Final Cut Events" within project directories inside that volume, FCP X will let you "add" them as SAN locations. This lets you use very inexpensive NAS storage in lieu of a much more expensive Fibre Channel solution. For shops that find FCP X fits their workflow, they'll find that NFS NAS systems definitely fit their pocket books.
So as you move forward with your Mac platforms into Mavericks and beyond, consider taking a second look at your NLE (Non-Linear Editor) of choice. You may find that other workflow options are opening up.
|With the New Year festivities well behind us, today seems like as good a time as any to chat about where video editing storage is (or should be) headed in 2014. |
First, I’m really excited about FCoE. FCoE is great technology. It's built into our (Small Tree) cards, so we get super fast offloads. It uses the Fibre Channel protocol, so it's compatible with legacy Fibre Channel. You can buy one set of switches and do everything: Fibre Channel, 10Gb and FCoE (and even iSCSI if you want).
Are there any issues to be concerned about with FCoE? One problem is that the switches are too darn expensive! I've been waiting for someone to release an inexpensive switch and it just hasn't happened. Without that, I'm afraid the protocol will take a long time to come to market.
Second, I'm quite sure SSDs are the way of the future. I'm also quite sure SSDs will be cheaper and easier to fabricate than complex spinning disks. So why aren’t SSDs ubiquitous yet? Where are the 2 and 4 TB SSD drives that fit a 3.5" form factor? Why aren't we rapidly replacing our spinning disks with SSDs as they fail?
Unfortunately, we're constrained by the number of factories that can crank out the NAND flash chips. Even worse, there are so many things that need them, including smartphones, desktop devices, SATA disks, SAS disks, PCIE disks. With all of these things clawing at the market for chips, it's no wonder they are a little hard to come by. I'm not sure things will settle down until things "settle down" (i.e., a certain form factor becomes dominant).
Looking back at 2013, there were several key improvements that will have a positive impact on shared storage in 2014. One is Thunderbolt. Small Tree spent a lot of time updating its drivers to match the new spec. Once this work was done, we had some wonderful new features. Our cards can now seamlessly hotplug and unplug from a system. So customers can walk in, plug in, connect up and go. Similarly, when it’s time to go home, they unplug, drop their laptop in their backpack, and go home. I think this opens the door to allowing a lot more 10Gb Ethernet use among laptop and iMac users.
Apple’s new SMB implementation in 2013 was also critical for improvements in video editing workflow. Apple’s moving away from AFP as their primary form of sharing storage between Macs, and the upshot for us has been a much better SMB experience for our customers. It’s faster and friendlier to heterogeneous environments. I look forward to seeing more customers moving to an open SMB environment from a more restrictive (and harder to performance tune) AFP environment.
So as your editing team seeks to simplify its workflow to maximize its productivity in 2014, keep these new or improved technological enhancements in mind. If you have any questions about your shared storage solution, don’t hesitate to contact me at email@example.com.
|Back in 2003, on September 24th, I drove up to St Paul, Minnesota and filed paperwork to make Small Tree a Minnesota LLC. |
When we started, there were 6 of us and I'm not sure we knew exactly what our plan was. I knew we wanted to write kernel drivers for "high end" networking stuff (like Intel's brand new 10Gb Ethernet cards). I didn't think much beyond that. I figured if we built it "they would come" (as the saying goes). I also needed a good excuse to buy one of the new G5 Power Macs (and make it tax deductible).
Since then, Small Tree's written drivers for 8 different Intel Ethernet chips, LACP drivers (for 10.3, back before apple had their own), iSCSI, AoE and FCoE drivers. We've also done a lot of work on storage integration and kernel performance improvements.
I'm very proud of all the hard work all of the people at Small Tree have put in and of all the great things we've accomplished. It's been quite a roller coaster ride over the years. Looking back, I'd absolutely do it again. We've got some great people and some great customers and I still love working here, even after 10 years :)