: Steve Modica's Blog
As most of you haven't heard, there's an election coming up next week. It's a mid-term election, so very few people will vote. It means the real hard core extreme groups get to come in and influence, since the moderates don't bother to show.
As most of you also haven't heard, critical elections already have taken place in other parts of the world, including Brazil, where President Dilma Rousseff won re-election by a narrow margin. Typically, I don't follow politics in Brazil, but fortunately for me, I had a front row seat to this Brazilian election.
It turns out, President Rousseff's re-election campaign purchased 96TB of storage from Small Tree (3 TitaniumZ 16 systems) so they could edit their campaign ads. Leading up to the election, we worked closely with members of her team on a number of exciting things like ungrounded 10Gb wall plates, switch firmware upgrades and a 24X7 edit cycle. Fun times :)
But in all seriousness, the experience working with her team further demonstrated the impact that post-production and shared storage have on our daily lives - in many instances without our even knowing it. After all, the commercials and other promotional materials that the campaign's post team worked on could have been the difference in President Rousseff being re-elected and that ultimately has an impact on the short-term, if not long-term future of Brazil. And it's likely that her team faced tight deadlines when trying to develop positive ads in response to attack ads from her opponent. In such instances, having a shared storage solution that can handle multiple editors accessing large media files is mission critical.
I'm proud that Small Tree's TitaniumZ-16 met the challenge and I'm glad to see she pulled it out. I hope to see her have a very successful term.
11 years ago, I jumped in my car and drove to St Paul Minnesota and filed the paperwork to start Small Tree Communications LLC.
In that time, I never would have imagined all the things we've been through. It's been a crazy ride.
I got to work with the Virginia Tech Super cluster of Macs and ride around in Dr Varadarajan's Porshe Cayenne.
I got to meet Bob Zelin, Ron Amborn (at Maxx Digital), and Walter Biscardi at Biscardi Creative.
I got to build military routers and weird embedded devices and drive around Ft Bragg and White Sands Missile Range.
I got to work with some great device driver guys like Jeff Perrault, Jim Hunter and Steve Betker (who retired from Small Tree recently) and I've got to hire on new people that are just as amazing.
In all this time, we haven't lost our passion for assembling new technology and finding new ways to help make it easy for customers to use it. My day still goes fastest when I'm explaining things to people.
Of course, this brings me to Chris Duffy who I think has been here at Small Tree longer than anyone else *but* me. We both came from SGI support and just love helping people with weird problems. To this day, we still help each other diagnose some of the strangest things. It's like CSI for computers.
Here's to the next 11 years. Hopefully, there will be fewer twists and turns :)
I’m heading out to IBC and there are a number of things I hope to see there. Of course, I’ve got customers asking me about SSDs, and engineers working on 40Gb Ethernet and people want to bring it all together. Really, what’s the hold up here?
My short wish list:
3.5” Server Chassis (8, 16, 24) with 12Gb SAS expanders onboard
2.5” Server Chassis (12 and 24) with 12Gb SAS expanders onboard
Balanced 40Gb switches that can legitimately aggregate 16 or 24 10Gb ports into 4 or 6 40Gb ports
4TB or larger SSDs that can handle enterprise workloads but cost less than $1000 per Terabyte
Thunderbolt 3 previews
40Gb Ethernet Adapters
8 or 10Terabyte 7200RPM SAS drives
New Wi-Fi technology that can run full duplex and offer backpressure and bandwidth reservation (can you imagine editing wirelessly?)
Obviously, I have a few of these technologies in hand already, but there are some major roadblocks to building a balanced server with them. SSDs are very expensive and still too small. We’ll need those 400MB/sec devices to justify putting 40Gb ports in a server.
For those of you that will be around next week after Labor Day, we are going to be running a special at Small Tree. If you purchase a TitaniumZ (8 or 16), we’re giving away two SANLink2 10Gb Ethernet Adapters. Using the two onboard 10Gb ports on the Titanium, you can immediately connect two clients and be editing over 10Gb. I think it’s a great back to school offer.
Contact Small Tree today to purchase your TitaniumZ system with two Promise SANLink2 10Gb Ethernet Adapters included - email@example.com or 866-782-4622. Purchase must be completed by 9/30/14.
During a normal week, I help a lot of customers with performance issues. Some of the most common complaints I hear include:
“I bought a new 10Gb card so I could connect my Macs together, but when I drag files over, it doesn’t go any faster.”
“I upgraded the memory in my system because Final Cut was running slow, but it didn’t seem to help very much.”
“I bought a faster Mac so it would run my NLE more smoothly, but it actually seems worse than before.”
All of these things have something in common. Money was spent on performance, the users didn’t have a satisfying experience, and they would be much happier had the money been spent in the right place.
Of course, the first one is easy. Putting a 10Gb connection between two Macs and dragging files between them isn’t going to go any faster than the slowest disk involved. If one of those Macs is using an old SATA spinning disk, 40-60MB/sec would be a pretty normal transfer rate. A far cry from the 1000MB/sec you might expect from 10Gb Ethernet! Who wouldn’t be disappointed?
Similarly, the second case where a user upgrades memory based on an anecdotal suggestion of a friend is all too common. On the one hand, memory upgrades are typically a great way to go, especially when you run a lot of things simultaneously. More memory almost always means better performance. However, this is assuming that you didn’t have some other serious problem that was overwhelming your lack of memory.
In the case of Final Cut 7, which is a 32 bit application, more memory isn’t going to help Final Cut directly. In fact, it’s much more likely that Final Cut would run better with a faster disk and perhaps a faster CPU. Since FCP 7 didn’t use GPU offload, even moving to a better graphics card might not have delivered a huge gain.
The last one, where buying a faster Mac actually made things worse, is a classic case of mismatched performance tuning. For this customer, the faster Mac also had a lot more memory. It turns out that Mac OS X will dynamically increase the amount of data it will move across the network in a burst (the TCP Receive Window). This resulted in the network overrunning Final Cut, causing it to stutter. The solution? Dial back the receive window to make sure FCP 7 can keep up. This will be corrected by some other changes in the stack that are coming soon. One day, slower applications will be able to push back on the sender a little more directly and a little more effectively than today.
These cases bring to mind a discussion I had with a 40Gb Ethernet vendor back at NAB in April. They wanted me to use their cards and perhaps their switches. The obvious question: Don’t your users want the speed of 40Gb Ethernet? Wouldn’t they want to run this right to their desktops?!
Of course they would. Everyone wants to go fast. The problem is that those 40Gb ports are being fed by storage. If you look closely at what raid controllers and spinning disks can do, the best you can hope for from 16 drives and a raid card is around 1GB/sec. A 40Gb cards moves about 4GB/sec. So if I sold my customers 40Gb straight to their desktops, I would need somewhere around 64 spinning disks just to max out ONE 40Gb port. It could be done, but not economically. It would be more like a science project.
Even worse, on Macs today, those 40Gb ports would have to connect with Thunderbolt 2, which tops out around 2.5GB/sec and is yet another choke point that would lead to disappointed customers and wasted money.
I think 40Gb Ethernet has a place. In fact, we’re working on drivers today. However, that place will depend on much larger SSDs that can provide 1GB/sec per device. Once we’re moving 8 and 16GB/sec either via a RAID card or ZFS logical volumes, then it will make sense to put 40Gb everywhere. The added advantage is that waiting to deploy 40Gb will only lead to better and more stable 40Gb equipment. Anyone remember the old days of 10Gb back in 2003 when cards were expensive, super hot, and required single mode fiber?
We've been developing some new software and hardware features for the TitaniumZ line and I'd like to come out and speak to your users group about them.
If you have a Final Cut, Adobe Premiere or Avid group and are interested in hearing about storage futures, Small Tree's new products, 40Gb Ethernet, or anything else I might know something about, let me know and I'll plan to come visit. Maybe I'll even give away a couple SANLink 10Gb Ethernet devices.
We got RSS working on our 10Gb cards a few days ago. This is a feature that splits up data coming into the card into multiple queues. Then we can let different CPUS handle pulling in that data and passing it up the stack. We found what we figured we'd find: When we setup multiple streams, we see data in multiple queues. We see more cpus involved in the work, and things go a lot faster.
What surprised us was how great this made SAMBA. When we tested SMB3 with Yosemite, we were able to hit line rate (10Gb/sec) between two systems! This is due to SMB Multichannel. It's amazing. Soon, we should be able to extend this across adapters as well (we actually can, but not to the same share). This will let us do things like FC and iSCSI do today, but with a NAS. We'll be able to stripe bandwidth.
I have to admit, as an old time UNIX guy that's been around inodes, fsck and corrupted filesystems all my life, snapshots sounded a little too good to be true.
The word was long known to me. Customers would say, "I took a snapshot of that disk so I could upgrade it and revert if I screwed something up." It's just that imaging a disk would take hours. You'd start the copy and go home for the night.
These new snapshots (like those supported by ZFS) were instantaneous. One click and you would “instantly” have a new copy of your data. How? That's not even possible. To make it even weirder, the new copy takes up no space!? Now it's starting to sound like perpetual motion.
The actual explanation is a lot simpler. Every filesystem is composed of data (your file data) and metadata (the name of the file, permissions, location of blocks, inode number, etc.). All this metadata is what organizes your data. You have what's called an "inode table" where all that stuff lives, and it "points to" the actual data you wrote. It might be video data, or your mom's apple pie recipe.
When you create a snapshot, you are instantly making a copy of that inode table. You now have two. All these inodes point to the same data. So the data was not copied.
Now the magic happens. When a user deletes a file from the original data, the inode for that file is removed, but the snapshot inode remains. ZFS will keep the data around as long as there's an inode in some snapshot somewhere pointing to it. The same is true if you edit a file. The old data is saved, but the new data gets written.
All this old stuff (old data) essentially becomes part of the snapshot. As more things change, the snapshot grows larger. If you were to delete “all” the data on the original filesystem, the snapshot would essentially grow to the size of the original filesystem. (The original filesystem would drop to 0.)
In some ways, it's a little like a trashcan. When you delete something, it doesn't really go away. It goes into the trash. If you wanted to, you could drag it out of the trash.
There's a similar way of recovering snapshots. You simply "clone" (or mount) them. When you do this, the snapshot inode table is mounted and it still points to all the old data. That file you deleted yesterday? If you mount yesterday's snapshot, it's right back where it was. Simply drag it back out.
Obviously, while snapshots make for a great method of saving previous images of a set of data, they are not a backup solution. If your RAID dies and can't be recovered, your snapshots die too! So for true backup protection, consider rsync or some other method of moving your data to another system.
Small Tree's TitaniumZ servers support snapshots and rsync and we have a very nice graphical interface so you can manage it all yourself. If you have any questions about snapshots or a backup solution that’s right for your editing team, don’t hesitate to contact me at firstname.lastname@example.org.
Yesterday, I had to say goodbye to my dog Pete. He's been with me for almost 10 years and helped me raise my kids, chase all the deer away, and keep my house safe from squirrels and mailmen.
He started feeling ill last week (while I was away at NAB) and this Friday, after several visits to the vet, we figured out it was a serious case of liver cancer, which was now causing significant pain.
I thought I'd include my favorite picture of Pete, after falling asleep in the back of my truck on the way home from up north. He'll always be missed. He was a good boy.
The entire family was with him at the end.
1. Shared storage is becoming the norm. It's not a "hack" anymore that's used to skirt licenses or the need for more disks. Vendors are beginning to embrace it more and more, and the storage software and protocols are adapting. There's never been a better time to implement a shared storage solution.
2. 10Gb is being adopted very quickly. Small Tree has 10Gb ports built into its TitaniumZ systems and vendors are releasing inexpensive 10GbaseT Thunderbolt PODS now. So it's time to get up to speed with 4K codecs and start using 10Gb Ethernet.
3. Don't skimp on storage space. The storage you use for every day editing needs to be kept below 80% full to avoid fragmentation. Over-provision your editing space and plan on having some sort of archive space as well. Small Tree has TitaniumZ archive options that are very inexpensive that let you store twice as much stuff for half the price.
4. Small Tree's new TitaniumZ operating system (ZenOS 10) uses a balanced storage allocation strategy so your performance remains constant as the disk begins to fill up. So you get performance and efficiency across the entire array, which also helps to mitigate any fragmentation issues.
5. Shared NAS storage like Small Tree's is easy to setup and manage. You don't need meta-data servers, licenses, or expensive fibre channel networks. You just rack it up, plug it in and go!
We’ve been working pretty hard on Thunderbolt products over the last few weeks and I thought I’d write up some of the interesting things we’ve implemented.
I’m sure most of you are aware that Thunderbolt is an external, hotplug/unplug version of PCIE. Thunderbolt 1 provided a 4X PCIE bus along with an equivalent bus for graphics only. Thunderbolt 2 allows you to trunk those two busses for 8X PCIE performance.
This is a new feature of Thunderbolt designed to deal with the uncertainty of what a user may plug in.
Normally, when a system boots up, all of the PCIE cards are in place. The system sorts out address space for each card and each driver is then able to map its hardware and initialize everything.
In the Thunderbolt world, we can never be sure what’s going to be there. At any time, a user could plug in not just one device, but maybe five! They could all be sitting on the users desk, daisy-chained, simply waiting for a single cable to install.
When this happens, the operating system needs the capability to reassign some of the address space and lanes so other devices can initialize and begin working.
This is where PCIE Pause comes into play. PCIE Pause allows the system to put Thunderbolt devices into a pseudo sleep mode (no driver activity) while bus address space is reassigned. Then devices are re-awakened and can restart operations. What’s important to note is that the hardware is not
reset. So barring the odd timing issue causing a dropped frame, a PCIE Pause shouldn’t even reset a network mount on a Small Tree device.
Wake On Lan
We’ve been working hard on a Wake On Lan feature. This allows us to wake a machine from a sleep state in order to continue offering a service (like File sharing, ssh remote login or Screen sharing). This may be important for customers wanting to use a Mac Pro as a server via Thunderbolt RAID and Network devices.
The way it works is that you send a “magic” packet via a tool like “WakeonMac” from another system. This tells the port to power up the system far enough to start responding to services like AFP.
What’s interesting about the chip Small Tree uses (Intel x540) is that it requires power in order to watch for the “magic” wake up packet. Thunderbolt wants all power cut to the bus when the machine goes to sleep. So there’s a bit of a conflict here. Does a manufacturer violate the spec by continuing to power the device, or do they not support WOL?
This is most definitely true for the early Thunderbolt/PCIE card cage devices. They were all very careful to follow the Thunderbolt specification (required for certification and branding) and this leaves them missing this “powered while sleeping” capability.
Interested in learning more about how you could be using Thunderbolt? Contact me at