|Back in 2003, on September 24th, I drove up to St Paul, Minnesota and filed paperwork to make Small Tree a Minnesota LLC. |
When we started, there were 6 of us and I'm not sure we knew exactly what our plan was. I knew we wanted to write kernel drivers for "high end" networking stuff (like Intel's brand new 10Gb Ethernet cards). I didn't think much beyond that. I figured if we built it "they would come" (as the saying goes). I also needed a good excuse to buy one of the new G5 Power Macs (and make it tax deductible).
Since then, Small Tree's written drivers for 8 different Intel Ethernet chips, LACP drivers (for 10.3, back before apple had their own), iSCSI, AoE and FCoE drivers. We've also done a lot of work on storage integration and kernel performance improvements.
I'm very proud of all the hard work all of the people at Small Tree have put in and of all the great things we've accomplished. It's been quite a roller coaster ride over the years. Looking back, I'd absolutely do it again. We've got some great people and some great customers and I still love working here, even after 10 years :)
|Pope Francis is out there saying lots of interesting things. Like him or not, he’s getting a lot of attention. He’s certainly changing the tone of the Catholic Church.|
That got me thinking. What’s changing the tone of computing these days? What needs to change, what’s in the middle of changing and what’s “gone forever?”
First, let’s talk about what needs to change.
The Application appliance model is here to stay on phones and tablets. Being an accomplished Android hack myself – I built some of the first Android kernels that could talk to military hardware – I understand the value of an open, flexible device. However, I don’t want to hand that to my dad. My dad wouldn’t get it. All of that value would be lost on him, and I’d be tasked with figuring out how to build him a one of a kind, customized Android phone that could email his pictures, sync with whatever Windows revision he’s running today, and let him connect to his Yahoo account. As simple as that sounds, I’d much rather get him an iPhone knowing the answer to any of these issues is a Google search away. His iPhone will perpetually be up to date and probably do what it’s supposed to without too much fuss.
So now we need this model for computing. As much as I want to handcraft my laptop so I can be sure I’m running the very latest OpenCL with a few cool kernel fixes I read about last week, I don’t really “need” to do that. That falls into the category of “hobbyist.” We don’t run out and tweak our cars so we can get to work faster. If we are tweaking our cars, it’s to scratch some itch that has nothing to do with getting to work.
The second thing that needs to change is that Microsoft needs to stop “Chasing the gauges.”
In the flight simulator world, “Chasing the gauges” is the act of looking at your gauges and trying to manipulate the plane to get your speed, altitude and angle where you want them. The problem is the gauges lag a little and so you invariably overcompensate. You tilt to the right until the horizon gauge shows level, but by then you are too far to the right, so you begin this back and forth oscillation and never quite settle down to level.
I’m no Windows fan. I’ve been doing Unix, Linux and Mac OS X for some 25 years. However, in all that time, I’ve had no choice but to map all my knowledge over to Windows. I had to learn how to setup networks, trace system calls, recover, reinstall, and capture crash dump information to solve problems. I had no choice. Windows is a fixture in the computing world and if you work with computers, you have to work with Windows.
What drives me crazy is that Microsoft keeps trying to “fix” all the things people complain about, while finding ways to make my background Window’s knowledge useless. They change the control panel, remove the start button, and allow vendors to replace well-known interfaces (like the control panel) with their own utilities. It’s as if Microsoft is reacting while still in the “backlash” cycle, so their changes simply start a new backlash and on we go, oscillating up and down hoping to get our plane level.
I think Microsoft needs to embrace a mechanism for consolidating all of their system administration “stuff” (ala Apple’s Preference Panel) and settle on it quickly so users of alternate platforms can easily adapt their knowledge to manage Windows too.
What’s in the middle of changing?
I think the move to solid-state storage is a no-brainer. It may seem like the drives are outrageously expensive, but consider that SSDs are mostly “printed” at large factories. NAND flash chips are a very well understood technology and the ability to print them is going to grow quickly over time. After all, these guys are operating at capacity. There’s little risk in building a new plant when you can’t meet existing demand.
Spinning disks may be cheaper today (and have larger capacity), but they are delicate, heavy and require lots of precision machining to build. They have supply chains requiring aluminum and precision machined parts as well as electronics. As we have seen, floods and storms in Southeast Asia can disrupt their supply. There will soon come a crossover point where SSDs are inexpensive enough and large enough that there’s simply no reason to continue trying to use spinning disks.
Another technology that is quickly becoming required rather than optional is a good GPU (graphics processing unit). In the olden days, the only people that cared about their GPU were high-end video/CAD customers and gamers. They’d get all slobbery about “polygons per second.” GPUs were largely one-way affairs. You could write to them very quickly (for rapid on-screen updates), but reading data back off of them was slow and expensive.
Today, this is no longer the case. PCIE and new programming interfaces like CUDA and OpenCL allow us to interact with GPUS as if they were mini-Cray Supercomputers. We can feed them complex matrix operations (whether it’s weather data, game data or Bitcoin hashes) and they spit back answers lickety-split. They are much faster than general purpose CPUs for this kind of activity. Having a powerful GPU (or more than one GPU) is rapidly becoming a requirement for any sort of powerful workstation.
What has changed?
The desktop computer appears to be dead. There are still computers that have a desktop form factor, but those computers are really workstations. They have a specific purpose and their users probably have a personal laptop they tote around with them when not using that workstation. For me, my work laptop and my personal laptop are largely the same thing. It’s just too inconvenient to try and manage two independent systems and keep things synched up.
Following this evolution to mobile computing, more and more of our stuff is online. Dropbox and Google Drive are seamless to use, but aren’t quite large enough or fast enough to store all our data yet, but over time, our network speeds are going to go up and the size and performance of cloud storage is going to explode. Our data requirements will probably plateau. After all, once you can video your entire day in 4K along with all your vital signs, what more data can you generate?
There will come a time when it’s just cheaper and easier to store all our stuff online. Rather than deleting it one day because our disks got full, the stuff will magically migrate off to older, slower storage back at some data center, to be recalled whenever future generations want to have a look.
|Visiting Australia is a once in a lifetime journey for many of us in North America. It's a 14 hour (very expensive) flight from LAX and because of the timezone shift, if you leave on Saturday, you arrive on Monday. You can forget Sunday ever existed. (Although on the way back, you arrive in LA before you even left Australia).|
I had the great pleasure of visiting Sydney Australia twice this year. First, I was able to fly down and do some sales training with Adimex and Digistor in March, and then last week to help Adimex and Digistor at their big SMPTE trade show at the Sydney Convention Center. I spoke with customers, gave a presentation every day about realtime storage, and demoed the Small Tree TitaniumZ Graphical User Interface for interested customers.
First, I have to say that Adimex and Digistor are great companies with a great bunch of professionals. They know their products and treat their customers well. Second, I think the show was a wonderful opportunity – for me personally and professionally. I didn't realize that such a large show took place in Australia. We had customers from China, New Zealand, Japan, Singapore and even Vanuatu (not to mention from all over Australia). People traveled a very long way to visit this show.
Customers were very interested in Small Tree's TitaniumZ product. I think the two most important features people asked about were ZFS expansion, which allows you to add storage without rebuilding your raids, and the new, integrated version of Mint from Flavoursys. Mint allows customers to do Avid project sharing and bin locking. So multiple Avid users can now seamlessly work together using Avid MC 6 or 7 on their Small Tree storage array. I definitely sensed a lot of interest in Avid now that MC 7 has released and Adobe has announced their shift to the cloud subscription model.
While I was running around at the conference answering questions, I also had the opportunity to do a lot of running in Sydney when I wasn’t at the trade show booth. I made sure to see the Sydney Opera House as well as the Botanical Gardens. Even in Winter, Sydney is one of the most beautiful cities you can visit anywhere in the world. If you get a chance to visit, make sure you take it!
If you’d like more information on what Small Tree was showing at the SMPTE conference – its TITANIUMZ line of products – visit us at www.Small-Tree.com.
|1. When you have lots of media coming in from various cameras to your shared storage, make sure you are ingesting that media using appropriate software.|
We have seen a few cases where people are dragging files in from the camera using the Finder, rather than the camera vendors import software.
When you do this, the media can sometimes have the "User Immutable" flag set. This flag prevents users from accidentally deleting files, even if they have appropriate permissions. You can see this flag via Right Click->get info. It's the "Locked" flag.
While this makes sense if the media is on the camera (where they expect you to do all deleting with the camera software interface), it does not make sense on your storage. However the flag is persistent and will also be set on any copies you make and any copies you make of those copies. It will also prevent the original files from being deleted when you "move" a large batch of material from one volume to another!
Obviously this will waste a lot of space and be very frustrating down the line when you have thousands of media files you can't delete. You'll also find that unsetting the Lock bit via "get info" is way to cumbersome for 10,000 files.
One simple answer is the command line. Apple has a command (as does FreeBSD) called "chflags". If you can handle using the "cd" command (Change Directory) to navigate to where all your Locked file are, you can run:
chflags -R nouchg *
This will iterate through all the directories and files (starting from whatever directory you're in) and clean off all the "Locked" bits.
2. Edit project files from your local machine, rather than shared storage.
There are a number of reasons to do this, and as time goes on, I seem to find more.
First, it's just safer. Not all apps lock project files. So it's possible that if you have enough editors all sharing the same space and everyone is very busy and the environment is hectic, someone could come along and open a project you already have open. If they "save" their copy after you save yours, your changes will be lost. It would be no different if it was a Word Document or Excel Spreadsheet. When multiple people save the same file, the last guy to save wipes out the first guy. (This is not a problem for shared media like clips and audio since those files are not being written, just pulled into projects).
Second, apps like Avid and FCP 7 all have foibles with saving to remote storage. Avid doesn't like to save "project settings" over Samba or AFP (although NFS and "Dave" from Thursby work fine). FCP seems to mess up its extended attributes when it saves, leading to "unknown file" errors and other strange behavior. (When this happens, you can easily fix it. See Small Tree Knowledge Base solution here: http://www.small-tree.com/kb_results.asp?ID=43).
Lastly, you may have different versions of apps on different machines. I recently had a customer that was using FCP 7.0 and attempting to open files written by FCP 7.0.3. The older app was unhappy with the newer format files and it created some strange error messages. While this would have been a problem no matter how the files were accessed (locally or over the network), the network share made it more confusing since it was not clear that the files came from another system. Had the user received the projects on a stick or via email, the incompatibility would have been much more obvious from the start.
If you have any questions regarding shared storage and improving your workflow, do not hesitate to contact me at firstname.lastname@example.org.
|As many of you know, we do a lot of military systems integration at Small Tree. When I'm not working on networking or storage performance, I get to play with little embedded things to make radio networks better/faster/smarter.|
So this week I was very happy to have a large order come in that needed 20 new units shipped out to the Army.
I ordered all the parts and started assembling these delicate little routers. Then I discovered, they didn't work. They "almost" worked. Data would sometimes flow in one direction, sometimes not at all. Sometimes the radios would be seen, sometimes not.
My error was in ordering the next "new" ARM cpu. It's less expensive and uses much less power (soldiers don't like to carry batteries). However this new cpu obviously doesn't work as well as the older one.
I'm sure it's some whacko timing issue. I'll have to dig into it. But for now, I had to overnight the old cpus so I could get this order out!
|Apple has always been on the leading edge of connectivity for their systems.|
Back in 2003, before we had formed Small Tree as a company, I can recall drooling over a Power Book laptop with an integrated Gigabit port. That was a crazy thing to have on a laptop at the time. Gigabit was still a little weird, very expensive, and not common as a drop at anyone’s desk. Yet here apple was putting it on a laptop.
Thunderbolt is a similarly aggressive move. It puts a great deal of IO horsepower on some very small systems.
Firstly, let’s consider what Thunderbolt is. Thunderbolt is a 4X (4 lane) PCIE 2.0 bus. It’s equal in performance (and protocol) to the top two slots of a traditional tower Mac Pro. Along with that 4X pipe, there’s a graphics output pipe for a monitor. These pipes are not shared! So using a daisy-chained monitor will not impinge on any attached IO devices.
Thunderbolt is capable of moving data at 10Gbits/sec FULL DUPLEX, meaning data can move in two directions at the same time, giving the pipe a total bandwidth of 20Gbits/sec.
As I read through the forums and opinion articles on Thunderbolt, one of the themes that pops up is “It’s Apple proprietary and expensive. Just use USB 3.0.” This is a reasonable point. USB 3.0 is capable of 4.8Gbits/sec (about half of the speed of Thunderbolt). Further, there are plans to speed up USB 3.0 to 10Gbits/sec to match Thunderbolt. So given these factors (and the low cost of most USB devices), it seems like an obvious choice.
However, there are some reasons that Thunderbolt may win the day for external high-speed connectivity (and relegate USB to it’s traditional low-end role).
First of all, most IO chips (Ethernet, SATA, SAS) are manufactured with a native PCIE backend. The chips are natively built to sit on a PCIE bus. So not only will you save the overhead of an additional protocol, the guys writing the code to support these devices only need to write one driver (PCIE). It just works whether the device is on a card or in a Thunderbolt toaster.
Another advantage of Thunderbolt is its power budget. Often, devices are powered by the port itself (very common with USB). USB can provide 4.5Watts of power to attached devices, whereas Thunderbolt offers a full 10Watts of power.
Lastly (and this is probably the most interesting thing about PCIE and Thunderbolt) is that Thunderbolt is a switched/negotiated protocol that is extremely flexible. Cards that want a 16X slot can work in a 4X slot. PCIE switches can (and do) exist to allow multiple machines to talk to one PCIE based device (like a RAID). So imagine a time in the future when devices can be connected to a “switch” in a back room and multiple systems can see them. Imagine those systems can have multiple connections to boost their bandwidth.
Thunderbolt may not be everywhere yet, but it’s really the first imaginings of a new way to handle IO outside of the “tower” type machines. I think it is easily the best choice for Mac users and will likely offer some amazing benefits in the next generation.
|Excerpted from Bob Metcalfe's Reddit post today:|
On May 22, 1973 with David R. Boggs, I (Bob Metcalfe) used my IBM Selectric with its Orator ball to type up a memo to my bosses at the Xerox Palo Alto Research Center (PARC), outlining our idea for this little invention called “Ethernet”, which we later patented.
(end of excerpt)
I've made my living via Ethernet since the mid 80s when I crawled around my office installing coaxial cabling and Ethernet terminators for my Novell Netware Network. (NE1000 cards anyone?)
Things have improved a lot since then. We all owe Bob and Dave a huge debt.
|I’m getting ready to head out to Las Vegas this weekend to get the Small Tree booth all setup (SL6005) and I’m really excited.|
First off, we have a brand new version of our Titanium platform coming out called “Titanium Z”. The Z platform is AWESOME and the folks here at Small Tree (including The Duffy) are very excited to start telling people about it.
First of all, in keeping with our history of bringing really high-tech functionality (like real time video editing) down into the commodity price space, we are now bringing down Storage Virtualization.
To offer Virtualization, we had to migrate Titanium to a new OS based on FreeBSD. In doing this, we were able to pull in ZFS technology. This gives us the ability to stripe RAID sets together, migrate data around, and add new RAID sets to existing volumes without rebuilding.
We’ve also updated all the hardware, increased performance 25% and kept our same great low price model. You get more for your money.
The Titanium 4 has also been extensively improved based on customer feedback. ZFS performance is so good, we ditched the need for a RAID controller in the new T5. At the same time, we added a 5th drive (more storage, more performance) and allowed for the addition of a dual port 10GbaseT card. So now, not only is the device mobile, fast and inexpensive, it also supports direct attaching with 10Gb Ethernet! You can bring along one of our ThunderNET boxes on your shoot and have your laptop editing over 10Gb Ethernet right out in the field.
Lastly, I’ve had tons of people bugging me about SSDs and 10Gb. I demoed a super fast box at the Atlanta Cutters called “Titanium Extreme” and we showed off real time video playing to my laptop (over Dual 10Gb ports) going 1.2GBytes/sec. (not a benchmark. Real video). We’ll have this guy along as well.
So if you want to stop by and visit us and see all this cool stuff, swing down the South Lower (6005). You can’t miss us. We will have a giant round screen hanging above us with all sorts of amazing stuff flying by put together by Walter Biscardi. We’d love to see you.
|Every year as NAB approaches, the marketing once again begins. Oh the marketing....|
As NAB approaches, I'd like to take a moment to remind people in the market for storage that Gigabytes/second is not what makes video play smoothly.
Vendors with no Computer Engineers on staff will pull together monstrous conglomerations of SSDs and RAID cards, run a few benchmarks (probably four or five different ones until they find one they like) and then claim they've hit some huge number of Gigabytes per second.
Small Tree has been supporting Server based video editing longer than anyone in the market. We were supporting Avid when they used SGI's 10 years ago (and they were SGI's largest customer). We know how things work. We helped develop them.
Playing video requires a RAID configuration that can handle multiple, clocked streams. Benchmarks on the other hand, tend to use a single stream, reading sequentially as fast as they can.
What's the difference you ask? Well, in the sequential case, the RAID controller gets to use lots of tricks to avoid the hard work of seeking around disks and reordering commands. The next block to be read is probably the next block, so things like "read ahead" work wonderfully. Don't just read the next 128k, read the next 1MB! It'll all be read next anyhow. It makes it very easy for sequential benchmarks to look good. In the Supercomputing world, meaningless TeraFLOP marketing numbers were referred to as "MachoFLOPS". We knew they meant nothing when vendors could spin assembly instructions in a tight loop and claim 1.5PetaFLOPS.
Small Tree's testing and development involves looking carefully at how the Video Editing Programs themselves read so we can carefully mimic that traffic during testing. This lets us be sure our equipment doesn't rely on sequential tricks to deliver real, multi-stream performance.
So when you walk up to a vendor at NAB and they start telling you about their MachoGigabytes per second, make sure you ask them about their sustained latency numbers. Small Tree knows all about latency and we back it up, every day with our products.
|Very recently, Small Tree had the opportunity to go down to Atlanta and visit Walter Biscardi and upgrade his data center and edit suites. In conjunction with this trip, we also did a presentation on the upgrade for the Atlanta Cutters and showed off a new SSD based Titanium shared storage system we put together. This new Titanium SSD was able to move 1.2GB/sec of *realtime* video to Adobe Premiere with no dropped frames. This is faster than you can go with 8Gb Fibre Channel and the fastest realtime video I've ever seen displayed live without a net!|
The upgrade involved pulling out Walter's existing SFP+ 10Gb switch, which had a mix of Gigabit SFP modules for his suites and 10Gb SFP+ modules for his server, and replacing it with a 10GbaseT switch from Small Tree that had 4 SFP+ ports (for the server) and 24 10GbaseT ports for the new Titanium and some of his edit suites.
Before we dived right into putting in the new switch and adding the Titanium 8, we spent a lot of time talking about power. Walter didn't want to spend $1000 for an expensive UPS, but he wanted a good UPS that could handle the new load and not break the bank. We settled on an Ultra Xfinity that offered 1200W of load. This allowed for plenty of overhead for the 660W titanium and kept the loading on the UPS to well under the recommended 80%.
After installing the new switch, we moved all the cables over. One of the wonderful aspects of 10GbaseT is that we didn't have to do anything special when replacing ports that used to be Gigabit. 10GbaseT clocks down to Gigabit and even 100Mbit. So there was no trouble with legacy equipment or special adapters.
Once the switch was in, we turned to the Titanium 8. We installed it and plugged it into its new UPS and cabled it into the switch. We bonded the two 10GbaseT ports coming from the Titanium so it would load balance all the incoming clients.
Once that was done, it was time to upgrade some of the more important edit suites to 10GbaseT. What good is having all that 10Gb goodness in the lab when you can't feel the power all the way to the desktop? We upgraded both of Walter's iMac systems to 10Gb (via ThunderNET boxes) and added another 10Gb card to his fastest Mac Pro in Suite 1.
The result was a cool 300MB/sec writing from his iMac and 600MB/sec reading using the Aja System test. As I tell people, this isn't the best way to measure NAS bandwidth because applications like Final Cut and Adobe use different APIs to read their media files.
With the NAB Show approaching, I hope many of you that are planning to attend will be able to swing by Small Tree’s booth (SL6005) to learn more about this recent install directly from Walter, as he’ll be on-hand. While you’re there, feel free to ask about the SSD based Titanium shared storage solution we’re “going plaid” with.
If you’d rather not wait until NAB to learn more, contact me at modica at small-tree.com