| | | | Excerpted from Bob Metcalfe's Reddit post today:
On May 22, 1973 with David R. Boggs, I (Bob Metcalfe) used my IBM Selectric with its Orator ball to type up a memo to my bosses at the Xerox Palo Alto Research Center (PARC), outlining our idea for this little invention called “Ethernet”, which we later patented.
(end of excerpt)
I've made my living via Ethernet since the mid 80s when I crawled around my office installing coaxial cabling and Ethernet terminators for my Novell Netware Network. (NE1000 cards anyone?)
Things have improved a lot since then. We all owe Bob and Dave a huge debt.
Steve
| | | | |
| | | | I’m getting ready to head out to Las Vegas this weekend to get the Small Tree booth all setup (SL6005) and I’m really excited.
First off, we have a brand new version of our Titanium platform coming out called “Titanium Z”. The Z platform is AWESOME and the folks here at Small Tree (including The Duffy) are very excited to start telling people about it.
First of all, in keeping with our history of bringing really high-tech functionality (like real time video editing) down into the commodity price space, we are now bringing down Storage Virtualization.
To offer Virtualization, we had to migrate Titanium to a new OS based on FreeBSD. In doing this, we were able to pull in ZFS technology. This gives us the ability to stripe RAID sets together, migrate data around, and add new RAID sets to existing volumes without rebuilding.
We’ve also updated all the hardware, increased performance 25% and kept our same great low price model. You get more for your money.
The Titanium 4 has also been extensively improved based on customer feedback. ZFS performance is so good, we ditched the need for a RAID controller in the new T5. At the same time, we added a 5th drive (more storage, more performance) and allowed for the addition of a dual port 10GbaseT card. So now, not only is the device mobile, fast and inexpensive, it also supports direct attaching with 10Gb Ethernet! You can bring along one of our ThunderNET boxes on your shoot and have your laptop editing over 10Gb Ethernet right out in the field.
Lastly, I’ve had tons of people bugging me about SSDs and 10Gb. I demoed a super fast box at the Atlanta Cutters called “Titanium Extreme” and we showed off real time video playing to my laptop (over Dual 10Gb ports) going 1.2GBytes/sec. (not a benchmark. Real video). We’ll have this guy along as well.
So if you want to stop by and visit us and see all this cool stuff, swing down the South Lower (6005). You can’t miss us. We will have a giant round screen hanging above us with all sorts of amazing stuff flying by put together by Walter Biscardi. We’d love to see you.
| | | | |
| | | | Every year as NAB approaches, the marketing once again begins. Oh the marketing....
As NAB approaches, I'd like to take a moment to remind people in the market for storage that Gigabytes/second is not what makes video play smoothly.
Vendors with no Computer Engineers on staff will pull together monstrous conglomerations of SSDs and RAID cards, run a few benchmarks (probably four or five different ones until they find one they like) and then claim they've hit some huge number of Gigabytes per second.
Small Tree has been supporting Server based video editing longer than anyone in the market. We were supporting Avid when they used SGI's 10 years ago (and they were SGI's largest customer). We know how things work. We helped develop them.
Playing video requires a RAID configuration that can handle multiple, clocked streams. Benchmarks on the other hand, tend to use a single stream, reading sequentially as fast as they can.
What's the difference you ask? Well, in the sequential case, the RAID controller gets to use lots of tricks to avoid the hard work of seeking around disks and reordering commands. The next block to be read is probably the next block, so things like "read ahead" work wonderfully. Don't just read the next 128k, read the next 1MB! It'll all be read next anyhow. It makes it very easy for sequential benchmarks to look good. In the Supercomputing world, meaningless TeraFLOP marketing numbers were referred to as "MachoFLOPS". We knew they meant nothing when vendors could spin assembly instructions in a tight loop and claim 1.5PetaFLOPS.
Small Tree's testing and development involves looking carefully at how the Video Editing Programs themselves read so we can carefully mimic that traffic during testing. This lets us be sure our equipment doesn't rely on sequential tricks to deliver real, multi-stream performance.
So when you walk up to a vendor at NAB and they start telling you about their MachoGigabytes per second, make sure you ask them about their sustained latency numbers. Small Tree knows all about latency and we back it up, every day with our products. | | | | |
| | | | Very recently, Small Tree had the opportunity to go down to Atlanta and visit Walter Biscardi and upgrade his data center and edit suites. In conjunction with this trip, we also did a presentation on the upgrade for the Atlanta Cutters and showed off a new SSD based Titanium shared storage system we put together. This new Titanium SSD was able to move 1.2GB/sec of *realtime* video to Adobe Premiere with no dropped frames. This is faster than you can go with 8Gb Fibre Channel and the fastest realtime video I've ever seen displayed live without a net!
The upgrade involved pulling out Walter's existing SFP+ 10Gb switch, which had a mix of Gigabit SFP modules for his suites and 10Gb SFP+ modules for his server, and replacing it with a 10GbaseT switch from Small Tree that had 4 SFP+ ports (for the server) and 24 10GbaseT ports for the new Titanium and some of his edit suites.
Before we dived right into putting in the new switch and adding the Titanium 8, we spent a lot of time talking about power. Walter didn't want to spend $1000 for an expensive UPS, but he wanted a good UPS that could handle the new load and not break the bank. We settled on an Ultra Xfinity that offered 1200W of load. This allowed for plenty of overhead for the 660W titanium and kept the loading on the UPS to well under the recommended 80%.
After installing the new switch, we moved all the cables over. One of the wonderful aspects of 10GbaseT is that we didn't have to do anything special when replacing ports that used to be Gigabit. 10GbaseT clocks down to Gigabit and even 100Mbit. So there was no trouble with legacy equipment or special adapters.
Once the switch was in, we turned to the Titanium 8. We installed it and plugged it into its new UPS and cabled it into the switch. We bonded the two 10GbaseT ports coming from the Titanium so it would load balance all the incoming clients.
Once that was done, it was time to upgrade some of the more important edit suites to 10GbaseT. What good is having all that 10Gb goodness in the lab when you can't feel the power all the way to the desktop? We upgraded both of Walter's iMac systems to 10Gb (via ThunderNET boxes) and added another 10Gb card to his fastest Mac Pro in Suite 1.
The result was a cool 300MB/sec writing from his iMac and 600MB/sec reading using the Aja System test. As I tell people, this isn't the best way to measure NAS bandwidth because applications like Final Cut and Adobe use different APIs to read their media files.
With the NAB Show approaching, I hope many of you that are planning to attend will be able to swing by Small Tree’s booth (SL6005) to learn more about this recent install directly from Walter, as he’ll be on-hand. While you’re there, feel free to ask about the SSD based Titanium shared storage solution we’re “going plaid” with.
If you’d rather not wait until NAB to learn more, contact me at modica at small-tree.com | | | | |
| | | | If you use Adobe Premiere for post-production projects, then you may have come across the problem where the application re-conforms and recreates peak files for projects every time the project opens or when changing back to Adobe from using another application, i.e. hide/show.
If you see this problem occurring, check that all systems are time synched with each other (IE NTP is enabled in the Time and Date preference panel). If you do not have internet access on the systems, make sure all the systems are at least locked to the server time.
The cause of this problem is that Adobe is creating new files on the server. If the server has a time that is "in the future" - relative to your client - Adobe Premiere will see these future files and decide that the project is out of date. Then it will want to rebuild all the files. When using any application - including Final Cut, Adobe Premiere or Avid across a NAS or SAN - it is imperative that all of the machines agree on the time and date so the applications can tell when a project is up to date and when it needs to be updated.
The issue above will also occur with other applications in a network based client-server environment. Using the Small Tree Titanium Server as an example, one can setup NTP so that it can synchronize its clock the same way and to the same NTP server as the Mac clients do.
To enable NTP client code on Titanium Server:
-Login to the Titanium GUI.
-Ensure that the Titanium has a Network Interface defined on your local network,
intranet, such that you statically define or use DHCP to obtain an IP address.
-Define a default router and DNS server using the
SETUP -> network -> interfaces GUI area. If you need to add a default router, make sure you apply the IP address of the router to the appropriate network interface.
-Go to:
HARDWARE -> setup
find the "Current server time" area and set/click on "Use ntp",
then define the NTP server, i.e. time.apple.com, as an example and
"apply".
-If you have the network IP address, default route and DNS setup
properly, it will come back with the correct Date/Time. If not,
you will need to revisit your network settings to correct.
If you’re having workflow issues with Adobe Premiere and your server, contact me at info@small-tree.com.
| | | | |
| | | | Storage is a tough market and customers are always willing to pay a little less to get a little less. My take away is this: In the war between Ethernet and EtherNOT based storage, such as Fibre Channel, the one that delivers the best value for the lowest price is going to win. As Warren Buffet likes to say, "In the short term, the market is a popularity contest. In the long term, it's a weighing machine." People need to buy based on value over time.
Fibre Channel has been hamstrung for a long time by its need for custom ASICs (chips used to implement the protocol in hardware). Fibre Channel wanted to overcome all of the limitations of Ethernet and so they invented a protocol that did just that. The problem of course is that those custom ASICS are not on motherboards. You don't get FC chips built into your DELL server (unless you order a special card or riser). You don't see Apple putting FC chips on Mac Pros (even tho they sold Xsan and XRAID for so long).
What's the result? Expensive chips. It's expensive to fab them and expensive to fix them. FC stuff is expensive. Vendors may find ways to lower the entry point, but somewhere or other, either via support, licensing or upgrades, the cost will be expensive.
Ethernet certainly has ASICS as well. There are network processors, MAC (media access control chips) and PHY chips (the chips that implement the physical layer). They can be incredibly expensive. The first 10Gb cards Small Tree sold were $4770 list price! But here's the thing...a 10Gb card today is $1000 or less. The chips are everywhere and they are rapidly going onto motherboards. Ethernet is truly ubiquitous and will continue to be for server and storage technologies.
If you'd like to discuss or debate Ethernet vs EtherNOT, send me an email at info@small-tree.com or hit me up on Twitter @svmodica. | | | | |
| | | | Not too long ago, I was asked to write up my predictions on storage and networking technology for the coming year. One of those predictions was the rise of new, combined file system/logical volume managers like ZFS and BtrFS.
These file systems don’t rely on RAID cards to handle things like parity calculations. They also don’t “hide” the underlying drives from the operating system. The entire IO subsystem - drives and controllers - is available to the operating system and data is laid out across the devices as necessary for best performance.
As we’ve begun experimenting ourselves with these technologies, we’ve seen a lot of very promising results.
First and foremost, I think it’s important to note that Small Tree engineers mostly came from SGI and Cray. While working there, most of our time in support was spent “tuning.” People wouldn’t buy SGIs or Crays simply to run a file server. Invariably, they were doing something new and different like simulating a jet fighter or rendering huge 3D databases to a screen in real-time. There would always be some little tweak required to the OS to make it all work smoothly. Maybe they didn’t have enough disk buffers or disk buffer headers. Maybe they couldn’t create enough shared memory segments.
Small Tree (www.small-tree.com) has always brought this same skill set down to commodity hardware like SATA drives and RAID controllers, Ethernet networks and Intel CPUS. These days, all of this stuff has the capability to handle shared video editing, but quite often the systems aren’t tuned to support it.
I think ZFS is the next big step in moving very high-end distributed storage down into the commodity space.
Consider this: A typical RAID card is really an ASIC (Application Specific Integrated Circuit). Essentially, some really smart engineering guys write hardware code (Verilog, VHDL) and create a chip that someone “prints” for them. SGI had to do this with their special IO chips and HUB chips to build huge computers like the Columbia system. Doing this is incredibly expensive and risky. If the chip doesn’t work right in its first run, you have to respin and spend millions to do it again. It takes months.
A software based file system can be modified on the fly to quickly fix problems. It can evolve over time and integrate new OS features immediately, with little change to the underlying technology.
What excites me most about ZFS is we can now consider the idea of trading a very fast - and expensive - hardware ASIC for a distributed file system that uses more CPU cores, more PCIE lanes and more system memory to achieve similar results. To date, with only very basic tuning and system configuration changes, we’ve been able to achieve Titanium level performance using very similar hardware, but no RAID controller.
So does this mean we’re ready to roll out tomorrow without a RAID controller?
No. There’s still a lot of work to do. How does it handle fragmentation? How does it handle mixed loads (read and write)? How does it handle different codecs that might require hundreds of streams (like H.264) or huge codecs that require very fast streams (like 4K uncompressed)? We still have a lot of work to do to make sure ZFS is production ready, but our current experience is exciting and bodes well for the technology.
If you’d like to chat further about combined file system/logical volume managers, other storage/networking trends, or have questions regarding your workflow, contact info@small-tree.com.
| | | | |
| | | | Back when I was at SGI slaying dragons I had the good fortune to visit America Online. At the time, AOL was moving about 20% of the USA’s email traffic through SGI Challenge XL servers.
This was around the time they crossed 10 million (with an M) users. That was a lot back then – there were t-shirts printed. Facebook is approaching 1 billion (with a B) users today.
As you can imagine, AOL introduced some serious issues of scale to our products. We’d never really had anyone use a Challenge XL server to handle 100,000 mail users (much less five gymnasiums full of Challenge XL servers to handle 10 million). Having so many systems together created some interesting challenges.
First off, when a customer has two systems and you have a bug that occurs once a year, the two-system customer may never see it. If they do see it, they might chalk it up to a power glitch. Your engineers may never get enough reports to fix the problem since it’s simply not reproducible in a way you can recreate.
Not so with 200 in a room. You might see that once a year glitch every day. That’s a very different prospect. Manufacturing will never see that in quality control running systems one at a time. It can only be seen with hundreds of systems together.
In AOL’s case, we had the “Twilight Hang” (and no, there were no vampires that sparkled). A machine would simply “stop.” There was no core dump. It could not be forced down and there would be no error messages. The machine was simply frozen in a twilight state. This is the worst possible situation because engineers and support personnel cannot gather data or evidence to fix the problem. There’s no way to get a fingerprint to link the problem to other known issues.
SGI mustered a very strong team of people (including me) to go onsite with a special bus analyzer to watch one of the machines that seemed to hit the problem more than the others. I was there for three weeks. In fact, my fiancé’ actually flew out on the last of the three weeks because it was her birthday and I was not scheduled to be gone that long.
I can recall one highlight from this trip was me sitting in a room with some of the onsite SGI and AOL people having a conference call with SGI engineering and SGI field office people. During this call, the engineering manager was explaining the theory that the /dev/poll device might be getting “stuck” because of a bug with the poll-lock. Evidently, the poll-lock might get locked and never “unlocked,” which would cause the machine to hang. I had to ask, “Carl, how many poll locks does it take to hang a system?” There was dead silence. I came to find out that the other SGI field people on the phone had hit mute and were rolling on the floor laughing. (Thanks guys). The Corporate SGI people were not amused.
Anyhow, the ultimate cause of the problem was secondary cache corruption. Irix 5.3 was not detecting cache errors correctly, and when it did it would corrupt the result every other time. Ultimately, they completely disabled secondary cache correction and to this day, you Irix users will notice a tuning variable called “r4k_corruption.” You have to turn that on to allow the machine to attempt to correct those errors (even at the risk of corrupting memory). The ultimate solution for AOL was to upgrade to R10k processors that “correctly” corrected secondary cache errors every time.
| | | | |
| | | | I did some testing with Strawberry from Flavoursys last week. Flavoursys provides software for project sharing and project management for video post-production. I’m happy to report that this is one of the most exciting new products I’ve had a chance to work with in a long time.
What is it?
Strawberry gives you the ability to share Avid projects and media from shared storage without the usual indexing issues or potential corruption problems. It provides for user and group access, metadata based search and safe read only access to projects and media that others are working on.
How does it do it?
It’s pretty ingenious. Strawberry is a client system that sits on your network. You permanently mount your shared storage to the Strawberry system so it has full administrative access. It will work all its magic on your server via this mount.
It will create project and media sub directories for your workstations. In my very simple setup, they were edit_1 for media and edit_1p for projects. There were also directories for edit_2, edit_3 and so on.
Users will access the Strawberry server via a web browser or built in app. They login with a name the administrator provides and are then able to create and open projects. When a project is created and opened, the appropriate project files and resources are created on the “main” storage and links are then generated in the users workstation project and media directories. Those links remain as long as the project remains open in Strawberry. When the project is closed, those links are removed.
Here's a screen snap of the blank Strawberry startup window:
And here's another shot of what it looks like to create a new project (it allows you to enter a rudimentary amount of metadata which is used for project naming):
What’s great about this setup is that this same user can open other users’ projects and “add” them to his own (read only) so that he can share timelines and media that might be important. All of this is managed via these dynamic links in the edit_1p and edit_1 subdirectories.
Is it simple to use?
Absolutely. As a user, I would simply login, click “create,” fill in some basic information - like my name - and select “open.”
Once this was done, I simply opened Avid Media Composer on my client and selected the External Project type and pointed to the edit_1p directory. I could see my new (empty) project and begin to work. My media would be stored in my edit_1 media directory, safely sequestered from other users so there would be no
re-indexing issues. I was able to create a number of projects and open them all read-only as add-ons to my new project.
Is it easy to setup?
This was probably the most difficult part about using Strawberry. There were several setup issues, but I know Flavoursys is currently addressing those and will have them fixed shortly. I think the product is very useful and worth the money now. However, I’ll caution potential buyers that there is the possibility they will have to have someone in to help with configuration early on.
To use Strawberry, you have to install a 1U Supermicro chassis onto your network. It boots into Red Hat, but there’s really no other instruction. I had to email Flavoursys to figure out what to do next.
The documentation is very new and pretty raw. So even when following the instructions step by step, I hit a number of puzzle points where I was unsure what to do next. This didn’t stop me from getting it set up, but it could be frustrating for editors who are expecting a plug-and-play solution.
While the system is very elegant in how it works, they brought it to market quickly by using some Windows compatibility tricks. This means there are extra layers that could potentially confuse and complicate the setup.
For example, the actual software is running on a Virtualbox virtual machine under Red Hat Linux. So the server is a Windows 2008 server running under Linux. Your users won’t be connecting to one of the Linux addresses you setup when you connected the machine. They will be connecting to a Windows Virtual address.
Additionally, the client software runs under Microsoft’s Silverlight. So all of your clients will need the Silverlight plugin to get to the Strawberry user interface. I didn’t find this to be a problem - I already had it installed on my laptop- but I had some difficulty getting Firefox to see the plugin on my test iMac (Safari worked fine).
As I understand it, Flavoursys is working on a native version of all of this to improve the performance and simplify the product.
My recommendation:
This is a great and innovative product. It will allow for existing Avid stations to exist in broader, heterogeneous environments and reduce significantly the amount of money small shops have to pay for Avid compatible shared storage. Small Tree is now offering Strawberry with its products for this very reason.
Strawberry also offers a great upgrade path for those that are not using Avid today, but believe they will need to in the future. These shops can go out and purchase NAS based storage now, knowing that down the road they can add Avid sharing capabilities without doing a forklift upgrade.
About the only caveat I would put in place on a Strawberry purchase would be that you’ll need someone with strong sysadmin skills for the setup. There are a number technical steps and concepts (like virtual machines and NFS vs Samba mounts) that you’ll want configured by a Pro. Once that work is completed, it should be smooth sailing.
For more information on workflow solutions, visit www.small-tree.com or contact info@small-tree.com.
| | | | |
| | | | I started in the computer industry around 1988 as a computer engineering co-op student with a small company called Herstal Automation. Herstal built memory boards for very old HP 1000 A600 and A900 computers. These computers were some of the very first real-time computers ever made. Auto companies and the medical industry used these to monitor real-time processes like engine performance or patient vital signs.
To give you some idea how old these were, the machine had toggle switches on the front so you could hand enter machine code instructions one at a time. This was a good way to enter tiny little test programs or force the machine to boot when you were stuck.
Typically, I would build up memory boards and boot the machine to test. Sometimes, a board would fail. When that happened, I would use the front panel toggle switches to put in a small assembly instruction program that would write all 1’s into memory. Then I would dump the memory and see if there were any single zeros (bad data line) or entire words that were 0 (bad address line). I could almost zero in on the exact pin or chip that was failing and replace it. I could unsolder a bad chip and replace it so cleanly that you couldn’t tell it wasn’t machine soldered.
One day, I recall a board that was behaving very strangely. I couldn’t seem to get it to power on. There simply were no lights.
I pulled the board out and did a visual inspection. This is tricky because if you look at a pattern (like pins on a board) the eye will see a clean pattern. It’s very easy to miss a bent pin. Our brains fool us into “seeing” the pin even when it’s not there. One has to intentionally look at each and every pin.
Even with all that inspection, I could not see a problem. Finally, I pulled out my voltmeter and started looking for a bad trace. Maybe one of the caps or resistors on the board was simply bad and was not passing current through.
I touched the power pin and the power lines on the downstream chips and could not read a connection. Hmmm… I started working backwards, closer to the pin, but still no connection. Eventually, I had both leads on the pin. Still no connection! The gold pin on the edge of the board was not conducting electricity.
I have to admit, I was not a good physics student. I hated physics. However I did very well in electronics and I know that gold conducts electricity. In fact, gold is great at conducting electricity. So I did the scratch test. I took a tiny little screwdriver and gently slid it against the pin (scratching gold pins on these boards was a definite no no). A thin film of plastic bubbled up.
Then I knew what was wrong.
In those days, electronic boards were assembled by hand by installing parts and bending the pins down to hold them in place. The pins were then nipped off and the boards were sent to a “wave solder” facility. These places would pass the board over a flowing “wave” of molten solder and all of the pins and pads would be soldered at once in a very uniform and reliable manner.
There was one problem with wave soldering in that if you had gold connector pins on your card (like a PCI card), the solder would stick to the gold and ruin it.
To avoid this, wave solder companies would use a special water-soluble tape on these edge connectors. After the wave solder was complete, the boards were put into an industrial dishwasher and the tape would dissolve. (This would also clean the boards and make them look nice and new).
So the reason I had this layer of plastic was that our wave solder company’s dishwasher was broken. It was not heating the water enough to completely dissolve the protective tape layer. I confirmed with a quick phone call and used a tooth brush and some hot water to fix the rest of the batch.
Obviously, in this day and age, this is a pretty rare problem. Boards you buy for your edit stations are built in large quantities and Quality Control Tested on pin grids and test rigs to quickly rule out any obvious problems.
That being said, there are things you need to consider when dealing with large PCIE boards that you might have to plug into an older machine:
1. Make sure you aren’t carrying around any excess static that might zap the board. If the vendor provides one, put on an appropriate grounding strap when installing a board.
2. Be very careful when installing not to knock loose any surface mount components. In the old days, things were soldered right through the board, but today, they are only surface soldered. If you bump one of those little chips too hard, you will knock it off. Depending on what it is, your board may not work at all, or will be intermittent.
3. What carefully for interference. Often, large graphics boards have huge heat sinks or cases and fans on them. Make sure none of these devices is touching a neighboring board. This could lead to electrical shorts or overheating
4. Make sure all wires and cables are strapped or tied down! If you leave unused connectors hanging, one day they are going to end up hitting a fan or a hot component and melting.
5. The 5Volt and 12Volt supply lines inside most computers aren’t going to kill you, however, imagine what might happen if you shorted a ring or watch against a 12V line. It would get extremely hot very quickly and may even melt (while touching your skin!) So don’t take these relatively low voltage supplies lightly.
When working with complex electronic assemblies, following these steps can save you a lot of time and frustration down the road.
| | | | |
|
|
Reply
Like
