When I first heard about Amazon’s Simple Storage Service (S3) in 2006, I wasn’t sure what anyone was ever going to use it for. It was not quite a web hosting service, because there were no extra services like PHP or MySQL. Or even FTP access. This was clearly aimed at developers, but most of those would probably have their own dedicated server boxes with tons of storage, right? Until I stumbled across a ‘use case’ for it this week. I signed up and found it to be a very flexible and useful service.
Preparing to serve millions of widgets
As you may have noticed, I created a little Flickr widget that uses a Flash movie to display the most recent images you posted to Flickr. I originally considered making it a WordPress plugin, but there wasn’t really a reason to do that. The images were already on Flickr, there was no WordPress data being used, nor was there any need for PHP. I decided to create a form that would generate code snippets, and host the Flash movie myself.
I estimate that my WP-Cumulus plugin is used by at least ten thousand blogs right now. If those get an average of a hundred pageviews each day, that would mean that WP-Cumulus gets served a million times a day. Because it is a WP plugin, each blog hosts the file, so this is not an issue for me. But with the Flickr widget, I was going to be hosting the movie myself. This one is a lot smaller (under 5k), but times one million that’s still 5 GB of bandwidth per day. This tiny little movie could potentially cause more data traffic than I was set up to handle.
In comes S3
What I remembered about S3 is that you only pay for what you actually use. With no monthly minimum. So if my project would fail to capture the imagination of the Flickr crowd, it wouldn’t cost me anything. On the other hand, Amazon’s infrastructure would allow almost unlimited growth. Their servers can handle pretty much anything, so the movie would be online all the time. Even a regular web host’s 99.9% uptime guarantee means your website can be down for nearly 45 minutes each month. If a lot of people depend on you to host something, that’s not cool.
Here’s what my activity overview looked like yesterday. So far, I think I can afford this
S3 for mere mortals
Because the service is primarily aimed at developers, you need to communicate with it using an API (Application Programming Interface). If you’re fluent in Java that is probably very convenient, but if you only drink the stuff you’ll need help. Luckily, there are a lot of cool tools out there to help mere mortals get their stuff onto S3.
The best place to start is probably “Amazon S3 Simple Storage Service – Everything You Wanted to Know” by Amit Agarwal. It lists the most important tools and explains how S3 works in layman’s terms. I’ve tried the S3fox extension for Firefox, and used it to upload my files. On my Windows box I think I’ll switch to CloudBerry Explorer, which does not need Firefox (but does need the .NET framework). Both use a familiar FTP-style interface that makes it very easy to upload an download files to and from S3.
But the best thing about S3 is that it can be used for a great many things. A program that caught my attention JungleDisk. It uses S3 to store backups of local files. Reliable off-site backups can be quite expensive, but with S3’s relatively low prices you probably won’t have to sell your car.
What Amazon has done is create a very flexible storage platform. It’s great if you need to store a tiny file that you want to be able to serve to millions of users, but the flexible pricing scheme makes it equally suitable for off-site backups of large files. Or anything in between. I wouldn’t be surprised if more clever uses for the system continue to pop up.