Amazon S3 – pay-per-view online file storage

Amazon Web Services logoWhen I first heard about Amazon’s Simple Storage Service (S3) in 2006, I wasn’t sure what anyone was ever going to use it for. It was not quite a web hosting service, because there were no extra services like PHP or MySQL. Or even FTP access. This was clearly aimed at developers, but most of those would probably have their own dedicated server boxes with tons of storage, right? Until I stumbled across a ‘use case’ for it this week. I signed up and found it to be a very flexible and useful service.

Preparing to serve millions of widgets

As you may have noticed, I created a little Flickr widget that uses a Flash movie to display the most recent images you posted to Flickr. I originally considered making it a WordPress plugin, but there wasn’t really a reason to do that. The images were already on Flickr, there was no WordPress data being used, nor was there any need for PHP. I decided to create a form that would generate code snippets, and host the Flash movie myself.

I estimate that my WP-Cumulus plugin is used by at least ten thousand blogs right now. If those get an average of a hundred pageviews each day, that would mean that WP-Cumulus gets served a million times a day. Because it is a WP plugin, each blog hosts the file, so this is not an issue for me. But with the Flickr widget, I was going to be hosting the movie myself. This one is a lot smaller (under 5k), but times one million that’s still 5 GB of bandwidth per day. This tiny little movie could potentially cause more data traffic than I was set up to handle.

In comes S3

What I remembered about S3 is that you only pay for what you actually use. With no monthly minimum. So if my project would fail to capture the imagination of the Flickr crowd, it wouldn’t cost me anything. On the other hand, Amazon’s infrastructure would allow almost unlimited growth. Their servers can handle pretty much anything, so the movie would be online all the time. Even a regular web host’s 99.9% uptime guarantee means your website can be down for nearly 45 minutes each month. If a lot of people depend on you to host something, that’s not cool.

Amazon S3 activity overview
Here’s what my activity overview looked like yesterday. So far, I think I can afford this :)

S3 for mere mortals

Because the service is primarily aimed at developers, you need to communicate with it using an API (Application Programming Interface). If you’re fluent in Java that is probably very convenient, but if you only drink the stuff you’ll need help. Luckily, there are a lot of cool tools out there to help mere mortals get their stuff onto S3.

The best place to start is probably “Amazon S3 Simple Storage Service – Everything You Wanted to Know” by Amit Agarwal. It lists the most important tools and explains how S3 works in layman’s terms. I’ve tried the S3fox extension for Firefox, and used it to upload my files. On my Windows box I think I’ll switch to CloudBerry Explorer, which does not need Firefox (but does need the .NET framework). Both use a familiar FTP-style interface that makes it very easy to upload an download files to and from S3.

Other uses

But the best thing about S3 is that it can be used for a great many things. A program that caught my attention JungleDisk. It uses S3 to store backups of local files. Reliable off-site backups can be quite expensive, but with S3′s relatively low prices you probably won’t have to sell your car.

What Amazon has done is create a very flexible storage platform. It’s great if you need to store a tiny file that you want to be able to serve to millions of users, but the flexible pricing scheme makes it equally suitable for off-site backups of large files. Or anything in between. I wouldn’t be surprised if more clever uses for the system continue to pop up.

5 Comments

  1. One thing to note is that both the applications I used will not keep a file’s ACL (access privileges) when overwriting it. So if you’ve set a file up to be available to ‘everyone’, you’ll need to do that again after updating that file. CloudBerry Explorer will give the overwritten file the same ACL settings as the bucket it’s in.

    This caused a major oops for me yesterday, because this is clearly different form how FTP works.

    Comment by Roy — February 18, 2009 @ 2:41 pm

  2. We just got a request for another user for exactly the same thing. We will raise a priority of the issue and try to address it in one of the closest releases.

    Thanks.
    Andy, CloudBerry Lab

    Comment by Andy — April 19, 2009 @ 10:32 am

  3. Just wanted to let you know that we fixed the issue in the latest version of CloudBerry Explorer. It will keep ACL on updated files as of version 1.6. Thanks for bringing it up!

    Comment by Andy — September 6, 2009 @ 8:44 pm

  4. Hi Roy,

    First of all, thanks for a great article, and a great widget.

    I was curious, though: you’ve obviously put a lot of work into this software and you’re even paying hosting fees to serve it up. It seems odd to me that premium widgets (and premium WordPress plugins) are free. I never donate, and I’m pretty sure no one else does either. But I use this software in my own blogs and I’ve made a lot of money as a result.

    I think it would fair to charge me something – you just need to make me pay rather than begging for a donation. I’m sure there are open-source devotees who would disagree, but ultimately aren’t they all just cheapskates who want free software?

    Anyway, just curious – how do you plan to “monetize” all of that hard work? Are you planning to charge for the widget? You’re an entrepreneur and you don’t seem like a student/slacker who does this stuff for fun. Maybe you make a living by selling ads here on your blog (I sure hope so) – or do you have another approach in mind?

    -Mark

    Comment by Carl Andrews — October 9, 2009 @ 7:47 am

    • Hi Mark. You’re right, nobody ever donates. This is a major issue in the WordPress theme and plugin development world and has been widely discussed. You’re also right that I’m an entrepreneur, but I don’t make a living off of Cumulus or the Flickr widget. Those are hobby projects to me, and I’ve learned a lot by creating and supporting them.

      Also, I really believe in Open Source software. It’s nice when people donate, but I’d never create a commercial version of these projects. I’ve actually stopped others from doing so (by enforcing the GPL license) and will continue to make sure these projects remain open.

      If at some point someone develops a WordPress App Store that lets you charge small amounts or plugins I’ll consider creating something for that as well. I think that could work, and it could make plugin development more sustainable for developers, probably improving the average quality too.

      Comment by Roy — October 12, 2009 @ 9:35 am