As of January 2014, it’s not worth bothering with the “s3fs” software for mounting Amazon S3 buckets on your local filesystem.
The idea of s3fs is simple and great: use FUSE (Filesystem in Userspace) to “mount” the S3 bucket the same way you’d mount, say, an NFS drive or a partition of a disk. Manipulate the files, let s3fs sync it in the background. Sure, you lose some reliability, but we’ve had NFS and SMB and all kinds of somewhat-latent-over-an-unreliable-link-but-mostly-with-filesystem-semantics software for decades now, right?
Well, forget it. s3fs as of January 2014, used on Mac OS X and against an existing set of buckets, is so utterly unreliable as to be useless.
First, s3fs cannot “see” existing folders. This is because folders are a bit of hack on S3 and weren’t done in a standardized, documented way when s3fs was first written. However, since then, at least two other ways of creating folders on S3 have gained currency: an older, deprecated way with Firefox plugin S3Fox, and a newer defacto standard way with Amazon’s own management dashboard/browser for S3. Whatever the historical reason, you can’t see the existing folders.
Second, although from mailing list posts, theoretically you can *create* an s3fs folder with the same name as your existing folder, and then its contents will magically become visible, empirically something rather different happens. A mkdir on the s3fs mount leads to the creation of a mangled *regular* file on the S3 dashboard. Now you have two “folders,” each of which is unusable as a folder on the other (S3 dashboard, or s3fs) system. Argh.
Finally, you might say, ok, fine, this will just make me use flat-level, non-folder-nested choices about my S3 architecture. (Leave aside for the moment that the very reason you want to use S3 is probably exactly so that you can have lots and lots and lots and lots (like 10^8+) files in a way that will cripple any reasonable filesystem tools that see them all in one “directory.”) However, even that doesn’t work reliably, as s3fs demonstrated today when it went into “write-only mode” such that I could create files locally that would show up on S3 but that subsequently would disappear from my local filesystem. WTF?!?
The unfortunate answer is: S3 is not a filesystem, and it was created by people who are smarter than you, and who have very craftily calculated that if you are forced to weave in the S3 API and its limitations into your application code, you will have a damn hard time ripping it out of your infrastructure, and so they are going to have you do just that. They do not want it to be used as a filesystem, and so guess what: you are *not* going to use it as a filesystem. Not gonna happen.
Say what you will, but our hometown heroes here in Seattle are no dummies. Embrace, extend, extinguish, baby. Not just for OS companies anymore…
(Yes, I know that s3fs is not an Amazon project. But it appears to be the community’s best attempt to put filesystem semantics around S3, and that attempt has been rejected by AWS’s immune system.)