Coming from a server and Linux background, moving a site to the cloud naturally aligns to services like EC2 and EBS volumes. Those are locally attached hard drives to your VM running in the cloud, where you access files in a filesystem like any local storage. However, one of the big pushes from AWS is to use their S3 storage solution. Today I ran some tests but eventually decided to stick with EBS storage for a few reasons that I'll describe here.
The big advantage of S3 is cost. It costs about twice as much to store data on magnetic EBS volumes versus using S3 storage. So when it comes to large quantities of data, S3 certainly has the advantage. S3 also offers some even lower cost options for data that you may not need to access right away, similar to tape backup.
Files stored in S3 are automatically synchronized to multiple availability zones, which means the availability and safety of those files is greater. However, EBS volumes use RAID automatically in the backend, so for a small project (or a personal web site like mine) that isn't of great concerns. I wouldn't use either solutions without a backup plan anyways.
One important thing to note about S3 is that it isn't a filesystem. It's an object storage, which means you always need to access it via APIs. That's fine when you're building an app, but it makes it very inconvenience to transfer files to and from S3, link to files, get statistics and so on. If you're used to the command line, S3 is best accessed through a console or programming interface.
Because S3 is more reliable and automatically synchronized among many availability zones, I thought at first that linking to S3 content would bring a significant speed boost, but it did not. In fact, not only is it slower to setup since you need to upload files to S3 as a first step instead of just saving them locally, but web users saw a small dip in performance as well.
To test this out, I uploaded over a thousand images with thumbnails to a S3 bucket. This is the benchmark run using EBS storage:
This is the benchmark run using S3:
Both features have their pros and cons. In my opinion, S3 has great advantages in many cases. If you need to access files from multiple desperate cloud locations, then S3's natural deduplication can be helpful. If you have a very large amount of data, then S3 will bring you a much larger cost saving. And if you're purely building a web app and won't need to manually access files, then the API interface would probably not be an issue for you.
Personally, I decided to stick with EBS storage. Having the files locally accessible through a filesystem was a major deciding factor, and the difference in cost for me was only a few dollars a month.
Your mileage may vary.