Web-server – What could be causing random image/css dropping, 503 errors on a apache/php hosting website

Regarding the Apache issue, one possible cause is that your MaxClient/MaxServer setting is too high. When you get spike in traffic you use up all your RAM and cause Apache to start using swap which will very quickly kill your performance. Next time you have the issue check the output of top/free and see if any swap is being used. If it is try reducing the MaxClient/MaxServer values.

I also had an issue with Apache 1.3 where some connections wouldn't ever close and after a few days there would 90% of the connections doing nothing leaving not clients for handling incoming new connections. I solved it by simply restarting Apache each day. From the sounds of it you don't have enough traffic or time between issues for this to be a likely cause.

So, here are a few points that may help you out.

1. Provider: For most of the front end side of things, if cost is your main factor then it is up to you to find out what provider suits your needs. Reliability, Cost, and Scaling are all factors that you will need to consider.
2. Note, unless you have the user download some kind of client side program (Flash, JS etc) your servers will have to receive the file and then upload it to S3 for them. This will induce a lot of load as well as bandwidth costs. However, it will also give you much better control over 'what' can be uploaded and how. Once you hand control over to the client you will not be able to truly control what gets uploaded.
3. S3 is great for storing static content and it will be key in creating a site like this and keeping costs in line. Make sure you properly control who has upload permissions to which buckets. For example, if you have css and javascript in one bucket, only you should be able to upload to that location, otherwise a malicious user could upload some nasty files to replace your content. On the other side if you are going to allow the user to upload content directly to save on bandwidth, you will have to make sure that is a separate bucket, ideally per user. This is not trivial to enforce, and nearly impossible if you provide the client direct upload access.

Depending on your upload configuration (Client Side Client vs Server Side Client) your needs will be different. Client Side will be cheaper up front for server costs, but be aware that someone will probably find a way to store any kind of file and you will be responsible for moderating that content. For the Server Side model, be prepared to have your server costs increase with user traffic as you will need to build out more servers to handle upload requests.

Once you have the content hosted you will also want to look into a CDN (Content Delivery Network) such as Amazon's CloudFront (if you want to stay on the Amazon stack) or Akamai Networks. These will increase your costs at first, but save you money on high usage content.

Amazon SimpleDB is an interesting Database style. It is 'eventually consistent' which means that data sent to the database may not be immediately accessible, similar to Amazon S3. If you are going use the database as a way to keep data synced across multiple nodes for many realtime transactions, I would not recommend it.