What is your general strategy to backup S3 buckets?
Depending on what data you are storing you may not be interested in backing up data from S3. For instance if you have general website assets that you already have a copy of in a repository elsewhere you probably don't need to backup the assets that live in S3.
Sometimes you may use S3 to store user uploads. These might have originated from an EC2 or they may have gone straight to S3. It makes sense to use Object Versioning to be able to recover from script errors or users deleting files but changing their mind. http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectVersioning.html
As far as I understand versioning is done on the object level, so if you wanted to "revert to how your bucket looked 3 days ago" you would need to build a script that could check all the versions and dates, and request the right version for each object. This would be possible to do, it just requires a little bit of effort at the application level first.
You could look at other methods, such as syncing all the S3 bucket objects to another service (a third party server, or an EBS backed EC2). This could be your daily or weekly snapshot. This method adds extra costs, maintenance and effort so might not be the best solution, particularly for 5TB of data.
"How do you backup your entire cloud infrastructure? What is your disaster recovery plan?" How to backup Route53? CloudFront settings?
Depending on how far you want to go, all this sort of information should be scripted and in configuration files. Those configuration files should be backed up. This touches on DEVOPS and the concept of infrastructure as code.
How much time will it take to recover from script error or losing access to root console?
This is question sounds difficult to answer. What sort of script error? The first question touches on one example (a script deleting a file that lives on S3) however there are plenty more.
You can look into SimianArmy https://github.com/Netflix/SimianArmy
The Simian Army is a suite of tools for keeping your cloud operating
in top form. Chaos Monkey, the first member, is a resiliency tool that
helps ensure that your applications can tolerate random instance
failures
As for access to "root console" if you're talking about access to your OS, or your EC2s...all that should be scripted via Puppet/Chef or similar and therefore your machines are "throwaway". There is nothing special about them, they contain no individual user data and you can bring one up or down without affecting your system.
If your talking about access to the AWS console, you would need to do things like email or call to gain access, or there may be outages that you need to account for.
Best Answer
The first thing that comes to mind is the fact that data transfer in and out of S3 is quite spendy. If you're backing up frequently (as you ought to be), costs could get out of hand just with transfer fees. That said, to answer your question, backups should be performed from a separate, hardened, server whose only task in life is to perform backups. No apache, remote access only via SSH with key authentication, etc. If you do these things along with ensuring that only a select few people have access to the server, your keys should be quite safe. If you are really paranoid, you can pgp-encrypt the file that contains your keys - the problem with this approach, though, is that it requires you to enter your passphrase each time the backup job runs. That's probably not something you want to sign up for, correct?
After hearing about your restrictive budget, I can't help but think that it would be better for you to change around your storage strategy. I'm not sure what your server situation is, but could you perhaps host the files locally on the server and then just use S3 for backups? There is a great backup script called duplicity that can perform compressed, encrypted, incremental backups to S3 (among several other backend storage types).
[Edit] If you do end up hosting on S3 and backing up to local disk, it looks like there's an "If-Modified-Since" header in the S3 API that will help with performing incremental backups. For backups like this, you're most likely going to need to homebrew something, though it won't be too difficult. Just use SimpleDB/BerleleyDB/etc to store meta information about which files you have backed up along with a pointer to where they reside on disk. Keeping the meta information in a DB will also make quick work out of verifying backups as well as creating reports on backup jobs.