I have a lot of data (svn repository and postgresql database) in EC2 instance, yesterday the server doesn't respond to http requests, and I `sudo reboot', after that the server couldn't boot up again, what should I do?
The system log in the AWS console shows nothing output from the instance, and the most recent sys log is:
Ubuntu 9.10 domU-12-31-39-00-A0-06 tty1
domU-12-31-39-00-A0-06 login: [1796075.232506] INFO: task rsyslogd:22423 blocked for more than 120 seconds.
[1796075.232522] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1796195.232509] INFO: task pdflush:17 blocked for more than 120 seconds.
[1796195.232525] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1796195.232639] INFO: task kjournald:296 blocked for more than 120 seconds.
[1796195.232646] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1796195.232728] INFO: task rsyslogd:22423 blocked for more than 120 seconds.
[1796195.232735] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1796195.232858] INFO: task amuled:874 blocked for more than 120 seconds.
[1796195.232865] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1796195.232962] INFO: task master:1716 blocked for more than 120 seconds.
[1796195.232968] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1796195.233065] INFO: task qmgr:1724 blocked for more than 120 seconds.
[1796195.233071] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1796195.233190] INFO: task java:4554 blocked for more than 120 seconds.
[1796195.233196] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1796195.233296] INFO: task java:4745 blocked for more than 120 seconds.
[1796195.233302] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1796195.233353] INFO: task java:11614 blocked for more than 120 seconds.
[1796195.233360] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
It seems I can't do anything now, I can't SSH into it, and can't operate in single user mode, too.
Any idea??
Best Answer
EC2 instances cannot be booted into safe mode.
What is the output of ec2-get-console-output for that instance? It might contain clues as to what's preventing the instance from booting properly.
If this is an EBS-backed instance you'll be able to "stop" the instance, detach the root EBS volume from it, attach the root EBS volume to another instance and make changs to the contents, then reattach it to the original instance and "start" it running again.
If this is an instance-store instance then you're out of luck - there's no way to go in and fix one of those puppies.