I'm using supervisor to start my node.js application on a micro EC2 instance. However, the app only stays running for some time until it eventually shuts down. Not exactly sure how long the app stays running but I'm guessing for about a few hours or so. Sometimes less. My question is where on the remote server should I be looking in order to debug this kind of issue? I'm running an Amazon Linux AMI.
Node app crashes after a while on micro EC2 instance even with supervisor
amazon ec2debuggingloggingnode.jssupervisord
Related Solutions
(Strange post, so hopefully this won't be as strange a reply).
On the subject of security flaws: it is considered general bad practice to store cgi-bin scripts within the document root of the web server. Even the W3C eludes to it under "Are compiled languages such as C safer ..." in their World Wide Web Security FAQ:
Consider the following scenario. For convenience's sake, you've decided to identify CGI scripts to the server using the .cgi extension. Later on, you need to make a small change to an interpreted CGI script. You open it up with the Emacs text editor and modify the script. Unfortunately the edit leaves a backup copy of the script source code lying around in the document tree. Although the remote user can't obtain the source code by fetching the script itself, he can now obtain the backup copy by blindly requesting the URL:
http://your-site/a/path/your_script.cgi~
(This is another good reason to limit CGI scripts to cgi-bin and to make sure that cgi-bin is separate from the document root.)
This is not as significant a threat as the ability to write a file within the document root. However, an attacker could obtain the source code of the cgi, devise a directed attack against it, and use it as a stepping stone into the server.
To mitigate this, you can add the following lines to the lighttpd.conf (or some variation therein) to direct cgi-bin to a directory separate from the /var/www/lighttpd document root.
$HTTP["url"] =~ "/cgi-bin/" { cgi.assign = ( "" => "" ) }
alias.url = ( "/cgi-bin/" => "/usr/lib/cgi-bin/" )
This requires both the cgi and alias modules for lighttpd.
If you don't have premium support, then "stuck" instances (where you can't stop/terminate them) and "stuck" volumes (where you can't detach/delete them) can be reported to Amazon on the EC2 forum:
Amazon AWS EC2 Forum
https://forums.aws.amazon.com/forum.jspa?forumID=30
Make sure you list the specific instance/volume ids involved.
Nobody but Amazon can really help in these situations.
Fortunately, you should not be charged for instance hours once it enters the "stopping" or "terminating" state.
Your original problem about not being able to connect to the instance through ssh could also be posted on the EC2 forum for help from the community, but it is a common problem and has many potential causes. I've written an article to help start diagnose this and to point out pieces of information you should include in your forum post:
Solving: "I can't connect to my server on Amazon EC2"
http://alestic.com/2009/08/ec2-connectivity
I've also written an article describing a method that can be used to diagnose EBS boot instances by looking at the log files on the disk even if you can't connect to the instance:
Fixing Files on the Root EBS Volume of an EC2 Instance
http://alestic.com/2011/02/ec2-fix-ebs-root
However, you won't be able to use this approach until Amazon helps you detach your EBS volume.
Best Answer
You can try to get node log using this supervisord configuration. After a crash, you will get what really happened. Provably the error might be because node crash for some unexpected error. Try to use console.logs