My java application can sometimes be killed by an external script. This can be done either with SIGTERM or with SIGKILL.
The application is a server which receives many connections per second, and it can be killed while trying to serve them.
I would like to restart the application whenever it's killed, so I have prepared a script for that purpose.
The problem is that, once the app has been killed, the new application instance can't bind to the port used by the previous instance, because the "Address is already in use". The previous instance's process has been definitely terminated, anyway the offending listening port is still there, but it is assigned to bash (or sh on other machines).
Obviouly, my goal is to restart the application and let it bind successfully to the previous address.
I've tried waiting more than 200 seconds before restarting to no avail, anyway I can't afford to wait that much.
I've encountered this problem on all the machines I've ran the application (which is a jetty server with java 1.6).
Any suggestion is appreciated,
thanks,
Silvio
EDIT Killing the jvm process is not the normal way I exit my application, this is used in case of problems (OutOfMemoryErrors) only. And I never need to kill it with SIGKILL, because SIGTERM always suffices, I resort to SIGKILL only in case SIGTERM fails, which has never happened. I'm working on a long term solution, meanwhile I have to keep my app running by applying stitches here and there.
EDIT To be more clear: this is the netstat -tunap | grep line I see before killing the process:
tcp6 0 0 :::8898 :::* LISTEN 22709/java
and this is after killing the process
tcp6 0 0 :::8898 :::* LISTEN 23665/sh
notice that the process with PID 22709 is killed and gone, but the port is still there (but locked by sh)
UPDATE after I kill my application, with netstat I can see a long list of pending connections in CLOSE_WAIT state, with my ip as destination. Also, I can see a sh process in state LISTEN listening on my port: when I kill it, a sleep process replaces it and listens on the same port: When I finally kill this sleep process, the port is released and I can restart successfully my server. That could be a solution to get my port released, but I fear that automatically killing processes in order to release a port is a bit risky
Best Answer
Since you only do this manually, you may have to add another check.
and kill the pid associated with your open socket, even if it is bash or sh.
Also, you mentioned that most of the time SIGTERM works. If that's the case, your app should catch the SIGTERM and jump into some graceful exit code that RSTs all open connections and then closes the socket.
HTH