Linux – How to troubleshoot an unexpected Linux shutdown

linuxredhatshutdown

I have two RHEL6.3 servers that just shutdown at the same exact time (25 seconds apart) for no apparent reason. They're on conditioned power, along with a number of other servers that didn't shut down, so it can't be the power. The room is properly chilled and them both shutting off at the exact same time due to temperature seems unlikely.

At the time of shutdown, both servers have the following in their /var/log/secure. I don't know what it means but found it peculiar.

Apr 10:42:52 localhost polkitd(authority=local): Unregistered Authentication Agent for session /org/freedesktop/ConsoleKit/Session1 (system bus name :1.25, object path /org/gnome/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)

In /var/log/messages, both systems have what seems like they received a clean shutdown request

Apr 10 10:42:52 localhost init: tty (/dev/tty2) main process (6183) killed by TERM signal
Apr 10 10:42:52 localhost init: tty (/dev/tty3) main process (6186) killed by TERM signal
Apr 10 10:42:52 localhost init: tty (/dev/tty4) main process (6188) killed by TERM signal
Apr 10 10:42:52 localhost init: tty (/dev/tty5) main process (6190) killed by TERM signal
Apr 10 10:42:52 localhost init: tty (/dev/tty6) main process (6192) killed by TERM signal

So I checked last to see if anyone logged in to do that and both have this entry with no one logging in prior for days.

reboot     system boot   2.6.32-279.el6.x Thu Apr 10 10:42 - 10:42  (00:00)

So if no one logged in to shut it down and I had two guys that were there witness the servers shutdown and they confirmed no one touched either of the servers, what else could cause this shutdown? Where else should I look for clues?

Best Answer

Assuming this is a kernel oops (kernel panic), you need to capture the output from the server console to understand what exactly happened. You can do so using:

Physical access to console, typically you'd need to take a picture of the screen to capture the panic (least reliable way to capture)
Remote IP KVM (make a screenshot)
Serial console (log console output via serial cable from problem computer to another) http://www.cyberciti.biz/faq/centos-rhel-6-install-serial-console/
Network console (log console output to another server in your network) https://fedoraproject.org/wiki/Netconsole

Things to double check on all servers to be successful:

Check kernel.panic option in /etc/sysrq.conf, that controls the timeout after which Linux will reboot after kernel panic
Check the kernel log level via kernel.printk in /etc/sysrq.conf, recommended with more debug would be: echo 'kernel.printk = 8 4 1 7' >> /etc/sysctl.conf

Related Solutions

Linux Bash – How to Sort du -h Output by Size

As of GNU coreutils 7.5 released in August 2009, sort allows a -h parameter, which allows numeric suffixes of the kind produced by du -h:

du -hs * | sort -h

If you are using a sort that does not support -h, you can install GNU Coreutils. E.g. on an older Mac OS X:

brew install coreutils
du -hs * | gsort -h

From sort manual:

-h, --human-numeric-sort compare human readable numbers (e.g., 2K 1G)

Linux – How to run a server on port 80 as a normal user on Linux

Short answer: you can't. Ports below 1024 can be opened only by root. As per comment - well, you can, using CAP_NET_BIND_SERVICE, but that approach, applied to java bin will make any java program to be run with this setting, which is undesirable, if not a security risk.

The long answer: you can redirect connections on port 80 to some other port you can open as normal user.

Run as root:

# iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080

As loopback devices (like localhost) do not use the prerouting rules, if you need to use localhost, etc., add this rule as well (thanks @Francesco):

# iptables -t nat -I OUTPUT -p tcp -d 127.0.0.1 --dport 80 -j REDIRECT --to-ports 8080

NOTE: The above solution is not well suited for multi-user systems, as any user can open port 8080 (or any other high port you decide to use), thus intercepting the traffic. (Credits to CesarB).

EDIT: as per comment question - to delete the above rule:

# iptables -t nat --line-numbers -n -L

This will output something like:

Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    REDIRECT   tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:8080 redir ports 8088
2    REDIRECT   tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:80 redir ports 8080

The rule you are interested in is nr. 2, so to delete it:

# iptables -t nat -D PREROUTING 2

Best Answer

Related Solutions

Linux Bash – How to Sort du -h Output by Size

Linux – How to run a server on port 80 as a normal user on Linux

Related Topic