Linux – How to diagnose a linux server freeze without any kernel panics or logs

linuxUbuntu

The past few months a small cluster of Ubuntu servers has been crashing for me. Most of the time when a server crashes I can find some evidence of the reason in either dmesg, syslog, or one of the various other log systems. However for this particular freeze nothing is happening at all in any of the logs. The system just literally freezes, no keyboard input is working, no ping coming from the system at all. It's offline but still drawing power.

If it was just one server I would blame the hardware but we are talking about multiple servers, with multiple generations of CPUs, RAM and motherboards all having the issues.

I've tried upgrading to various kernels (currently on 4.15) but that also did not cause the issue to stop.

What I'm mostly looking for is some way to increase the kernel logging or some other way to get some kind of information from the frozen server as to what it was doing before it froze.

Best Answer

Hard lockups are often due to power management problems. Try to:

disable, in the BIOS, any power management setting (eg: P/C states) or, alternatively, use the "maximum performance" profile;
inside the OS, use the "performance" power governor (ie: cpupower frequency-set -g performance);
use a polling idle kernel setting using the idle=poll boot parameter;
disable any C states using intel_idle.max_cstate=0 processor.max_cstate=0 kernel boot parameters.

In order to discover the cause of your lockup, you should apply one change at time. Moreover, please note that step #3 and #4 will have a significant impact of power consumption/efficiency, so you should use the suggested kernel cmdline for test/diagnose only.

Related Solutions

Linux – How to run a server on port 80 as a normal user on Linux

Short answer: you can't. Ports below 1024 can be opened only by root. As per comment - well, you can, using CAP_NET_BIND_SERVICE, but that approach, applied to java bin will make any java program to be run with this setting, which is undesirable, if not a security risk.

The long answer: you can redirect connections on port 80 to some other port you can open as normal user.

Run as root:

# iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080

As loopback devices (like localhost) do not use the prerouting rules, if you need to use localhost, etc., add this rule as well (thanks @Francesco):

# iptables -t nat -I OUTPUT -p tcp -d 127.0.0.1 --dport 80 -j REDIRECT --to-ports 8080

NOTE: The above solution is not well suited for multi-user systems, as any user can open port 8080 (or any other high port you decide to use), thus intercepting the traffic. (Credits to CesarB).

EDIT: as per comment question - to delete the above rule:

# iptables -t nat --line-numbers -n -L

This will output something like:

Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    REDIRECT   tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:8080 redir ports 8088
2    REDIRECT   tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:80 redir ports 8080

The rule you are interested in is nr. 2, so to delete it:

# iptables -t nat -D PREROUTING 2

Ubuntu System Freeze – How to Diagnose an Ubuntu System Freeze After Reboot

My first action would be to take server out of service and run a full Memtest+ run on it to check that memory is not failing. Next check SMART from HDD's for any issues. Next would be to following instructions at http://www.kernel.org/doc/Documentation/networking/netconsole.txt to capture anything like this in future.

Best Answer

Related Solutions

Linux – How to run a server on port 80 as a normal user on Linux

Ubuntu System Freeze – How to Diagnose an Ubuntu System Freeze After Reboot

Related Topic