Linux – wget giving Segmentation fault

debianlinuxsegmentation-faultvpswget

I recently did apt-get update followed by apt-get upgrade on my Debian VPS server. Suddenly (at least) wget stopped working with error Segmentation fault. It's not even trying to do something. So after some googling I've installed gdb and tried to debug wget:

~# gdb wget
(gdb) run
Starting program: /usr/bin/wget
Error while mapping shared library sections:
'/lib/libc.so.5': not in executable format: File truncated
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()

Looking at /lib/ gives me this:

~# ls -l /lib/
total 288
lrwxrwxrwx  1 root root    21 Oct 14 08:52 cpp -> /etc/alternatives/cpp
drwxr-xr-x  2 root root  4096 May  7  2015 discover
drwxr-xr-x  2 root root  4096 May  7  2015 ifupdown
drwxr-xr-x  2 root root  4096 May  7  2015 init
-rwxr-xr-x  1 root root 71416 Oct  5  2014 klibc-IpHGKKbZiB_yZ7GPagmQz2GwVAQ.so
-rwxr-xr-x  1 root root 23904 Jan 26 22:08 libc.so.5
lrwxrwxrwx  1 root root    17 Nov  8  2014 libip4tc.so.0 -> libip4tc.so.0.1.0
-rw-r--r--  1 root root 31416 Nov  8  2014 libip4tc.so.0.1.0
lrwxrwxrwx  1 root root    17 Nov  8  2014 libip6tc.so.0 -> libip6tc.so.0.1.0
-rw-r--r--  1 root root 31416 Nov  8  2014 libip6tc.so.0.1.0
lrwxrwxrwx  1 root root    15 Nov  8  2014 libipq.so.0 -> libipq.so.0.0.0
-rw-r--r--  1 root root 10544 Nov  8  2014 libipq.so.0.0.0
lrwxrwxrwx  1 root root    16 Nov  8  2014 libiptc.so.0 -> libiptc.so.0.0.0
-rw-r--r--  1 root root  5816 Nov  8  2014 libiptc.so.0.0.0
lrwxrwxrwx  1 root root    20 Nov  8  2014 libxtables.so.10 -> libxtables.so.10.0.0
-rw-r--r--  1 root root 51896 Nov  8  2014 libxtables.so.10.0.0
drwxr-xr-x  3 root root  4096 May  7  2015 lsb
drwxr-xr-x  2 root root  4096 May  7  2015 modprobe.d
drwxr-xr-x  3 root root  4096 May  7  2015 modules
drwxr-xr-x  2 root root  4096 May  7  2015 startpar
drwxr-xr-x  8 root root  4096 Jan 27 15:19 systemd
drwxr-xr-x 15 root root  4096 May  7  2015 terminfo
drwxr-xr-x  4 root root  4096 Jan 27 15:13 udev
drwxr-xr-x  4 root root 12288 Jan 27 15:20 x86_64-linux-gnu
drwxr-xr-x  2 root root  4096 May  7  2015 xtables

As we can see that "something happened" to libc.so.5 on Jan-26 but I don't really know what.

I've checked dmesg | grep "wget":

(truncated)
[247973.065698] wget[9959]: segfault at 0 ip           (null) sp 00007ffc31e37558 error 14 in wget[400000+63000]
[248272.905735] wget[10886]: segfault at 0 ip           (null) sp 00007ffcf3e14038 error 14 in wget[400000+63000]
[248572.716856] wget[11824]: segfault at 0 ip           (null) sp 00007fffdcc9e2d8 error 14 in wget[400000+63000]
[248873.509595] wget[12742]: segfault at 0 ip           (null) sp 00007ffd84ad95d8 error 14 in wget[400000+63000]
[249173.463737] wget[13689]: segfault at 0 ip           (null) sp 00007ffca87ba518 error 14 in wget[400000+63000]
[249472.948078] wget[14664]: segfault at 0 ip           (null) sp 00007ffe54fd32a8 error 14 in wget[400000+63000]
[249776.082815] wget[15640]: segfault at 0 ip           (null) sp 00007ffecc36c758 error 14 in wget[400000+63000]
[250072.819988] wget[16595]: segfault at 0 ip           (null) sp 00007fff07f4f9d8 error 14 in wget[400000+63000]
[250373.130105] wget[17570]: segfault at 0 ip           (null) sp 00007ffcec9c01d8 error 14 in wget[400000+63000]
[250672.640891] wget[18493]: segfault at 0 ip           (null) sp 00007ffe9f78cc08 error 14 in wget[400000+63000]
[250975.950585] wget[19412]: segfault at 0 ip           (null) sp 00007fff46300128 error 14 in wget[400000+63000]

Server is a VPS running on Debian at Wedos.com:

~# uname -a
Linux server1 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux

Could anybody help me what's going on and how to fix segfault? It's production server with live traffic.

Thanks!

Best Answer

The key here is the line

'/lib/libc.so.5': not in executable format: File truncated

The C library is one of the core components of a Unix-like system, including Linux. That it is corrupted means that virtually nothing will work. You are lucky that your system is limping along, but it's quite possible that starting or restarting anything will fail.

DO NOT REBOOT THE SYSTEM. In its current state, it almost certainly won't come back up.

Especially because this is a VPS, and because this happened together with an upgrade, I'm wondering if maybe you ran out of resources like disk space or inodes. I wouldn't put it beyond something like this to have broken more things on your system, so you may need to implement some form of disaster recovery. The easiest option might very well be to restore your most recent backup.

At the very least, you will have to reinstall the C library. On my Debian system, this is provided by the libc6 family of packages, primarily libc6. You can run dpkg -S libc.so to list all packages that provide files with names containing the substring libc.so. If you can't run this on the VPS in question, set up a separate system (could be a low-provisioned VM somewhere; that doesn't really matter much) with the same release of Debian, update the package lists, and run the command there; it will tell you which packages are candidates for reinstallation.

Once you have the very basics back to working, you may want to check out How to verify that package-installed files match originals? for how to verify that (at least mostly) nothing else has been corrupted as well.

Personally, especially since presumably this is an important system (since you state it is a "production server with live traffic"), I'd probably prefer just initiating whatever disaster recovery plans you have in place. Chances are that it will be easier and quicker than trying to repair things in-place and never quite knowing what errors you might possibly have missed. This is really just a variation of needing to nuke from orbit to be sure.

Related Topic