Nginx – How to fix OpenSSL Padding Oracle vulnerability (CVE-2016-2107) for nginx on debian jessie

debiannginxopenssl

As far as i understood, it should be sufficient to upgrade openssl (done a long time ago, now installed all available updates again (no openssl there)) and restart nginx.
I even tried to stop nginx fully (verified it with ps) and start it again.

But ssllabs still tells me, that i am vulnerable. What else do i need to do, or what can be causing that its still vulnerable?

versions:

ii  nginx                              1.9.10-1                          all          small, powerful, scalable web/proxy server
ii  nginx-common                       1.9.10-1                          all          small, powerful, scalable web/proxy server - common files
ii  nginx-full                         1.9.10-1                          amd64        nginx web/proxy server (standard version)
ii  openssl                            1.0.1t-1+deb8u2                   amd64        Secure Sockets Layer toolkit - cryptographic utility

ii  libssl-dev:amd64                   1.0.1t-1+deb8u2                   amd64        Secure Sockets Layer toolkit - development files
ii  libssl-doc                         1.0.1t-1+deb8u2                   all          Secure Sockets Layer toolkit - development documentation
ii  libssl1.0.0:amd64                  1.0.1t-1+deb8u2                   amd64        Secure Sockets Layer toolkit - shared libraries
ii  libssl1.0.2:amd64                  1.0.2f-2                          amd64        Secure Sockets Layer toolkit - shared libraries

lsof related to nginx

lsof 2>/dev/null |grep -i libssl|grep nginx
nginx     17928              root  mem       REG              251,0    430560    2884885 /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2
nginx     17929          www-data  mem       REG              251,0    430560    2884885 /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2
nginx     17930          www-data  mem       REG              251,0    430560    2884885 /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2
nginx     17932          www-data  mem       REG              251,0    430560    2884885 /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2
nginx     17933          www-data  mem       REG              251,0    430560    2884885 /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2

Best Answer

I got it.

I installed certbot from debian unstable, which installed 1.0.2f-2. unstable is pinned to priority "-100" (do not install from unstable unless requested with -t unstable). This means the version is between the jessie version 1.0.0X-Y and the current unstable version 1.0.2.h-1. This prevented an upgrade to the next version in unstable, while the upgrade in stable is an "older" version with respect to the version number.

Related Solutions

Ubuntu – Compile apache 2.2.23 on Ubuntu 12.04.1 LTS with openssl error

Compile openssl, then compile and link Apache against --with-ssl=/usr/local/ssl (the default path). Don't forget "--enable-so"

I had many issue with the maintainer libs from debian distros and always compile openssl from scratch.

If you are using PHP and use --with-openssl=/usr/local/ssl, the same libraries have to be used.

Debian – Building curl, httpd and others with custom openssl build, while avoiding default system openssl

There are two different linker operations involved, build time (used to find libraries and "symbols", performed by ld or equivalent), and run time (performed by ld.so) which loads libraries into process memory and performs relocations.

When you use -L during build, ld only confirms the libraries and required symbols (typically functions which comprise the API) are available. At run time, libraries with different paths can be used. LD_LIBRARY_PATH, LD_PRELOAD and RPATH (-R or -rpath) are some of the ways of controlling that – the last one being the way to specify this at build time (so it's a property of the binary, and less dependent on the user environment).

With any library, the features, API, and ABI can change, leading to errors or instability if the linker finds or loads the "wrong" binary. Even though the symbols can match as expected (or you would get an error before execution), the wheels may come off during process execution. There are solutions to this, versioning of the library filename, and versioning of the symbols in a library, though those won't be of much use here. (A complication is that the SHLIB_MAJOR/SHLIB_MINOR in all the OpenSSL 1.0.x series is "1.0.0", so those libraries cannot be readily distinguished unless you hack the Makefile.)

Curl itself checks for this kind of problem, it compares the build-time libcurl version to the run-time version, and may issue:

WARNING: curl and libcurl versions do not match. Functionality may be affected.

A contemporary ld will do similar at build time when it notes potentially conflicting libraries in dependencies.

For a clean build, with no run time environment trickery, you can:

build OpenSSL making sure an RPATH is explicit, this is because libssl.so links against libcrypto.so, and install to a specific path
build the package (i.e. curl) with the correct link-time and run-time specification so that it finds the specific libssl.so/libcrypto.so

A cheap and nasty way to fix the problem would be to usr chrpath to alter the ELF RPATH in binaries after building, I don't recommend this generally, and definitely not for curl, read on.

Now for the complications: curl supports lots of protocols, some of those protocols are provided via other libraries, and where the protocol supports TLS (or uses crypto functions) it will tend to also link against libssl.so and/or libcrypto.so, e.g. for LDAP and SSH. What you do not want is:

  curl ↦ libcurl.so → libldap.so → libssl.so
       ↦ lib...
       ↦ libcrypto.so
       ↳ libssl.so

where curl itself would use one version, but libldap.so may expect a different version to load; this may also fail at run-time if there's an ABI incompatibility. You can be unlucky with symbol or load ordering, and end up with two (or more!) versions of a library loaded.

There are generally three ways to try to work around that:

turn off all the features you don't need when building, for curl at least LDAP and SSH2 will have to go
if that's not possible, build new versions of the dependent libraries which also link against the new OpenSSL
if that's not possible, you might be able to use .a libraries for some static linking. For curl, this is not the same thing as just using --enable-static. It can work, I've done it in the distant past, but it may not work here so I won't mention it further; and it can cause many headaches, not least with with PIC/PIE.

GnuTLS can further muddy the water, though it's not a problem to use both libraries concurrently (at least, not that I've seen so far); that doesn't hold for other libraries that aim for ABI compatibility, like LibreSSL. Curl does go to some trouble to use only a single SSL library in its build (it currently supports 8, AFAIK), but it doesn't know what for example, libssh2 might be linked against.

So, depending on versions, you can try building OpenSSL and curl respectively:

# OpenSSL
./config --prefix=/usr/local --openssldir=/usr/local/openssl no-gost \
          shared zlib no-ssl2 -fPIC -Wl,-R,/usr/local/openssl
 make depend && make && make install_sw

# curl
./configure --with-ssl=/usr/local/openssl --enable-shared \
            --enable-ldap=no --without-libssh2 \
            CFLAGS=-Wl,-R,/usr/local/openssl/lib
make && make install

(Despite documentation to the contrary, --with-ldap=no and --without-ldap have no useful effect, use --enable-ldap=no. libssh2 (for scp) is not usually on by default.)

After install, ldd /usr/local/bin/curl should contain no surprises: exactly one reference to each of the expected libssl.so and libcrypto.so.

OpenSSL build (non-autoconf) will set the RPATH correctly on the openssl binary, but it does not set it on the libraries by default. There are platform specific variations in how libraries are located (much of the RPATH handling in the OpenSSL build is a decoy: it applies to platforms other than Linux). This might not be required for curl, but it won't cause problems.

When you use autoconf's configure there is no guarantee that non-default library locations will result in a matching RPATH in the binaries, CFLAGS/LDFLAGS can usually fix that. Here though, with OpenSSL's two-step process, config doesn't quite get along with variables like LDFLAGS, so add at the end of the config line instead, and Configure handles it correctly when creating the Makefile.

You can check an ELF executable or library run-time path with:

readelf --dynamic elfbinary | grep PATH

You can debug library loading with:

LD_DEBUG=files,libs /usr/local/bin/curl ...

(Note the original question is old, and OpenSSL 1.0.x is no longer supported officially, though distributions may vary. The instructions herein are generally applicable to building special versions of OpenSSL 1.0.x for specific purposes. Building an stunnel binary to reverse-proxy an old network device so that it can work with a contemporary browser is a good example. )

You can find some further explanation and instructions in my answer to this question: Get ld to pick the correct library.

Best Answer

Related Solutions

Ubuntu – Compile apache 2.2.23 on Ubuntu 12.04.1 LTS with openssl error

Debian – Building curl, httpd and others with custom openssl build, while avoiding default system openssl

Related Topic