Apache reverse proxy with Jenkins and TLS/SSL

I am running a Jenkins CI server in a VM (OS: Debian 8). I took over this VM from the previous admin. As I decided to secure the running Jenkins over TLS/SSL, I requested a certificate and set up an Apache (2.4.10) as a reverse proxy.

I referred to this Jenkins Wiki instructions for configuration: https://wiki.jenkins-ci.org/display/JENKINS/Running+Jenkins+behind+Apache

As in the complete example on that page I configured my Apache reverse proxy as follows:

NameVirtualHost *:80
NameVirtualHost *:443

<VirtualHost *:80>
    ServerName myhostname.tld
    Redirect permanent / https://myhostname.tld/
</VirtualHost>

<VirtualHost *:443>

    ServerName myhostname.tld:443

    SSLEngine on

    SSLCertificateFile      /path-to-my-cert/cert.pem
    SSLCertificateKeyFile   /path-to-my-key/keyfile.key

    ProxyRequests     Off
    ProxyPreserveHost On
    AllowEncodedSlashes NoDecode

    <Proxy *>
            Order deny,allow
            Allow from all
    </Proxy>

    ProxyPass         /  http://localhost:8080/ nocanon
    ProxyPassReverse  /  http://localhost:8080/
    ProxyPassReverse  /  http://myhostname.tld/

    RequestHeader set X-Forwarded-Proto "https"
    RequestHeader set X-Forwarded-Port "443"

</VirtualHost>

As far as I can see, this seems to be the correct setup for a reverse proxy with TLS/SSL regarding to the documentation. I can perfectly access the Jenkins server via HTTPS but the only thing that won't work, is the redirect from HTTP to HTTPS.

I did set up another Apache server, which isn't acting as a reverse proxy, where the similar redirect is working without problems from HTTP to HTTPS. So I assume, it seems to b a config problem with the reverse proxy commands in my vhost config file.

I searched a lot for answers and I found a similar question here: Correct Apache reverse proxy config with SSL for Jenkins and Sonar

Unfortunately, the solution posted there is, what I already have included in my config, as the accepted answer there seems to be the code from the Jenkins wiki site. I also found a lot of other proposed solutions, which all seem to be a huge mixture of different approaches (some are better, some are worse) and therefore I feel rather lost in confusing hints and wrong configurations.

What I also found is the Apache wiki site: https://wiki.apache.org/httpd/RedirectSSL

Where the official redirect solution also seems to be, what I am already using.

I did also check the backend config inside Jenkins itself and I did find the "Jenkins URL" field where the URL was http://. But changing this to https:// also didn't have any effects.

I double checked my config again and again and I also took into account checking for a trailing slash on my domains, but also there seems to be everything correct.

I'd be thankful for any help or further resources.

Thanks.

EDIT:

So, again, I am really sorry for the delay due to longer illness recently.

Anyway, I cloned the VM and I did set up a new certificate. There is the same, clean vhost config with the same redirect. I am able, to call the site perfectly with HTTPS without any errors. But the HTTP -> HTTPS redirect still won't work.

Here is the output I got from curl -I -H "Host:..." http://...

HTTP/1.1 200 OK
Date: Mon, 06 Mar 2017 14:29:44 GMT
X-Content-Type-Options: nosniff
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: no-cache,no-store,must-revalidate
X-Hudson-Theme: default
Content-Type: text/html;charset=UTF-8
Set-Cookie: JSESSIONID.8d3ecf0e=47s4cwdffsf4t3qnwix3n2;Path=/;HttpOnly
X-Hudson: 1.395
X-Jenkins: 2.49
X-Jenkins-Session: 67e643e
X-Hudson-CLI-Port: 56889
X-Jenkins-CLI-Port: 56889
X-Jenkins-CLI2-Port: 56889
X-Frame-Options: sameorigin
X-Instance-Identity: MIIBIjANBgkqhk....
X-SSH-Endpoint: jenkinsdomain.tld:47955
Transfer-Encoding: chunked
Server: Jetty(9.2.z-SNAPSHOT)

I am absolutely sure, there is no browser caching problem affecting this. I used different browser, cleared all caches, used private sessions, etc. Nothing works.

EDIT 2:

After some digging around and various attempts, I am one step closer to the point of failure.

Regarding to different sources (I can't post all these links here, as I have not enough reputation points), what I did was:

I found the jenkins config file and edited the JENKINS_ARGS line, which holds different parameters for booting up the Jenkins server. I added this param:

--httpListenAddress=127.0.0.1

So my boot up line there looks like this:

JENKINS_ARGS="--webroot=/var/cache/jenkins/war --httpPort=$HTTP_PORT --httpListenAddress=127.0.0.1 --ajp13Port=$AJP_PORT"

Where:

$HTTP_PORT=8080

$AJP_PORT is dasabled (set to -1 in the config file)

I found different tutorials (e.g. from the Jenkins site itself and on Digital Ocean and others) and all of them went the same way. The step further here is, that Jenkins now serves only HTTPS! When trying to access the site on HTTP, I get the error, that my connection is refused.

Thus, I can verify, that Jenkins is served via the Apache reverse proxy on HTTPS and that Apache seems to try forwarding HTTP. Which makes me a little bit more confused, as the vhost config is really clean and the instructions there seem to get executed (if they weren't , HTTPS wouldn't work). Also, this is the preferred way, how it has to be done, referencing the official Apache documentation.

I tried some other things, just to see, if it has any effect there. I added a header rewrite like this one:

<Location />
  ProxyPassReverse /
  Order deny,allow
  Allow from all
</Location>
Header edit Location ^http://www.example.com/ https://www.example.com/

This is proposed on the Hudson server documentation (as Jenkins is based upon Hudson). Didn't work.

I tried rewriting from HTTP to HTTPS, like the official Apache documentation points out, via mod_rewrite which also didn't work (this was also proposed here by another user).

In any other configuration and attempt, to rewrite the URL, I end up in exactly the same error (ERR_CONNECTION_REFUSED).

So, right now, I am trying to understand, what makes Apache refusing to redirect to Jenkins? Jenkins is listening on port 8080 and on address 127.0.0.1 (which was 0.0.0.0 before, I changed it like I already described).

Adding some debug output. When I curl localhost from the server itself, I do get:

# curl -I http://localhost

HTTP/1.1 301 Moved Permanently
Date: Wed, 08 Mar 2017 10:04:31 GMT
Server: Apache/2.4.10 (Debian)
Location: https://jenkins.domain.tld/
Content-Type: text/html; charset=iso-8859-1

Which seems correct, as this seems to be my redirect to my HTTPS site.

When I curl localhost:8080 (where Jenkins is bound to) from the server itself, I do get my the correct answer:

# curl -I http://localhost:8080

HTTP/1.1 200 OK
Date: Wed, 08 Mar 2017 10:07:41 GMT
X-Content-Type-Options: nosniff
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: no-cache,no-store,must-revalidate
X-Hudson-Theme: default
Content-Type: text/html;charset=UTF-8
Set-Cookie: JSESSIONID.9e9c97x3=2wx6r12gl694bf51b7ln8vymm;Path=/;HttpOnly
X-Hudson: 1.395
X-Jenkins: 2.49
X-Jenkins-Session: 66c04def
X-Hudson-CLI-Port: 39397
X-Jenkins-CLI-Port: 39397
X-Jenkins-CLI2-Port: 39397
X-Frame-Options: sameorigin
X-Instance-Identity: MVIBIjAN....
X-SSH-Endpoint: jenkins.domain.tld:46063
Transfer-Encoding: chunked
Server: Jetty(9.2.z-SNAPSHOT)

The error occurs from remote clients:

# curl -I http://jenkins.domain.tld
curl: (7) Failed to connect to jenkins.domain.tld port 80: Connection refused

Also, just to be sure:

# a2enmod proxy_http

Considering dependency proxy for proxy_http:
Module proxy already enabled
Module proxy_http already enabled