PuppetDB: Connection refused

I'm trying to get PuppetDB to run on my puppetmaster machine. The problem is that I always get the following message when doing an agent run:

err: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit 'replace facts' command for handsoff.dc0.testing.de to PuppetDB at handsoff:8081: Connection refused - connect(2)

The puppetmaster, puppetdb and database are all running on the same host "handsoff".

This is my /etc/puppet/puppetdb.conf

[main]
server = handsoff
port = 8081

My /etc/puppetdb/jetty.ini

[jetty]
# Hostname or IP address to listen for clear-text HTTP.  Default is localhost
# host = <host>

# Port to listen on for clear-text HTTP.
port = 8080


# The following are SSL specific settings. They can be configured
# automatically with the tool `puppetdb ssl-setup`, which is normally
# ran during package installation.

# The host or IP address to listen on for HTTPS connections
ssl-host = localhost

# The port to listen on for HTTPS connections
ssl-port = 8081

# Private key path
ssl-key = /etc/puppetdb/ssl/private.pem

# Public certificate path
ssl-cert = /etc/puppetdb/ssl/public.pem

# Certificate authority path
ssl-ca-cert = /etc/puppetdb/ssl/ca.pem

certificate-whitelist = /etc/puppetdb/ssl-whitelist

And the ssl-whitelist file:

localhost
handsoff
handsoff.dc0.testing.de

lsof shows me the listening ports

# lsof -i -P | grep puppetdb | grep 80
java     2549      puppetdb   20u  IPv6   4982      0t0  TCP localhost:8080 (LISTEN)
java     2549      puppetdb   34u  IPv6   9353      0t0  TCP localhost:8081 (LISTEN)

In my /etc/hosts I have these lines

127.0.0.1   localhost
127.0.1.1   handsoff handsoff.dc0.testing.de

So from my point of view puppet should be able to connect to puppetdb on port 8081 through localhost and through handsoff. (I tested both, both with the same results)

Any ideas what might be wrong with this setup?

Edit: I also tried to set this in my puppetdb.conf:

[main]
server = localhost
port = 8081

But then I get err: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit 'replace facts' command for handsoff.dc0.testing.de to PuppetDB at localhost:8081: hostname does not match the server certificate

I'm using self-signed certificates (didn't change anything on that after I did the install with apt-get). My guess was that the whitelisting would allow me to use the certificate this way. Is that wrong?