Forwarding logs from rsyslog to graylog over tls

centos7graylogrsyslogtlsx509

I'm trying to forward logs from rsyslog to graylog over tls.

rsyslog configuration:

# make gtls driver the default
$DefaultNetstreamDriver gtls
#
# # certificate files
$DefaultNetstreamDriverCAFile /etc/ssl/rsyslog/ca.pem
$DefaultNetstreamDriverCertFile /etc/ssl/rsyslog/rsyslog-cert.pem
$DefaultNetstreamDriverKeyFile /etc/ssl/rsyslog/rsyslog-key.pem
#
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer graylog.mydomain.com
$ActionSendStreamDriverMode 1 # run driver in TLS-only mode
*.* @@graylog.mydomain.com:6514 # forward everything to remote server

I generated the necessary certificates as described in the rsyslog documentation:

certtool --pkcs8 --generate-privkey --outfile ca-key.pem
certtool --pkcs8 --generate-self-signed --load-privkey ca-key.pem --outfile ca.pem

certtool --pkcs8 --generate-privkey --outfile graylog.key.pem 
certtool --pkcs8 --generate-request --load-privkey graylog.key.pem --outfile graylog.request.pem
certtool --pkcs8 --generate-certificate --load-request graylog.request.pem --outfile graylog.cert.pem --load-ca-certificate ca.pem --load-ca-privkey ca-key.pem

certtool --pkcs8 --generate-privkey --outfile rsyslog.key.pem 
certtool --pkcs8 --generate-request --load-privkey rsyslog.key.pem --outfile rsyslog.request.pem
certtool --pkcs8 --generate-certificate --load-request rsyslog.request.pem --outfile rsyslog.cert.pem --load-ca-certificate ca.pem --load-ca-privkey ca-key.pem

Graylog config

All certifcate files are in /etc/ssl/graylog/input-ca/

The graylog input is configured like this:

TLS cert file: /etc/ssl/graylog/input-ca/graylog-cert.pem
TLS private key file: /etc/ssl/graylog/input-ca/graylog-key.pem
TLS Client Auth Trusted Certs: /etc/ssl/graylog/input-ca

But when rsyslog pushes a log message to garylog I get this error:

2016-10-06T13:19:27.734+02:00 WARN  [AbstractNioSelector] Failed to initialize an accepted socket.
java.security.cert.CertificateParsingException: signed fields invalid
    at sun.security.x509.X509CertImpl.parse(X509CertImpl.java:1791) ~[?:1.8.0_91]
    at sun.security.x509.X509CertImpl.<init>(X509CertImpl.java:195) ~[?:1.8.0_91]
    at sun.security.provider.X509Factory.parseX509orPKCS7Cert(X509Factory.java:469) ~[?:1.8.0_91]
    at sun.security.provider.X509Factory.engineGenerateCertificates(X509Factory.java:354) ~[?:1.8.0_91]
    at java.security.cert.CertificateFactory.generateCertificates(CertificateFactory.java:462) ~[?:1.8.0_91]
    at org.graylog2.plugin.inputs.transports.util.KeyUtil.loadCertificates(KeyUtil.java:90) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.transports.util.KeyUtil.loadCertificates(KeyUtil.java:100) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.transports.util.KeyUtil.initTrustStore(KeyUtil.java:73) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.transports.AbstractTcpTransport$1.createSslEngine(AbstractTcpTransport.java:199) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.transports.AbstractTcpTransport$1.call(AbstractTcpTransport.java:186) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.transports.AbstractTcpTransport$1.call(AbstractTcpTransport.java:182) ~[graylog.jar:?]
    at org.graylog2.plugin.inputs.transports.NettyTransport$1.getPipeline(NettyTransport.java:110) ~[graylog.jar:?]
    at org.jboss.netty.channel.socket.nio.NioServerBoss.registerAcceptedChannel(NioServerBoss.java:134) [graylog.jar:?]
    at org.jboss.netty.channel.socket.nio.NioServerBoss.process(NioServerBoss.java:104) [graylog.jar:?]
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [graylog.jar:?]
    at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42) [graylog.jar:?]
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [graylog.jar:?]
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [graylog.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]

Any ideas what might be the problem ?

Best Answer

Ensure that your Graylog process can read the certificate files and key. (It's normally running as the "graylog" user, not root.) "Failed to initialize an accepted socket" is exactly the error that appears Graylog can't read these files.

You typically want your key, certificate and other files under /etc/graylog (I use /etc/graylog/keys) and owned by the graylog user. Make sure that the key is not readable to group or other.

Related Topic