Ubuntu – MCollective daemon not binding a network socket

mcollectivepuppetUbuntu

I’ve just installed PE using on Ubuntu 10.04 LTS using puppet-enterprise-2.0-ubuntu-10.04-amd64.tar.gz (downloaded from http://puppetlabs.com/misc/pe-files/) and right now the same machine is running the master, the console, and the agent roles.

I seem to have an issue with the MCollective server, which even though it seems to be starting fine, doesn’t open up port 61613 as it should.

Daemon starting fine

$ sudo /etc/init.d/pe-mcollective start
 * Starting mcollective daemon                  [ OK ]

Showing up is PS

$ ps aux | grep mcol | grep -v grep
root     25636  0.0  0.5  64680 12056 ?        S    07:21   0:00 /opt/puppet/bin/ruby /opt/puppet/sbin/mcollectived --pid /var/run/pe-mcollective.pid

Pid file contains correct value

$ sudo cat /var/run/pe-mcollective.pid
25636

However I don’t see any process listening on port 61613

$ sudo netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      2395/mysqld     
tcp        0      0 127.0.0.1:11211         0.0.0.0:*               LISTEN      25542/memcached 
tcp        0      0 0.0.0.0:8140            0.0.0.0:*               LISTEN      25620/pe-httpd  
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      10096/apache2   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2345/sshd       
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      2822/master     
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      25620/pe-httpd

When I look at the logs I see errors in both mcollective server and client logs

$ sudo tail -f /var/log/pe-*/*
==> /var/log/pe-puppet-dashboard/mcollective_client.log <==
I, [2011-12-20T07:26:22.731870 #30944]  INFO -- : stomp.rb:79:in `on_connectfail' Connction to stomp://mcollective@puppetmaster.example.org:61613 failed on attempt 1646

==> /var/log/pe-mcollective/mcollective.log <==
I, [2011-12-20T07:26:24.852641 #25636]  INFO -- : stomp.rb:79:in `on_connectfail' Connction to stomp://mcollective@puppetmaster.example.org:61613 failed on attempt 20

When I inspect the mcollective ruby script /opt/puppet/sbin/mcollectived I can see it references the configuration file /etc/puppetlabs/mcollective/server.cfg which has the following configuration directives:

plugin.stomp.pool.size = 1
plugin.stomp.pool.host1 = puppetmaster.example.org
plugin.stomp.pool.port1 = 61613
plugin.stomp.pool.user1 = mcollective
plugin.stomp.pool.password1 = ***************
plugin.stomp.pool.ssl1 = true
plugin.stomp.base64 = true

I can resolve plugin.stomp.pool.host1, if I change plugin.stomp.pool.port1 for instance to 61614 I see it reflected in the mcollective.log errors mentioned above. I set loglevel = debug but it doesn’t give me more hints. I ran the script manually (/opt/puppet/bin/ruby /opt/puppet/sbin/mcollectived) but it doesn’t give me any clue either.

Q: How can I troubleshoot this issue further?

Best Answer

The issue was with java keystore:

pe-activemq started fine

$ sudo /etc/init.d/pe-activemq start
 * pe-activemq started

However by watching PS I noticed activemq tried to start but failed almost immediately. I modified the startup script to do some debugging:

start() {
    echo -n "Starting $APP_NAME... "
    getpid
    if [ "X$pid" = "X" ]
    then
        # original command
        # COMMAND_LINE="$CMDNICE $WRAPPER_CMD $WRAPPER_CONF wrapper.syslog.ident=$APP_NAME wrapper.pidfile=$PIDFILE wrapper.daemonize=TRUE $ANCHORPROP $IGNOREPROP $LOCKPROP"
        # custom command with deamonize=FALSE
        COMMAND_LINE="$CMDNICE $WRAPPER_CMD $WRAPPER_CONF wrapper.syslog.ident=$APP_NAME wrapper.pidfile=$PIDFILE wrapper.daemonize=FALSE $ANCHORPROP $IGNOREPROP $LOCKPROP"
        echo "executing [$COMMAND_LINE]"

Starting pe-activemq gave the following error

jvm 1    | ERROR | Failed to start ActiveMQ JMS Message Broker. Reason: java.io.IOException: Transport Connector could not be registered in JMX: Failed to bind to server socket: stomp+ssl://0.0.0.0:61613 due to: java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.DefaultSSLContextImpl)

It looked like an issue with the keystore. The /opt/puppet/activemq/conf/activemq-wrapper.conf configuration file gives you the following details:

set.default.ACTIVEMQ_BASE=/opt/puppet/activemq
wrapper.working.dir=/var/log/pe-activemq
[...]
# Enable SSL of the Stomp Connection (Note, this provides encryption only as per #10596)
wrapper.java.additional.7=-Djavax.net.ssl.keyStorePassword=puppet
wrapper.java.additional.8=-Djavax.net.ssl.keyStore=%ACTIVEMQ_BASE%/conf/broker.ks
# The trust store need not be present.
wrapper.java.additional.9=-Djavax.net.ssl.trustStorePassword=puppet
wrapper.java.additional.10=-Djavax.net.ssl.trustStore=%ACTIVEMQ_BASE%/conf/broker.ts

FYI /opt/puppet/activemq/conf is a symlink to /etc/puppetlabs/activemq In that directory, the broket.ts file exists, but broker.ks was missing

Based on this page (http://activemq.apache.org/how-do-i-use-ssl.html), I did the following

$ cd /etc/puppetlabs/activemq
$ sudo keytool -genkey -alias broker -keyalg RSA -keystore broker.ks
=> provided puppet password everywhere

Then activemq was starting fine

jvm 1    |  INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[/opt/puppet/activemq/data/kahadb]
jvm 1    |  INFO | KahaDB is version 3
jvm 1    |  INFO | Recovering from the journal ...
jvm 1    |  INFO | Recovery replayed 1 operations from the journal in 0.022 seconds.
jvm 1    |  INFO | ActiveMQ 5.5.0 JMS Message Broker (localhost) is starting
jvm 1    |  INFO | For help or more information please see: http://activemq.apache.org/
jvm 1    |  INFO | Installing StaticsBroker
jvm 1    |  INFO | Starting StatisticsBroker
jvm 1    |  INFO | Listening for connections at: tcp://myserver:61616
jvm 1    |  INFO | Connector openwire Started
jvm 1    |  INFO | Listening for connections at: stomp+ssl://myserver:61613
jvm 1    |  INFO | Connector stomp+ssl Started
jvm 1    |  INFO | ActiveMQ JMS Message Broker (localhost, ID:myserver-44300-1324455724257-0:1) started
jvm 1    |  INFO | jetty-7.1.6.v20100715
jvm 1    |  INFO | ActiveMQ WebConsole initialized.
jvm 1    |  INFO | Initializing Spring FrameworkServlet 'dispatcher'
jvm 1    |  INFO | ActiveMQ Console at http://0.0.0.0:8161/admin
jvm 1    |  INFO | WebApp@2109578614 at http://0.0.0.0:8161/camel
jvm 1    |  INFO | WebApp@2109578614 at http://0.0.0.0:8161/demo
jvm 1    |  INFO | WebApp@2109578614 at http://0.0.0.0:8161/fileserver
jvm 1    |  INFO | Started SelectChannelConnector@0.0.0.0:8161

ISSUE RESOLVED

Related Topic