I am trying to configure a JBoss AS 7.1.1 cluster in domain mode with mod_cluster at the apache2 proxy. All hosts are running Ubuntu and are Rackspace Cloud Servers.
My problem is that when I hit my app at http://project.example.com/ I just get a 503
JBoss log reports:
org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Error [MEM: MEM: Can't read node: {4}] sending command STATUS to proxy proxy-ip/proxy-ip:6666, configuration will be reset
The mod_cluster management page has this information:
DUMP output
balancer: [1] Name: main-server-group Sticky: 1 [JSESSIONID]/[jsessionid] remove: 0 force: 0 Timeout: 0 maxAttempts: 1
node: [1:1],Balancer: main-server-group,JVMRoute: slave:server-two,LBGroup: [],Host: project-jb-node1,Port: 8159,Type: ajp,flushpackets: 0,flushwait: 10,ping: 10,smax: 26,ttl: 60,timeout: 0
node: [2:2],Balancer: main-server-group,JVMRoute: master:server-one,LBGroup: [],Host: project-jb-master,Port: 8009,Type: ajp,flushpackets: 0,flushwait: 10,ping: 10,smax: 26,ttl: 60,timeout: 0
node: [3:3],Balancer: main-server-group,JVMRoute: master:server-two,LBGroup: [],Host: project-jb-master,Port: 8159,Type: ajp,flushpackets: 0,flushwait: 10,ping: 10,smax: 26,ttl: 60,timeout: 0
node: [4:4],Balancer: main-server-group,JVMRoute: slave:server-one,LBGroup: [],Host: project-jb-node1,Port: 8009,Type: ajp,flushpackets: 0,flushwait: 10,ping: 10,smax: 26,ttl: 60,timeout: 0
INFO output:
Node: [1],Name: slave:server-two,Balancer: main-server-group,LBGroup: ,Host: project-jb-node1,Port: 8159,Type: ajp,Flushpackets: Off,Flushwait: 10,Ping: 10,Smax: 26,Ttl: 60,Elected: 0,Read: 0,Transfered: 0,Connected: 0,Load: 100
Node: [2],Name: master:server-one,Balancer: main-server-group,LBGroup: ,Host: project-jb-master,Port: 8009,Type: ajp,Flushpackets: Off,Flushwait: 10,Ping: 10,Smax: 26,Ttl: 60,Elected: 0,Read: 0,Transfered: 0,Connected: 0,Load: 100
Node: [3],Name: master:server-two,Balancer: main-server-group,LBGroup: ,Host: project-jb-master,Port: 8159,Type: ajp,Flushpackets: Off,Flushwait: 10,Ping: 10,Smax: 26,Ttl: 60,Elected: 0,Read: 0,Transfered: 0,Connected: 0,Load: 100
Node: [4],Name: slave:server-one,Balancer: main-server-group,LBGroup: ,Host: project-jb-node1,Port: 8009,Type: ajp,Flushpackets: Off,Flushwait: 10,Ping: 10,Smax: 26,Ttl: 60,Elected: 0,Read: 0,Transfered: 0,Connected: 0,Load: 100
After a restart the apache error.log has:
[Sun May 26 19:17:51 2013] [notice] Apache/2.2.22 (Ubuntu) mod_cluster/1.2.0.Final configured -- resuming normal operations
[Sun May 26 19:17:52 2013] [warn] manager_handler STATUS error: MEM: Can't read node
[Sun May 26 19:17:53 2013] [warn] manager_handler STATUS error: MEM: Can't read node
[Sun May 26 19:17:55 2013] [warn] manager_handler STATUS error: MEM: Can't read node
[Sun May 26 19:18:00 2013] [warn] manager_handler STATUS error: MEM: Can't read node
[Sun May 26 19:18:26 2013] [error] proxy: CLUSTER: (balancer://main-server-group). All workers are in error state
If I hit http://project.example.com/index.html I get this in the apache error.log:
[Sun May 26 19:34:33 2013] [warn] proxy: No protocol handler was valid for the URL /index.html. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.
The hostnames (project-jb-master
etc) are mapped to IPs in /etc/hosts and I can connet using telnet and can ping.
Googling around this I find lots of mentions of the instance-id so I set this in the full-ha profile:
<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">
<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
<connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp"/>
<virtual-server name="default-host" enable-welcome-root="false">
<alias name="localhost"/>
<alias name="project.example.com"/>
</virtual-server>
</subsystem>
but the original message is still there.
Best Answer
Multicast isn't currently allowed on Rackspace Cloud Servers. The traffic volume of multicast traffic ends up overloading the switching layer and it degrades network performance for all traffic.
For this reason the configuration should look something like this:
Apache config:
ServerAdvertise is not enabled
JBoss config:
proxy-list
Defines a comma delimited list of httpd proxies with which this node will initially communicate. Value should be of the form:
advertise
Using the default configuration, this property can by manipulated via the
jboss.mod_cluster.proxyList
system property.If enabled, httpd proxies will be auto-discovered via multicast announcements. This can be used either in concert or in place of a static proxyList.
mod_cluster Documentation