Hadoop – Master not running

hadoophbase

hi folks I am getting this exception while running HBase in my master logs and HMaster is not running.

2012-05-20 11:54:38,206 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/23.21.190.123:2181
2012-05-20 11:54:38,236 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to localhost/23.21.190.123:2181, initiating session
2012-05-20 11:54:38,291 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server localhost/23.21.190.123:2181, sessionid = 0x1376a1960900000, negotiated timeout = 180000
2012-05-20 11:54:38,323 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=Master, sessionId=ip-10-28-213-145.ec2.internal:60000
2012-05-20 11:54:38,350 INFO org.apache.hadoop.hbase.metrics: MetricsString added: revision
2012-05-20 11:54:38,350 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUser
2012-05-20 11:54:38,350 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsDate
2012-05-20 11:54:38,350 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUrl
2012-05-20 11:54:38,350 INFO org.apache.hadoop.hbase.metrics: MetricsString added: date
2012-05-20 11:54:38,350 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsRevision
2012-05-20 11:54:38,350 INFO org.apache.hadoop.hbase.metrics: MetricsString added: user
2012-05-20 11:54:38,350 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsVersion
2012-05-20 11:54:38,350 INFO org.apache.hadoop.hbase.metrics: MetricsString added: url
2012-05-20 11:54:38,350 INFO org.apache.hadoop.hbase.metrics: MetricsString added: version
2012-05-20 11:54:38,350 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
2012-05-20 11:54:38,351 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
2012-05-20 11:54:38,351 INFO org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
2012-05-20 11:54:38,452 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Master=ip-10-28-213-145.ec2.internal:60000
2012-05-20 11:54:40,150 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/23.21.190.123:54310. Already tried 0 time(s).
2012-05-20 11:54:41,151 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/23.21.190.123:54310. Already tried 1 time(s).
2012-05-20 11:54:42,153 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/23.21.190.123:54310. Already tried 2 time(s).
2012-05-20 11:54:43,155 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/23.21.190.123:54310. Already tried 3 time(s).
2012-05-20 11:54:44,156 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/23.21.190.123:54310. Already tried 4 time(s).
2012-05-20 11:54:45,157 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/23.21.190.123:54310. Already tried 5 time(s).
2012-05-20 11:54:46,159 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/23.21.190.123:54310. Already tried 6 time(s).
2012-05-20 11:54:47,160 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/23.21.190.123:54310. Already tried 7 time(s).
2012-05-20 11:54:48,161 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/23.21.190.123:54310. Already tried 8 time(s).
2012-05-20 11:54:49,162 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/23.21.190.123:54310. Already tried 9 time(s).
2012-05-20 11:54:49,165 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
java.net.ConnectException: Call to localhost/23.21.190.123:54310 failed on connection exception: java.net.ConnectException: Connection refused
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:767)
    at org.apache.hadoop.ipc.Client.call(Client.java:743)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy5.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
    at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:113)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:215)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:177)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)
    at org.apache.hadoop.hbase.util.FSUtils.getRootDir(FSUtils.java:364)
    at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:81)
    at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:346)
    at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:282)
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304)
    at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:860)
    at org.apache.hadoop.ipc.Client.call(Client.java:720)
    ... 16 more
2012-05-20 11:54:49,168 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2012-05-20 11:54:49,168 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads
2012-05-20 11:54:49,168 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60000
2012-05-20 11:54:49,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60000: exiting
2012-05-20 11:54:49,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60000: exiting
2012-05-20 11:54:49,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60000: exiting
2012-05-20 11:54:49,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60000: exiting
2012-05-20 11:54:49,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000: exiting
2012-05-20 11:54:49,170 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60000: exiting
2012-05-20 11:54:49,170 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60000: exiting
2012-05-20 11:54:49,170 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60000: exiting
2012-05-20 11:54:49,170 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60000: exiting
2012-05-20 11:54:49,170 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60000: exiting
2012-05-20 11:54:49,171 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60000
2012-05-20 11:54:49,194 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
2012-05-20 11:54:49,217 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2012-05-20 11:54:49,217 INFO org.apache.zookeeper.ZooKeeper: Session: 0x1376a1960900000 closed
2012-05-20 11:54:49,217 INFO org.apache.hadoop.hbase.master.HMaster: HMaster main thread exiting

I have searched alot but unable to find the solution.This is my /etc/hosts file

127.0.0.1   localhost hbase-system
23.21.190.123   hbase.com.com hbase localhost

I have tried it even by commenting the localhost entry i-e # 127.0.0.1 localhost but didn't able to run HMaster.
This is my hbase-site.xml

<configuration>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:54310/hbase</value>
</property>

And this is my hadoop core-site.xml

<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/app/hadoop/tmp</value>
</property><property>
<name>fs.default.name</name>
<value>hdfs://localhost:54310</value>
</property>
</configuration>

Please help me if anyone get the problem
Thanks

Best Answer

  1. Add hadoop core file from the hadoop home to hbase lib folder
  2. Check the version of hbase in the clusters. If there is a difference it might giv error.
  3. Use Ip Address instead of localhost
  4. Check ssh connection.
Related Topic