Linux – Hadoop namenode port getting blocked

firewallhadooplinuxportUbuntu

I have installed 7 VM instances of Ubuntu 14.04 LTS servers. First instance runs the namenode service and all other 6 nodes run datanode service.I think my NameNode is getting crashed or blocked due to some issue.
After rebooting if I check JPS command output my namenode is running. In core-site.xml the fs.defaultfs property is set to hdfs://instance-1:8020.
but in the netstat -tulpn output 8020 port is not there.

this is the JPS output right after rebooting.

root@instance-1:~# jps
3017 VersionInfo
2613 NameNode
3371 VersionInfo
3313 ResourceManager
3015 Main
2524 QuorumPeerMain
2877 HeadlampServer
1556 Main
3480 Jps
2517 SecondaryNameNode
3171 JobHistoryServer
2790 EventCatcherService
2842 AlertPublisher
2600 Bootstrap
2909 Main

this is the netstat output that I checked after jps.

root@instance-1:~# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      600/rpcbind     
tcp        0      0 0.0.0.0:9010            0.0.0.0:*               LISTEN      2524/java       
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1164/sshd       
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      1158/postgres   
tcp        0      0 127.0.0.1:19001         0.0.0.0:*               LISTEN      1496/python     
tcp        0      0 0.0.0.0:42043           0.0.0.0:*               LISTEN      2524/java       
tcp        0      0 10.240.71.132:9000      0.0.0.0:*               LISTEN      1419/python     
tcp        0      0 0.0.0.0:7432            0.0.0.0:*               LISTEN      1405/postgres   
tcp6       0      0 :::111                  :::*                    LISTEN      600/rpcbind     
tcp6       0      0 :::22                   :::*                    LISTEN      1164/sshd       
tcp6       0      0 :::7432                 :::*                    LISTEN      1405/postgres   
udp        0      0 0.0.0.0:68              0.0.0.0:*                           684/dhclient    
udp        0      0 0.0.0.0:111             0.0.0.0:*                           600/rpcbind     
udp        0      0 10.240.71.132:123       0.0.0.0:*                           3323/ntpd       
udp        0      0 127.0.0.1:123           0.0.0.0:*                           3323/ntpd       
udp        0      0 0.0.0.0:123             0.0.0.0:*                           3323/ntpd       
udp        0      0 0.0.0.0:721             0.0.0.0:*                           600/rpcbind     
udp        0      0 0.0.0.0:29611           0.0.0.0:*                           684/dhclient    
udp6       0      0 :::111                  :::*                                600/rpcbind     
udp6       0      0 :::123                  :::*                                3323/ntpd       
udp6       0      0 :::721                  :::*                                600/rpcbind     
udp6       0      0 :::22577                :::*                                684/dhclient  

As I said I don't see 8020 port. After one minute I checked JPS output and the namenode is gone.

this is the jps output one minute after rebooting.

root@instance-1:~# jps
3794 Main
3313 ResourceManager
3907 EventCatcherService
4325 Jps
2530 RunJar
3082 RunJar
2524 QuorumPeerMain
2656 Bootstrap
2877 HeadlampServer
1556 Main
2517 SecondaryNameNode
3171 JobHistoryServer
2842 AlertPublisher
2600 Bootstrap

As I said namenode is not there. I repeated the above process couple of times and everytime I get the same results port 8020 not there and namenode getting crashed. I think it is a firewall issue , what do you think?

Thanks in advance.

Best Answer

You should check the NameNode's daemon log under the $HADOOP_HOME/logs directory first.

As a motivation,

During startup, the NameNode must complete certain actions before it can serve client requests:

  1. Read file system metadata from the fsimage file.
  2. Read edit logs and apply logged operations to the file system metadata.
  3. Write a new checkpoint.
  4. Remain in safe mode until a sufficient number of blocks have been reported by datanodes.