Nfs – mount.nfs: Failed to resolve server

domain-name-systemnfs

I have a problem with nfs client not being able to resolve a resolvable DNS name.

[root@testserver-2 ~]# host nfs-server-host-name
nfs-server-host-name has address 10.37.4.131

[root@testserver-2 ~]# nslookup nfs-server-host-name
Server:         127.0.0.1
Address:        127.0.0.1#53

Name:   nfs-server-host-name
Address: 10.37.4.126

[root@testserver-2 ~]# showmount -e nfs-server-host-name
clnt_create: RPC: Unknown host

[root@testserver-2 ~]# ss -lnp |grep rpc
LISTEN     0      128                       *:111                      *:*      users:(("rpcbind",7627,8)

[root@testserver-2 ~]# mount -t nfs -o defaults,auto,proto=tcp nfs-server-host-name:/ifs/exports/EXPORT /mnt/export
mount.nfs: Failed to resolve server nfs-server-host-name: Name or service not known

Local resolving is set up via dnsmasq:

[root@testserver-2 ~]# cat /etc/resolv.conf
options rotate timeout:2 attempts:4
nameserver 127.0.0.1
nameserver 10.1.1.1
nameserver 8.8.8.8

[root@testserver-2 ~]# cat /etc/dnsmasq.conf 
resolv-file=/etc/resolv.conf
server=/nfs-server-host-name/10.37.4.1 #IP address of Isilon smart connect resolver
listen-address=127.0.0.1

nsswitch configuration (default Centos 6):

[root@testserver-2 ~]# grep hosts /etc/nsswitch.conf
hosts:      files dns

Please note that nfs-server-host-name resolving is handled by an Isilon smart connect resolver, which is configured as a resolver for nfs-server-host-name locally via dnsmasq and works (as seen in the above example). Replacing nfs-server-host-name with an IP address is not an option as there are multiple NFS nodes which are balanced by the resolver therefore the IP is variable and cannot be hardcoded. It is therefore OK that the two results above from hosts and nslookup are different. This is intended and expected behaviour.

Please also note that nfs-server-host-name is a very precise example of what the hostname looks like. The actual hostname looks almost identically, it's not a FQDN. In fact there are no dots in the hostname at all. This is a feature I'm not in control of.

When mounted manually with one valid node IP address the export mounts fine. When one node's address is put to /etc/hosts, it mounts. When resolved via DNS means, it does not work for the nfs client but works for other net tools such as hosts, dig or ping.

This seems to be not-so-uncommon problem but all tips I found so far say "replace hostname with IP" which is something I cannot do.

What I missed?

Best Answer

I have yet to figure out what the real problem is but the solution is to remove rotate option from /etc/resolv.conf and put a dot after nfs-server-host-name in /etc/fstab to prevent domain search (which occurred even when there was no search option in /etc/resolv.conf), i.e.:

nfs-server-host-name.:/nfs/export/...
                    ^
                    ^---up here

It may have something to do with kernel build option CONFIG_NFS_USE_LEGACY_DNS, which is set to yes on my kernel.

The behaviour of nfs-client (and nfs-client only) was that it only queried the second nameserver in /etc/resolv.conf no matter how many there were as long as the rotate option was present. It did, however, work when there was no second nameserver at all. Beats me...