Access libvirt+KVM virtual machines with DNS

dnsmasqdomain-name-systeminternal-dnskvm-virtualizationlibvirt

I have an Ubuntu Trusty machine running KVM + Libvirt to manage small virtual machines, and using the standard NetworkManager to connect to regular networks.

I want to be able to access the virtual machines via DNS from the host.

Libvirt uses a virtual private subnet (192.168.122.0/24), NAT'd to access the rest of the world via a bridge (virbr0) on my eth0. Dnamasq priveds DHCP+DNS to this virtual network.

This is the libvirt config for the virtual network:

<network>
  <name>default</name>
  <uuid>400c59ff-c276-4154-ab73-9a8a8d1c6be3</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:f4:bd:37'/>
  <domain name='kvm'/>
  <dns forwardPlainNames='no'>
    <forwarder addr='127.0.1.1'/>
    <host ip='192.168.122.1'>
      <hostname>host</hostname>
      <hostname>host.kvm</hostname>
    </host>
  </dns>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

Libvirt starts a dnsmasq instance listening on 192.168.122.1:53 which answers all requests for .knv and forwards any other requests to my host. This dnsmasq config is automatically generated by libvirt:

/var/lib/libvirt/dnsmasq/default.conf

##WARNING:  THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
##OVERWRITTEN AND LOST.  Changes to this configuration should be made using:
##    virsh net-edit default
## or other application using the libvirt API.
##
## dnsmasq conf file created by libvirt
strict-order
user=libvirt-dnsmasq
no-resolv
server=127.0.1.1
domain=kvm
expand-hosts
domain-needed
local=//
pid-file=/var/run/libvirt/network/default.pid
except-interface=lo
bind-dynamic
interface=virbr0
dhcp-range=192.168.122.2,192.168.122.254
dhcp-no-override
dhcp-leasefile=/var/lib/libvirt/dnsmasq/default.leases
dhcp-lease-max=253
dhcp-hostsfile=/var/lib/libvirt/dnsmasq/default.hostsfile
addn-hosts=/var/lib/libvirt/dnsmasq/default.addnhosts

NetworkManager has an instance of dnsmasq listening on 127.0.1.1:53 which it uses for all DNS quieries before passing to whichever DNS servers my host gets assigned by the external DHCP system.

In order for my host Ubuntu system to use the dnsmasq of libvirt, I point the dnsmasq of NetworkManager to use 192.168.122.1 for the domain kvm:

/etc/NetworkManager/dnsmasq.d/libvirt.conf

server=/kvm/192.168.122.1

And this works, for the most part…

me@host ~ $ ps aufx
...cut...
root     11010  0.2  0.0 342084  6348 ?        Ssl  10:59   0:00 NetworkManager
root     11018  0.0  0.0  10232  3732 ?        S    10:59   0:00  \_ /sbin/dhclient -d -sf /usr/lib/NetworkManager/nm-dhcp-client.action -pf /run/sendsigs.omit.d/network-manager.dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-b8043 
nobody   11228  0.0  0.0  32252  1564 ?        S    10:59   0:00  \_ /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/run/sendsigs.omit.d/network-manager.dnsmasq.pid --listen-address=127.0.1.1 --
root     11033  1.0  0.1 513356 15160 ?        Sl   10:59   0:01 /usr/sbin/libvirtd -d
libvirt+ 11085  0.0  0.0  28208   948 ?        S    10:59   0:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf

me@host ~ $ sudo netstat -nulpd | grep dnsmasq
udp  0  0  127.0.1.1:53      0.0.0.0:*  11228/dnsmasq   
udp  0  0  192.168.122.1:53  0.0.0.0:*  11085/dnsmasq   
udp  0  0  0.0.0.0:67        0.0.0.0:*  11085/dnsmasq  

me@host ~ $ host test.kvm
test.kvm has address 192.168.122.193
;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached

But creates a massive number of dnsmasq AAAA queries all waiting for a response.

me@host ~ $ sudo netstat -nulpd | grep dnsmasq
udp  0  0  0.0.0.0:39329  0.0.0.0:*  11228/dnsmasq   
udp  0  0  0.0.0.0:2469   0.0.0.0:*  11085/dnsmasq   
udp  0  0  0.0.0.0:14805  0.0.0.0:*  11228/dnsmasq
...cut...
udp  0  0  0.0.0.0:51569  0.0.0.0:*  11228/dnsmasq   
udp  0  0  0.0.0.0:31091  0.0.0.0:*  11085/dnsmasq   
udp  0  0  0.0.0.0:39305  0.0.0.0:*  11085/dnsmasq

me@host ~ $ sudo netstat -nulpd | grep dnsmasq | wc -l
131

And a tcpdump shows them to be mainly AAAA requests:

me@host ~ $ sudo tcpdump -vni any udp port 53
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
11:04:49.453864 IP (tos 0x0, ttl 64, id 56217, offset 0, flags [none], proto UDP (17), length 55)
    127.0.0.1.58535 > 127.0.1.1.53: 31275+ A? mysql.kvm. (27)
11:04:49.453948 IP (tos 0x0, ttl 64, id 20062, offset 0, flags [DF], proto UDP (17), length 55)
    192.168.122.1.7098 > 192.168.122.1.53: 41491+ A? mysql.kvm. (27)
11:04:49.454013 IP (tos 0x0, ttl 64, id 20063, offset 0, flags [DF], proto UDP (17), length 71)
    192.168.122.1.53 > 192.168.122.1.7098: 41491* 1/0/0 mysql.kvm. A 192.168.122.193 (43)
11:04:49.454068 IP (tos 0x0, ttl 64, id 37088, offset 0, flags [DF], proto UDP (17), length 71)
    127.0.1.1.53 > 127.0.0.1.58535: 31275* 1/0/0 mysql.kvm. A 192.168.122.193 (43)
11:04:49.454321 IP (tos 0x0, ttl 64, id 56218, offset 0, flags [none], proto UDP (17), length 55)
    127.0.0.1.56040 > 127.0.1.1.53: 47999+ AAAA? mysql.kvm. (27)
11:04:49.454381 IP (tos 0x0, ttl 64, id 20064, offset 0, flags [DF], proto UDP (17), length 55)
    192.168.122.1.19631 > 192.168.122.1.53: 20542+ AAAA? mysql.kvm. (27)
...cut...
11:05:09.510237 IP (tos 0x0, ttl 64, id 20515, offset 0, flags [DF], proto UDP (17), length 55)
    192.168.122.1.19631 > 192.168.122.1.53: 35761+ MX? mysql.kvm. (27)
11:05:09.510237 IP (tos 0x0, ttl 64, id 56674, offset 0, flags [DF], proto UDP (17), length 55)
    127.0.0.1.46085 > 127.0.1.1.53: 53641+ AAAA? mysql.kvm. (27)
11:05:09.510315 IP (tos 0x0, ttl 64, id 56675, offset 0, flags [DF], proto UDP (17), length 55)
    127.0.0.1.46085 > 127.0.1.1.53: 26166+ MX? mysql.kvm. (27)
11:05:09.510334 IP (tos 0x0, ttl 64, id 20516, offset 0, flags [DF], proto UDP (17), length 55)
    192.168.122.1.19631 > 192.168.122.1.53: 4247+ AAAA? mysql.kvm. (27)
11:05:09.510407 IP (tos 0x0, ttl 64, id 56676, offset 0, flags [DF], proto UDP (17), length 55)
    127.0.0.1.46085 > 127.0.1.1.53: 49331+ AAAA? mysql.kvm. (27)
11:05:09.510433 IP (tos 0x0, ttl 64, id 20517, offset 0, flags [DF], proto UDP (17), length 55)
    192.168.122.1.19631 > 192.168.122.1.53: 63294+ MX? mysql.kvm. (27)
^C
934 packets captured
1857 packets received by filter
0 packets dropped by kernel

I tried lowering the priority of AAAA records in /etc/gai.conf

precedence ::ffff:0:0/96  100

Even tried disabling IPv6 entirely /etc/sysctl.conf:

# Disable IPv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

But the AAAA requests are still being sent and name resolution becomes unbearably slow.

Is there a way for either libvirt or NetworkManager to ignore or respond negatively to these requests so I don't have to wait for all the requests to time out before using the A record that was already received?

Best Answer

If a forwarder is configured, dnsmasq will forward all DNS queries that it has no explicit data for. This includes records for configured static DHCP clients that have no active lease, AAAA records unless IPv6 addresses are defined explicitly, and more.

There are several ways to avoid this:

Don't configure a forwarder

Simply omit the fowarder entries in the network definition. Probably not desirable, unless the virtual network is really isolated. This is the only possibility that libvirt currently supports (12/2014), AFAIK.

local domain in dnsmasq.conf

Configure the domain as "local" in dnsmasq:

 domain=local.net,192.168.10.0/24
 local=/local.net/
 local=/10.168.192.in-addr.arpa/

In theory, this could be abbreviated as domain=local.net,192.168.10.0/24,local, but a dnsmasq bug fixed only recently causes this to fail.

libvirt doesn't support this. In order to use this configuration, you need to set up the bridge manually in your OS and configure the libvirt network like this:

   <network>
     <name>local</name>
     <forward mode='bridge'/>
     <bridge name='br0'/>
   </network>

You don't have to create a libvirt virtual network at all in this configuration, just use <interface 'type=bridge'> in your VM definition files.

auth zone in dnsmasq.conf

The auth-zone parameter has a similar effect to local. However it has other implications that I don't claim to fully understand. I suppose this configuration would be desirable if names in the virtual network are supposed to be resolved from the outside.

domain=local.net
auth-zone=local.net

This setup isn't supported by libvirt either, so the same procedure for setting up the bridge must be applied as above.

Related Solutions

Mapping USB drive direct to libvirt KVM virtual machine

Check out the 'USB passthrough | Using Libvirt' section of this page from the Edubuntu wiki. More info here and here.

If you don't ever plan to hot plug/unplug the usb drive while the guest is running you could also try adding a section like

<disk type='block' device='disk'>
  <driver name='qemu' type='raw'/>
  <source dev='/dev/sdb'/>
  <target dev='sdb' bus='scsi'/>
</disk>

to your VM's xml definition file. Just change the source dev to match what was assigned to your usb when it was plugged in. However, you can't guarantee the usb's dev assignment will survive reboot if other devices are added or removed, so I'd recommend the approach in the first link above instead.

In case the link disappears, here is the relevant USB passthrough section of the linked docs:

Using Libvirt

fired up a pre-existing vm
virsh start maverick2
plugged in a usb drive
found the usb address using lsusb, which gave me
{{ Bus 002 Device 006: ID 1058:1023 Western Digital Technologies, Inc. }}}

defined a xml file with the device info:

<hostdev mode='subsystem' type='usb'> <source> <vendor id='0x1058'/> <product id='0x1023'/> </source> </hostdev>
passed the usb drive to the vm
sudo virsh attach-device maverick2 /tmp/a.xml
HOWEVER this does not work with apparmor enabled. You must either disable apparmor, or add
/dev/bus/usb/*/[0-9]* rw,
to either /etc/apparmor.d/libvirt-qemu (which gives all guests full access to physical host devices) or to
/etc/apparmor.d/libvirt/libvirt-<uuid>
which will give only the one guest that access. (Thanks to jdstrand for help getting that straight.)

Linux – KVM/Libvirt bridged/routed networking not working on newer guest kernels

Ok, I have found the solution myself. Apparently, the article on the Hetzner wiki wasn't complete. Redirects were still being sent, resulting in Hetzner (where we have our server) blocking the traffic. I am not sure how it had to do with what kernel version the guests were running, if someone can explain that I would very much like to hear it.

You can find out if you suffer from the same problem by executing the following command:

for file in `find /proc/ -iname send_redirects`; do echo $file; cat $file; done

which outputs something like:

/proc/sys/net/ipv4/conf/all/send_redirects
0
/proc/sys/net/ipv4/conf/default/send_redirects
1
/proc/sys/net/ipv4/conf/lo/send_redirects
1
/proc/sys/net/ipv4/conf/eth0/send_redirects
0
/proc/sys/net/ipv4/conf/br0/send_redirects
1
/proc/sys/net/ipv4/conf/vnet1/send_redirects
1
/proc/sys/net/ipv4/conf/vnet0/send_redirects
1
/proc/sys/net/ipv4/conf/virbr0/send_redirects
1
/proc/sys/net/ipv4/conf/virbr0-nic/send_redirects
1
/proc/sys/net/ipv4/conf/vnet2/send_redirects
1
/proc/sys/net/ipv4/conf/vnet3/send_redirects
1

I have to admit I do not know which ones should or should not be 0, but I have set them all to 0, and now it works fine, so I'll leave it at that.

for file in `find /proc/ -iname send_redirects`; do echo 0 > $file; done