Centos – VirtualBox, VLAN, CentOS 7: guests and host can’t communicate

centosvirtualboxvlan

I'm using CentOS 7 on everything (except the Mac noted below). Host has VirtualBox 5.1.8. Network is 192.168.10.0/24. There are no firewalls anywhere.

Everything works as expected in this scenario:

Nothing is tagged, hosts and guests can communicate on any port, to/from any ip. Network interface on each guest is bridged. Life is good here.

This scenario fails:

I created VLAN interfaces on the host and each guest. We'll call this eth0.10. Each guest continues to use eth0 (because using eth0.10 effectively removed it from the network). Network interface on each guest is bridged.

Note: when I mention ping here I realize that's just ICMP but my tests have also included TCP tests. Using ping for brevity.

I can now ping guest (192.168.10.5) to guest (192.168.10.10) but I can't ping guest (.10.5) to host (.10.50). Host (.10.50) to guest (.10.5 or .10.10) doesn't work either.

When I ping guest (.10.5 or .10.10) to some other physical system, a Mac/OS X, also in VLAN10 (.10.200) I get a response. When I ping host (.10.5) to the Mac (.10.200) I get a response. The reverse of this is also true.

I've also ran Wireshark (packet sniffer) on the Mac (.10.200). I used the filter 'vlan host 192.168.10.5' and I can see the vlan id 10 in the packet! The same is true for every single host in vlan 10.

So everybody but the host can see the guests. The guests can all see each other and everybody else but not the host. Crazy right?

I've read a few things about Open Vswitch but I don't know if this is what I need. It seems that I'm overlooking something fundamental here but I've checked the work from so many angles now.

Any suggestions would be greatly appreciated!

Best Answer

I was able to replicate your exact scenario.
Here is my test env

 +---------------------------------------------------------+                                +-----------------------------------------+
 |                                                         |                                |                                         |
 |                    Mac OS X El Capitan                  |                                |          Mikrotik router board          |
 |      Host is also setup with vlan0 VLAN ID 20           |                                |                                         |
 |      192.168.10.3                                       |                                |                                         |
 |                                                         |                                |                                         |
 |             Both VMs are bridged to en0                 |en0         Trunk               |  VLAN 20 192.168.10.250                 |
 |            +-----------------------------------------------------------------------------+  VLAN 30 192.168.30.250                 |
 |            |                            |               |            VLANs 20 and 30     |                                         |
 |   +------------------+         +-------------------+    |                                |                                         |
 |   |                  |         |                   |    |                                |                                         |
 |   |    Cent OS 7     |         |    Cent OS 7      |    |                                |                                         |
 |   |    Node 1        |         |    Node 2         |    |                                |                                         |
 |   |                  |         |                   |    |                                |                                         |
 |   |  192.168.10.2    |         |    192.168.10.4   |    |                                +-----------------------------------------+
 |   +------------------+         +-------------------+    |
 |      VLAN 20                          VLAN 20           |
 +---------------------------------------------------------+

Exact same thing happens.
When both VMs are bridged to en0 :

  1. They can ping each other. 192.168.10.2 < -- > 192.168.10.4
  2. They can ping VLAN 20 Int 192.168.10.250 that leaves on Mikrotik, so they have external world connectivity.
  3. Mac host which is also setup with vlan0 VLAN ID 20 192.168.10.3 can ping Mikrotik
  4. VMs can't ping host and host can't ping VMs.

When bridging VMs to vlan0 instead of en0 - they lose connectivity to outside world (can't ping mikrotik)

So it appears that situation is indeed very similar to the way bridging is done in KVM with macvtap. With macvtap VMs can't communicate with host, so here the issue is explained https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/App_Macvtap.html

This situation is actually not an error — it is the defined behavior of macvtap. Due to the way in which the host's physical Ethernet is attached to the macvtap bridge, traffic into that bridge from the guests that is forwarded to the physical interface cannot be bounced back up to the host's IP stack. Additionally, traffic from the host's IP stack that is sent to the physical interface cannot be bounced back up to the macvtap bridge for forwarding to the guests.

It appears that the same mechanism is in effect with bridged VLANs. I don't know for sure, just speculating here.

Edit: I found this blog from rackspace which explains exactly this issue http://blog.rackspace.com/vms-vlans-and-bridges-oh-my-part-2

Related Topic