We have a number of XEN servers all running CentOS or RHEL. The VM's that they run are all required to be on their own VLAN for no other reason than the customer expects them to be. Long story short however, I can't change this right now.
We are also required to have bonding enabled on the interfaces. So to accommodat this we enslave eth1 and eth2 to bond0. We then create a seperate interface called bond0.VLANID where VLANID corresponds to the correct vlan; eg ifcfg-bond0.204
DEVICE=bond0.204
BOOTPROTO=static
ONBOOT=yes
VLAN=yes
BRIDGE=xenvlan204
Bridge to XEN
As you will see, we eventually have to bridge this out to XEN, and we do this by adding another interface called xenvlan204 (in this instance) which contains; ifcfg-xenvlan204
DEVICE=xenvlan204
BOOTPROTO=none
ONBOOT=yes
TYPE=bridge
XEN Vm Config
Finally in our XEN config for each VM, we add
vif = [ "bridge=xenvlan204" ]
This then allows the vm host to access that particular vlan
The Problem
We've noticed a few problems with this setup. One being that we currently create the interfaces manually. Which means if we add more vlan enabled interfaces and bridges we usually have to restart xend which is something I'm not so hot about. Also lower level staff have their heads melted by the number of interfaces and the risk of a mistake occurring is high.
Secondly, it can take sometime for a host to come up if it has a number of vlan taged interfaces.
Thirdly, its just not scaling well on the management aspects
The Question
Is there a better more flexible way to do this (in particular with Xen that ships with centos 5.3, 5.4 and 5.5 as we have to support all three) that leverages either scripting or other solutions to allow an arbitrary amount of interfaces to be created when a vm is instanced.
Your advise and expertise is more that welcomed.
Best Answer
Not an expert on Xen, but what you're doing with the bridged interfaces, plus the vlan tag, is quote OK, as long as you stay away from bonding mode 0 and 6 (those are known to have issues with bridges on top of them).
One more thing, since you're using the RHEL based Xen, you must be also using libvirt. As far as I have seen it using KVM, there was never any need to reload anything to see a new bridge available
Lastly, as I mentioned, I'm no Xen expert, so I might be shooting at nothing here, but can't xend be reloaded instead of restarted?