Linux NAT over VLAN

1) This seems like an extremely broken configuration so I'm not surprised you're getting strange behavior with the ARP replies - I don't think it would be unexpected behavior for replies sent from the VLAN interface to arrive back at the untagged interface - they are on the same physical network with the same MAC address after all.

More importantly..

2) This configuration is quite common and will work well. In the last few days I've brought up a couple of KVM hosts running RHEL6 with pretty much the exact config you describe - untagged VLAN to the NIC for management and tagged interfaces for all the bridged virtual interfaces.

That's close to what we have, right down to the Untangle gateway. We do it a little differently, though. It helps visualize if you start from a completely flat network with no vlans. Represent this with everything untagged on vlan 1.

Now we want to add support for wifi traffic on vlan 2. To do this, set both ends of every trunk line (lines connecting two switches) to also be tagged for vlan 2. There is no need to switch vlan 1 from untagged to tagged, as you do in your current proposal; all you need to do is add the port as tagged member of vlan 2. Additionally, ports needing to talk to wireless clients should be added as tagged members of vlan 2. This includes the port your untangle server is connected to, and the ports for any servers (like dhcp) that wifi traffic should be able to see without routing. Again, you want to leave them untagged on vlan 1; just add them as tagged members of vlan 2 also.

One important key here is that our central switch supports layer 3 routing, and we have an ACL there that tells it when it's allowed to route traffic from one vlan to another. For example, all of our printers and our printer server are on vlan 1. We use a software package on the print server to count jobs and bill students for print usage, so we do want to allow wifi traffic to hit the print server. We do NOT want to allow wifi traffic to hit individual printers directly, which would bypass that software, and so the printers are restricted in the ACL, but the print server is allowed.

You will also need to do some work on your untangle box itself, depending on how things are set up. Look under Config->Networking->Interfaces and edit your internal interface. There you want to see your untangle server's Primary IP Address and Netmask set for an address on your vlan 1 subnet. We also have an IP Address Alias setup for each vlan we use, NAT policies defined for each vlan network address and netmask, and routes for each vlan to send traffic for those vlans to the internal interface.

I should add that we run our untangle in router mode with a single internal interface and have dhcp/dns on a windows server box. Your setup may be different if you use bridge mode or want to run dhcp/dns off of untangle, or use separate interfaces for each network.

Now your network is prepared to add access points. Whenever you add an access point to the network, set it's port as untagged for vlan 2, and tagged for vlan 1. That vlan 1 tag here is optional, but I often find it helpful.

Finally, depending on the size of your installation, you may find that one vlan for wifi is not enough. You generally want to keep it down to about one /24's worth of clients online at a time. Fewer is better. Any more than that and broadcast traffic will start to eat up your airtime. You can get away with larger address spaces (say, /22's), as long as all of the addresses aren't in use at one time. That's how we handle it here. I support about 450 residential college students on a single SSID with a /21 subnet, but I'm really stretching it and probably should start carving my assignments up so that broadcast traffic from students in different buildings doesn't interfere with each other. If this is more of a single large building like a high-school, you probably want to choose different SSIDs per vlan. If it's a multi-building campus where the buildings are separated by some distance and you won't be pushing coverage to the space between buildings, you can get by with one SSID for all of the vlans.

Hopefully, your controller/wifi vendor covers all that, but if you're like us you don't have the funds for $600/access point or $3000+ per controller unit. It might be worth remembering that you can use simple consumer routers as access points by turning off dhcp and using a LAN port rather than WAN port for the uplink. You'll miss out on some reporting and automatic power and channel adjustments, but with some good access points and some careful work at setup you can put together quite a large network this way.