Linux Networking – Force Different Users to Use Different Network Interfaces

I have a linux machine set up with a one physical NIC, connected with a managed switch. The connection is a VLAN trunk. On the machine there are two vlan interfaces for which there are different IP addresses (part of different VLANs).

What I want to achieve is that, depending on the interactive user logging in on this machine, the user is forced to use the network interface for a certain VLAN.

Current network setup

NIC = enp1s0
VLAN3 itf = enp1s0.3 - bridged in br0.3 with IP = 192.168.3.2
VLAN4 itf = enp1s0.4 - bridged in br0.4 with IP = 192.168.4.2

user3 (uid=1003) when logging in should have all network traffic he creates go via br0.3

user4 (uid=1004) when logging in should have all network traffic he created go via br0.4

OPTION 1 : nft only.

My 1st attempt, was using nftables to force this.

# hook output, type route (allows to mangle with packets)
nft add table net4
nft add chain net4 mangle '{ 
    type route hook output priority -300; policy accept;
}'

nft add rule ip net4 mangle meta skuid 1004 ip saddr set 192.168.4.2 return
nft add rule ip net4 mangle meta skuid 1004 oif br0.3 drop

This doesn't work as expected. The packet's source address is changed but it goes out over br0.3. The return packet is coming back via br0.4. The outgoing packet remains bound to br0.3 even though the saddr was changed.
I saw this behaviour by having two terminals open with following commands:

tcpdump -i br0.3 -nn host 192.168.3.2 or host 192.168.4.2
tcpdump -i br0.4 -nn host 192.168.3.2 or host 192.168.4.2

According to: https://wiki.nftables.org/wiki-nftables/index.php/Netfilter_hooks the output hook is after the routing decision, so it seems this cannot be changed anymore.

I tried a rule like this:

nft add rule ip net4 mangle meta skuid 1004 ip saddr set 192.168.4.2 meta oifname set br0.4 return

But setting oif or oifname is not allowed.

According to https://www.nftables.org/documentation/HOWTO/netfilter-extensions-HOWTO-4.html#ss4.5 there is a ROUTE patch that may provide the ability to change the oif. But the example is for iptables and I can neither use this patch for nftables or iptables it seems.

I tried an alternative nft rule set as well, using marking combined with ip rule and ip route:

nft add rule ip net4 mangle meta skuid 1004 ip saddr set 192.168.4.2 meta mark set 4 return
nft add rule ip net4 mangle meta skuid 1004 oif br0.3 drop

ip rule add priority 10000 fwmark 4 table 4
ip route add table 4 default via 192.168.4.1 dev br0.4

This doesn't work either, as expected, as it seems indeed the output hook is after the routing decision.

OPTION 2: network namespace with a little nft

I tried an alternative: network namespaces.

# delete the network interface and bridge
ip link set dev br0.4 down
ip link set enp1s0.4 down
ip link del br0.4
ip link del enp1s0.4

# create netns and create new vlan interface in new netns
NAMESPACE=netns4
ip netns add $NAMESPACE
ip -n $NAMESPACE link set dev lo up
ip link add link enp1s0 name enp1s0.4 netns $NAMESPACE type vlan id 4
ip -n $NAMESPACE link add name br0.4 type bridge
ip -n $NAMESPACE link set dev enp1s0.4 master br0.4
ip -n $NAMESPACE link set enp1s0.4 up
ip -n $NAMESPACE link set br0.4 up
ip -n $NAMESPACE addr add 192.168.4.2/24 dev br0.4
ip -n $NAMESPACE route add default via 192.168.4.1 dev br0.4

When I now switch back and forth between the namespaces as root all works as expected. If I combine this with a nft rule not allowing user4 to use br0.3 I have more or less what I need.

nft add rule ip net4 mangle meta skuid 1004 oif br0.3 drop

But it is not straight forward to have a user session set up with this namespace. systemd-logind does not allow a configuration whereby the resulting user session will be created with non-default namespaces. Similarly a graphical login cannot configure namespaces per user or user group. My system has sddm which allows for Namespaces= in the [General] section but it means all users will join all the specified namespaces.

A terminal login, requires a hack like this using sudoers (because the profile stuff is run as the user) and a file /etc/profile.local and only works for bash.

uid=`/usr/bin/id -u`
uname=`/usr/bin/id -un`
if [ $uid = 1004 ]; then
        namespace=netns4
        if [ -r /run/netns/$namespace ]; then
                namespace_inode=`ls -i /run/netns/$namespace | cut -d ' ' -f 1`
                proc_inode=`ls -l /proc/$$/ns/net | sed 's/.*\[\([0-9][0-9]*\)\]/\1/'`
                if [ $proc_inode != $namespace_inode ]; then
                        uname=`/usr/bin/id -un`
                        sudo /usr/bin/ip netns exec $namespace /usr/bin/su - $uname "${@}"
                fi
        fi
fi

and sudoers must have a line:

user4 ALL = (root) NOPASSWD: /usr/bin/ip netns exec netns4 /usr/bin/su - user4
user4 ALL = (root) NOPASSWD: /usr/bin/ip netns exec netns4 /usr/bin/su - user4 *

I don't seem able to get the same done for Xsession /usr/etc/X11/xdm/Xsession

Anyone an idea how to achieve my original requirement?