I am attempting to make possible for any machine connected to my VPN to be able to access services within containers on my server.
For this I have set a server with PROXMOX and subscribed to a Free ZeroTier VPN account.
Currently, I have a set of LXC containers running on my PROXMOX machine and a VM which is connected to the VPN. The choice of a VM and not a container is motivated by the fact, that a VM has access to the TUN device, meanwhile an unprivileged container does not.
Here's the general idea:
(Some Machine running ZeroTier) -> (ZeroTier VPN) -> (Linux VM running ZeroTier) -> (Container)
Of course, (Some Machine running Zero Tier) and (Linux VM running ZeroTier) are both on the same ZeroTier network.
What I want is to be able to exchange both UDP and TCP traffic.
What I have accomplished so far, is to run an instance of netcat listening on UDP on the (Linux VM running ZeroTier) and have (Some Machine running Zero Tier) connect through ZeroTier vpn and exchange messages, or connect through the VPN to the SSH on the Linux machine.
What I was unable to accomplish is to have (Some Machine running Zero Tier) connect to a service on (Container) using an UFW configuration.
These are the interfaces on (Linux VM running ZeroTier):
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 96:a8:af:d9:7f:99 brd ff:ff:ff:ff:ff:ff
altname enp0s18
inet 192.168.1.104/24 brd 192.168.1.255 scope global ens18
valid_lft forever preferred_lft forever
inet6 fe80::94a8:afff:fed9:7f99/64 scope link
valid_lft forever preferred_lft forever
3: ztyouvjg5f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2800 qdisc fq_codel state UNKNOWN group default qlen 1000
link/ether 96:47:56:a9:29:3a brd ff:ff:ff:ff:ff:ff
inet 10.xx.xx.163/24 brd 10.xx.xx.255 scope global ztyouvjg5f
valid_lft forever preferred_lft forever
inet6 fexx::xxxx:xxxx:xxxx:xxd7/64 scope link
valid_lft forever preferred_lft forever
Where ens18 is the interface my VM communicates with my LAN and ztyouvjg5f is the interface through which the VM is connected to the ZeroTier network.
Meanwhile UFW is configured to allow the following connections to go through:
Status: active
To Action From
-- ------ ----
22/tcp ALLOW Anywhere
9993/udp ALLOW Anywhere
27960/udp ALLOW Anywhere
22/tcp (v6) ALLOW Anywhere (v6)
9993/udp (v6) ALLOW Anywhere (v6)
27960/udp (v6) ALLOW Anywhere (v6)
192.168.1.103 27960/udp ALLOW FWD Anywhere on ztyouvjg5f
Now, for a bit of clarity;
I use 22/tcp to connect to the VM through ssh.
The port 9993/udp is ZeroTier VPN port.
The port 27960/udp is the port of my service.
The address 192.168.1.103 is the IP address of the container on my LAN which has a service on it that listens on port 27960/udp.
The intention here was to allow whatever was coming from on port 27960/udp from the VPN to 192.168.1.103:27960.
To enable port forwarding on my system I have modified the following file on my VM:
/etc/default/ufw
Setting ACCEPT on the FORWARD POLICY:
DEFAULT_FORWARD_POLICY="ACCEPT"
And also
/etc/ufw/sysctl.conf
Uncommenting the following lines:
net/ipv4/ip_forward=1
net/ipv6/conf/default/forwarding=1
net/ipv6/conf/all/forwarding=1
Finally, inside:
/etc/ufw/before.rules
I have added the following rules (I have only manually inserted the *nat block):
#
# rules.before
#
# Rules that should be run before the ufw command line added rules. Custom
# rules should be added to one of these chains:
# ufw-before-input
# ufw-before-output
# ufw-before-forward
#
# PORT FORWARDING
*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -i ens18 -p udp --dport 27960 -j DNAT --to-destination 192.168.1.103
-A POSTROUTING -s 192.168.1.0/24 -o ens18 -j MASQUERADE
# don't delete the 'COMMIT' line or these nat table rules won't be processed
COMMIT
# Don't delete these required lines, otherwise there will be errors
*filter
:ufw-before-input - [0:0]
:ufw-before-output - [0:0]
:ufw-before-forward - [0:0]
:ufw-not-local - [0:0]
# End required lines
# allow all on loopback
-A ufw-before-input -i lo -j ACCEPT
-A ufw-before-output -o lo -j ACCEPT
# quickly process packets for which we already have a connection
-A ufw-before-input -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-output -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-forward -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
# drop INVALID packets (logs these in loglevel medium and higher)
-A ufw-before-input -m conntrack --ctstate INVALID -j ufw-logging-deny
-A ufw-before-input -m conntrack --ctstate INVALID -j DROP
# ok icmp codes for INPUT
-A ufw-before-input -p icmp --icmp-type destination-unreachable -j ACCEPT
-A ufw-before-input -p icmp --icmp-type time-exceeded -j ACCEPT
-A ufw-before-input -p icmp --icmp-type parameter-problem -j ACCEPT
-A ufw-before-input -p icmp --icmp-type echo-request -j ACCEPT
# ok icmp code for FORWARD
-A ufw-before-forward -p icmp --icmp-type destination-unreachable -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type time-exceeded -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type parameter-problem -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type echo-request -j ACCEPT
# allow dhcp client to work
-A ufw-before-input -p udp --sport 67 --dport 68 -j ACCEPT
#
# ufw-not-local
#
-A ufw-before-input -j ufw-not-local
# if LOCAL, RETURN
-A ufw-not-local -m addrtype --dst-type LOCAL -j RETURN
# if MULTICAST, RETURN
-A ufw-not-local -m addrtype --dst-type MULTICAST -j RETURN
# if BROADCAST, RETURN
-A ufw-not-local -m addrtype --dst-type BROADCAST -j RETURN
# all other non-local packets are dropped
-A ufw-not-local -m limit --limit 3/min --limit-burst 10 -j ufw-logging-deny
-A ufw-not-local -j DROP
# allow MULTICAST mDNS for service discovery (be sure the MULTICAST line above
# is uncommented)
-A ufw-before-input -p udp -d 224.0.0.251 --dport 5353 -j ACCEPT
# allow MULTICAST UPnP for service discovery (be sure the MULTICAST line above
# is uncommented)
-A ufw-before-input -p udp -d 239.255.255.250 --dport 1900 -j ACCEPT
# don't delete the 'COMMIT' line or these rules won't be processed
COMMIT
With this configuration the UDP traffic on port 27960 should be forwarded to the (Container) which listens on port 27960. The container doesn't have ufw installed, but the PROXMOX firewall is configured to let udp traffic on 27960 pass through.
I can connect to the container from within my LAN.
But the traffic however doesn't arrive to the container if i try to connect to it through the VPN.
Is there something I am missing?
Thank you.
Best Answer
So, after two days of banging my head on the wall I managed to solve my issue.
Somehow, ip forwarding wasn't enabled on my Linux VM, even if I remember editing the
/etc/sysctl.conf. After enabling it I was able to forward traffic on my LAN but still wasn't able to send traffic across VPN.The big trick i pulled out from ZeroTier documentation was to configure a managed route and setup iptable rules.
First of all, I added managed routes (it's not my picture, i took it from a github post):
click here to view the image
Afterwards I edited
/etc/sysctl.confand uncommentednet.ipv4.ip_forwardThen i added these iptable rules:
Finally, I persisted the rules:
Here's a link to the docs:
https://zerotier.atlassian.net/wiki/spaces/SD/pages/224395274/Route+between+ZeroTier+and+Physical+Networks
I brought the information here in case it ever gets taken down.