I'm trying to create a multisite network using pfsense linked together with Tinc VPN. This is my current topology:
Router A Router B
**************** ****************
* * * *
* 10.0.0.1/16 *--------* 10.1.0.1/16 *
* * * *
**************** ****************
| |
| |
**************** ****************
* * * *
* 10.0.0.11/16 * * 10.1.0.16/16 *
* * * *
**************** ****************
Node 1 My Desktop
My desktop and both routers can hit every machine on the network, but node 1 can only hit router A.
Router B is currently sitting behind Verizon's router with the VPN port forwarded. The link is established. Although, I doubt this would be the root of the issue here.
I've triple checked my pfsense configs and they are identical to one another. I'm really not sure what is preventing node 1 from communicating with the rest of the network. I've basically opened everything up. I have any to any rules for all interfaces yet node 1 cannot find the route.
If it is of any relevance, router A and node 1 are hosted in the cloud through Vultr. I have the private network enabled and node 1 is requesting addresses from the DHCP server on router A. Vultr does assign private IPs in the 10.X.X.X space with the same subnet. Could my IP space be conflicting with theirs? Vultr does not deploy a gateway and the IPs they assign are entirely static.
You can use any IPs you like on the private network. We assign one IP by default, but you can ignore it and use other ones if you like.
I really don't know why node 1 cannot hit the other subnet and I'm hoping that someone might be able to help me figure that out.
10.1.0.16
bkvaluemeal@Formula:~$ ping -c 3 10.1.0.1
PING 10.1.0.1 (10.1.0.1) 56(84) bytes of data.
64 bytes from 10.1.0.1: icmp_seq=1 ttl=64 time=0.330 ms
64 bytes from 10.1.0.1: icmp_seq=2 ttl=64 time=0.319 ms
64 bytes from 10.1.0.1: icmp_seq=3 ttl=64 time=0.305 ms
--- 10.1.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.305/0.318/0.330/0.010 ms
bkvaluemeal@Formula:~$ ping -c 3 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=63 time=9.82 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=63 time=8.86 ms
64 bytes from 10.0.0.1: icmp_seq=3 ttl=63 time=38.0 ms
--- 10.0.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 8.864/18.925/38.085/13.553 ms
bkvaluemeal@Formula:~$ ping -c 3 10.0.0.11
PING 10.0.0.11 (10.0.0.11) 56(84) bytes of data.
64 bytes from 10.0.0.11: icmp_seq=1 ttl=62 time=11.5 ms
64 bytes from 10.0.0.11: icmp_seq=2 ttl=62 time=10.5 ms
64 bytes from 10.0.0.11: icmp_seq=3 ttl=62 time=9.37 ms
--- 10.0.0.11 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 9.370/10.482/11.555/0.892 ms
bkvaluemeal@Formula:~$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether e0:3f:49:ad:81:03 brd ff:ff:ff:ff:ff:ff
inet 10.1.0.16/16 brd 10.1.255.255 scope global dynamic eno1
valid_lft 6915sec preferred_lft 6915sec
inet6 fe80::20dc:2028:faee:5420/64 scope link
valid_lft forever preferred_lft forever
3: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 54:27:1e:55:ae:33 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.163/24 brd 192.168.1.255 scope global dynamic wlp3s0
valid_lft 76214sec preferred_lft 76214sec
inet6 fe80::d9de:6606:5307:968b/64 scope link
valid_lft forever preferred_lft forever
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether 52:54:00:d1:33:dd brd ff:ff:ff:ff:ff:ff
6: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:82:c6:99:06 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 scope global docker0
valid_lft forever preferred_lft forever
10.1.0.1
PING 10.1.0.16 (10.1.0.16): 56 data bytes
64 bytes from 10.1.0.16: icmp_seq=0 ttl=64 time=0.177 ms
64 bytes from 10.1.0.16: icmp_seq=1 ttl=64 time=0.312 ms
64 bytes from 10.1.0.16: icmp_seq=2 ttl=64 time=0.194 ms
--- 10.1.0.16 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.177/0.228/0.312/0.060 ms
PING 10.0.0.1 (10.0.0.1): 56 data bytes
64 bytes from 10.0.0.1: icmp_seq=0 ttl=64 time=8.926 ms
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=8.335 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=8.290 ms
--- 10.0.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 8.290/8.517/8.926/0.290 ms
PING 10.0.0.11 (10.0.0.11): 56 data bytes
64 bytes from 10.0.0.11: icmp_seq=0 ttl=63 time=11.052 ms
64 bytes from 10.0.0.11: icmp_seq=1 ttl=63 time=9.573 ms
64 bytes from 10.0.0.11: icmp_seq=2 ttl=63 time=9.913 ms
--- 10.0.0.11 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 9.573/10.179/11.052/0.632 ms
10.0.0.1
PING 10.1.0.16 (10.1.0.16): 56 data bytes
64 bytes from 10.1.0.16: icmp_seq=0 ttl=63 time=8.307 ms
64 bytes from 10.1.0.16: icmp_seq=1 ttl=63 time=9.256 ms
64 bytes from 10.1.0.16: icmp_seq=2 ttl=63 time=9.109 ms
--- 10.1.0.16 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 8.307/8.891/9.256/0.417 ms
PING 10.1.0.1 (10.1.0.1): 56 data bytes
64 bytes from 10.1.0.1: icmp_seq=0 ttl=64 time=8.618 ms
64 bytes from 10.1.0.1: icmp_seq=1 ttl=64 time=8.579 ms
64 bytes from 10.1.0.1: icmp_seq=2 ttl=64 time=8.702 ms
--- 10.1.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 8.579/8.633/8.702/0.051 ms
PING 10.0.0.11 (10.0.0.11): 56 data bytes
64 bytes from 10.0.0.11: icmp_seq=0 ttl=64 time=1.142 ms
64 bytes from 10.0.0.11: icmp_seq=1 ttl=64 time=2.385 ms
64 bytes from 10.0.0.11: icmp_seq=2 ttl=64 time=2.053 ms
--- 10.0.0.11 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.142/1.860/2.385/0.525 ms
10.0.0.11
root@node1:~# ping -c 3 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1 icmp_seq=1 ttl=64 time=1.10 ms
64 bytes from 10.0.0.1 icmp_seq=2 ttl=64 time=1.04 ms
64 bytes from 10.0.0.1 icmp_seq=3 ttl=64 time=0.749 ms
--- 10.0.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 0.749/0.968/1.106/0.156 ms
root@node1:~# ping -c 3 10.1.0.1
PING 10.1.0.1 (10.1.0.1) 56(84) bytes of data.
--- 10.1.0.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2042ms
root@node1:~# ping -c 3 10.1.0.16
PING 10.1.0.16 (10.1.0.16) 56(84) bytes of data.
--- 10.1.0.16 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2048ms
root@node1:~# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether XX:XX:XX:XX:XX:XX brd XX:XX:XX:XX:XX:XX
inet 45.77.X.X/23 brd 45.77.X.X scope global ens3
valid_lft forever preferred_lft forever
inet6 2001:19f0:X:X:X:X:X:X/64 scope global mngtmpaddr dynamic
valid_lft 2591544sec preferred_lft 604344sec
inet6 fe80::5400:X:X:X/64 scope link
valid_lft forever preferred_lft forever
3: ens7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 5a:01:01:3c:13:c8 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.11/16 brd 10.0.255.255 scope global ens7
valid_lft forever preferred_lft forever
inet6 fe80::5801:1ff:fe3c:13c8/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:65:df:2f:a1 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:65ff:fedf:2fa1/64 scope link
valid_lft forever preferred_lft forever
Best Answer
"the ping go from one side, Desktop --> Node1 but not when launched from Node1 --> Desktop" - This statement, if accurate, eliminates all possibilities of a routing issue.
In order for a ping to work from Desktop->Node1, you also have to receive a reply back from Node1->Desktop. That indicates everything is setup fine on the VPN and routing.
Instead this is a firewall issue. Because the ping from Node1 is dropped at Router B, yet Router B can ping Desktop, then the firewall issue is probably on Router B. Router B is allowing outbound and "related" connections but is not allowing inbound connections.
Based on the information you posted I would have to say it has to do with what you have called "Firewall LAN rules." Change it to ANY/ANY/ANY/ANY (or whatever) for testing. The firewall rules are ambiguous so I can't say one way or another.
EDIT:
We discovered that Node 1 had two network interfaces. One on the "private network" with IP 10.0.0.11 and another with a public IP address AND default gateway. In addition, the NAT rules were causing the traffic to be NATted across the VPN tunnel. Therefore, Desktop could successfully PING Node 1 because the traffic arriving at Node 1 appeared to be coming from 10.0.0.1. But when trying to PING Desktop from Node1, Node1 did not have a route to 10.1.0.0/16.
Once we cleaned up the routing table on Node1 and set the NAT rules to automatic on Router A and B, everything started working as intended.