I'm fairly new to Fortinet devices, so please let me know if I'm making a rookie mistake.
I have a FortiGate 90D (v5.2.5,build701) which has an IPSec site-to-site VPN connection to another firewall and I can access nodes across the VPN. I have a static Route to forward traffic for the subnet on the other side of the VPN through the VPN. I can do a traceroute and see that the traffic goes to the FortiGate and then over the VPN.
If the VPN goes down, the FortiGate starts routing traffic through it's public IP and out to the Internet where it gets stopped (as expected).
However, once the VPN comes back up, any host that had recently tried to send traffic across the VPN will have their traffic continue to go out to the Internet while other hosts (on the same subnet) will have their traffic routed over the VPN. It's as if the FortiGate remembers that some hosts were previously routed over the Internet, while any new traffic is correctly routed over the VPN. This persists until I reboot the host that is having its traffic routed over the Internet.
Can someone tell me what's going on?
Best Answer
First off, this behavior is as expected.
The Fortigate (as a stateful firewall) will create a session from the information of the first packet arriving. It will determine the route to apply and whether forwarding is permitted or not. After these decisions, subsequent traffic belonging to the same session is forwarded without any further decisions to make. (As a point aside, in Fortinet firewalls established sessions are indeed offloaded from the CPU to a network processing ASIC at that point - which is incapable of looking up routes or policies. The principle holds true even if traffic is not offloaded, though.)
Now, when the tunnel goes down, the route across is deleted from the Routing Table and VPN sessions are deleted. The next best route to a private network usually is the default route, so for arriving traffic destined to the network behind the tunnel new sessions are created and routed out the WAN interface (where they are discarded at the next router, hopefully).
When the tunnel is coming up again, new sessions are correctly routed and forwarded across the tunnel because their first packet trigger the decisions I mentioned above. Existing sessions are not affected - this would imply that with every change of the routing table, the FGT would have to re-evaluate each session if there was a better route than before. This is impractible at least. Traffic will follow the "wrong" route until the session is timed out.
There is an easy fix for this kind of scenario: create static blackhole routes on the FGT for all private, RFC1918 networks you might use behind VPN tunnels, with 192.168.x.x/24, 172.16.x.y/16 and 10.x.y.z/8 being the most popular. You need to set the distance parameters for these blackhole routes to 254 to keep them inactive as long as other, intended routes exist. The trick is that for blackhole routes, the FGT will not create sessions. So when the tunnel is down and the tunnel route discarded, the blackhole route is used - packets will be discarded immediately. Each packet arriving will trigger a session setup and a routing decision and eventually will be forwarded across the tunnel right after it comes up again.
In the Fortinet support forums you will find a couple of threads explaining this. This post offers a bulk command file ready to import all RFC1918 blackhole routes: https://forum.fortinet.com/FindPost/120872