Linux/Vyatta failover with GRE and OSPF/BGP

failovergrelinuxvyatta

I have strange problem with routing failover in scenario:
I'm trying do this failover via ospf or bgp, in both happens the same strange behavior with tunnels:
For 192.7.0.0 TUN providing default route to R1 – main site (we need all traffic).

        172.16.0.1(10.3.3.1)            10.3.3.0/30                      172.16.0.2(10.3.3.2)
        +-------------------------------->TUN0<--------------------------------------------+
        |                                                                                  |
        |                 +--------+                                                       |
        |                 |        |EXTIP1                                                 |
        |    +------------+  VPN1  +--------->IPSEC<------------+                          |
        |    |            |        |                            |                          |
        |    v            +--------+                            |                          |
       ++------+                                                |                          |
       |       |                                                |--------+        +--------++
LAN +--->|  R1   |                                                +        |            |         |
192.0.0.0/24       |                                          EXTIP3|  VPN7  +------->|  R7     <--+LAN
       ++------+                                                +        |        |         |192.7.0.0/24
        |    ^                                                  |--------+        +--------++
        |    |                                                  |                          |
        |    |                                                  |                          |
        |    |                                                  |                          |
        |    |            +--------+                            |                          |
        |    |            |        |                            |                          |
        |    +------------+ VPN2   |EXTIP2                      |                          |
        |                 |        +------------->IPSEC<--------+                          |
        |                 +--------+                                                       |
        |                                                                                  |
        |                                                                                  |
        |                                     PREFFERED PATCH                              |
        +------------------------------------>TUN1<----------------------------------------+
        172.16.0.3(10.3.3.5)                10.3.3.4/30                  172.16.0.4(10.3.3.6)

For initial start everything is working ok, when main link TUN1 comes down failover happens and after few seconds (ospf) or minutes (bgp) link are convergent and network flow is ok via TUN0.
BUT when TUN1 will came back flow is messed up, routers will change the path according to the configuration (TUN1 is always proffered path) but network flow is not going.
I can ping 192.7.0.0 <-> 192.0.0.0 with small packets but for example VNC, HTTP or oher apps don't work anymore.
I've discovered when TUN1 is back and i'll reset link of tunnels by hand via command:

sudo ip link set tun0 down
sudo ip link set tun1 down
sudo ip link set tun1 up
few seconds pause
sudo ip link set tun0 up

everything is back to normal
So my question is:

Is this wrong thinking about failover implementation?

Is this a bug?

Is this a feature?

Thank you for answering

Vyatta 6.4 8.04 on R1 R7
Vyatta 6.4 5.31 on VPN[1|2|7]

Update:
I've ran mroute on TUN0 from 192.0.0.0

mturoute.exe 192.7.0.162
* ICMP Fragmentation is not permitted. *
* Speed optimization is enabled. *
* Maximum payload is 10000 bytes. *
- ICMP payload of 1472 bytes is too big.
+ ICMP payload of 92 bytes succeeded.
+ ICMP payload of 782 bytes succeeded.
+ ICMP payload of 1127 bytes succeeded.
+ ICMP payload of 1299 bytes succeeded.
- ICMP payload of 1385 bytes is too big.
+ ICMP payload of 1342 bytes succeeded.
- ICMP payload of 1363 bytes is too big.
+ ICMP payload of 1352 bytes succeeded.
+ ICMP payload of 1357 bytes succeeded.
- ICMP payload of 1360 bytes is too big.
+ ICMP payload of 1358 bytes succeeded.
- ICMP payload of 1359 bytes is too big.
Path MTU: 1386 bytes.

After failover TUN0->TUN1 also from 192.0.0.0

mturoute.exe 192.7.0.162
* ICMP Fragmentation is not permitted. *
* Speed optimization is enabled. *
* Maximum payload is 10000 bytes. *
- ICMP payload of 1472 bytes is too big.
+ ICMP payload of 92 bytes succeeded.
...- ICMP payload of 782 bytes failed. (IP_REQ_TIMED_OUT)
+ ICMP payload of 437 bytes succeeded.
.- ICMP payload of 609 bytes failed. (IP_REQ_TIMED_OUT)
.- ICMP payload of 523 bytes failed. (IP_REQ_TIMED_OUT)
+ ICMP payload of 480 bytes succeeded.
.- ICMP payload of 501 bytes failed. (IP_REQ_TIMED_OUT)
+ ICMP payload of 490 bytes succeeded.
+ ICMP payload of 495 bytes succeeded.
.- ICMP payload of 498 bytes failed. (IP_REQ_TIMED_OUT)
+ ICMP payload of 496 bytes succeeded.
.- ICMP payload of 497 bytes failed. (IP_REQ_TIMED_OUT)
Path MTU: 524 bytes.

i'm not understood why.

update#2

EXTIP1,EXTIP2,EXTIP3 are 40Mbit F/O
EXTIP2 and EXTIP3 are from same ISP

Best Answer

It seemed this was related to pmtud, after disabling pmtud on R7 (net.ipv4.ip_no_pmtu_disc=1) it seems to work ok. This is the post abut tunnels collapse http://utcc.utoronto.ca/~cks/space/blog/linux/IPSecPacketDropProblemII

useful ip command:

ip route show table cache
Related Topic