Cisco – DMVPN with spokes behind NAT

I am deploying DMVPN. The main complexity is that some spokes are behind NAT and those can't transfer traffic directly to each other, so required to transfer it through hub. However, it dosn't work as expected. The traffic is still being sent directly to other spoke.

The topology is one cloud, dual hub. In the test lab (GNS3) I have four spokes: two of them have "direct" access to spoke (no address translation) and two others are behind other router wihch does NAT, each spoke has its own (so no two spokes behing a single nat).

Dynamic routing is done with EIGRP, because Cisco explicitly recommends it in later DMVPN-related documents, and says OSPF doesn't get excessive testing. However, I am not sure and could use OSPF if it can perform better.

(some hosts are shown as down in the picture, this picture is made after I made all described tests and analysis)

Relevant configs are:
hub1:

!
interface Tunnel0
 bandwidth 1000000
 ip address 10.100.255.253 255.255.255.0
 no ip redirects
 ip mtu 1400
 no ip next-hop-self eigrp 65000
 no ip split-horizon eigrp 65000
 ip nhrp map multicast dynamic
 ip nhrp network-id 65000
 ip nhrp holdtime 300
 ip nhrp redirect
 tunnel source Ethernet0/0
 tunnel mode gre multipoint
!
interface Ethernet0/0
 description internet.e0/0
 ip address 172.31.0.2 255.255.255.0
!
interface Ethernet0/3
 description service.e0/0
 ip address 10.100.200.253 255.255.255.240
 standby 1 ip 10.100.200.252
 standby 1 priority 110
 standby 1 preempt delay reload 60
 standby 1 name HSRP
!
router eigrp 65000
 network 10.100.200.176 0.0.0.15
 network 10.100.255.0 0.0.0.255
!
ip route 0.0.0.0 0.0.0.0 172.31.0.1
!

hub2:

!
interface Tunnel0
 bandwidth 800000
 ip address 10.100.255.254 255.255.255.0
 no ip redirects
 ip mtu 1400
 no ip next-hop-self eigrp 65000
 no ip split-horizon eigrp 65000
 ip nhrp map multicast dynamic
 ip nhrp map multicast 172.31.0.2
 ip nhrp map 10.100.255.253 172.31.0.2
 ip nhrp network-id 65000
 ip nhrp holdtime 300
 ip nhrp nhs 10.100.255.253
 ip nhrp shortcut
 ip nhrp redirect
 tunnel source Ethernet0/0
 tunnel mode gre multipoint
!
interface Ethernet0/0
 description internet.e0/1
 ip address 172.31.0.3 255.255.255.0
!
interface Ethernet0/3
 description service.e0/1
 ip address 10.100.200.254 255.255.255.240
 standby 1 ip 10.100.200.252
 standby 1 preempt delay reload 60
 standby 1 name HSRP
!
router eigrp 65000
 network 10.100.200.240 0.0.0.15
 network 10.100.255.0 0.0.0.255
!
ip route 0.0.0.0 0.0.0.0 172.31.0.1
!

spoke1 (direct internet):

!
interface Loopback0
 ip address 10.100.200.5 255.255.255.252
!
interface Tunnel0
 bandwidth 100000
 ip address 10.100.255.1 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp map 10.100.255.253 172.31.0.2
 ip nhrp map 10.100.255.254 172.31.0.3
 ip nhrp map multicast 172.31.0.2
 ip nhrp map multicast 172.31.0.3
 ip nhrp network-id 65000
 ip nhrp holdtime 300
 ip nhrp nhs 10.100.255.253
 ip nhrp nhs 10.100.255.254
 ip nhrp shortcut
 tunnel source Ethernet0/0
 tunnel mode gre multipoint
!
interface Ethernet0/0
 description internet.e1/0
 ip address 172.31.1.2 255.255.255.0
!
router eigrp 65000
 network 10.100.200.4 0.0.0.3
 network 10.100.255.0 0.0.0.255
!
ip route 0.0.0.0 0.0.0.0 172.31.1.1
!

spoke2 is similar to 1, only addresses differ. lo0 has 10.100.200.9/30

spoke3 (behind NAT):

!
interface Loopback0
 ip address 10.100.200.13 255.255.255.252
!
interface Tunnel0
 bandwidth 100000
 ip address 10.100.255.3 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp map 10.100.255.253 172.31.0.2
 ip nhrp map 10.100.255.254 172.31.0.3
 ip nhrp map multicast 172.31.0.3
 ip nhrp map multicast 172.31.0.2
 ip nhrp network-id 65000
 ip nhrp holdtime 300
 ip nhrp nhs 10.100.255.253
 ip nhrp nhs 10.100.255.254
 ip nhrp shortcut
 tunnel source Ethernet0/0
 tunnel mode gre multipoint
!
interface Ethernet0/0
 description spoke3-gw.e0/3
 ip address 192.168.1.2 255.255.255.0
!
router eigrp 65000
 network 10.100.200.12 0.0.0.3
 network 10.100.255.0 0.0.0.255
!
ip route 0.0.0.0 0.0.0.0 192.168.1.1
!

spoke3-gw (the one which does NAT for spoke3):

!
interface Ethernet0/0
 description internet.e1/2
 ip address 172.31.3.2 255.255.255.0
 ip nat outside
 ip virtual-reassembly in
!
interface Ethernet0/3
 description spoke3.e0/0
 ip address 192.168.1.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
!
ip nat inside source list 1 interface Ethernet0/0 overload
ip route 0.0.0.0 0.0.0.0 172.31.3.1
!
access-list 1 permit 192.168.1.0 0.0.0.255
!

spoke4 and spoke4-gw pair is similar, lo0 has 10.100.200.17/30. "grey" addresses are same 192.168.1.0/24. Cisco explicitely states this is valid case and should work.

internet has all those 172.31.{0,1,2,3,4}.1 addresses and just plain forwards packets between .2-s and .0.3;
service is just a system which has 10.100.200.241 address and defaults to 10.100.200.252 (HSRP-managed).

There is no IPSEC configured. I am trying to do things gradually, it is overhelming to try to do everyting right from the first shot.

I've done tests with commands like "ping 10.100.200.13 source 10.100.200.17" (from spoke3 to spoke4) and so on.

This "seems" to work. I.e. the said ping works. But capturing packets on link "internet.e1/3 – spoke4-gw.e0/0" showed that we still have direct packets from spoke3-gw to spoke4-gw and vice versa. "show ip nhrp" on spoke3 and spoke4 shows exactly this (resolved to other's -gw "white" address). Inspecting NAT "show ip nat translations" on -gw's showed that yes, there is translation of GRE from spoke3 to spoke4-gw on spoke3-gw (and similar on spoke4-gw). This is the way why said ping works. I am just lucky in this lab to have Cisco doing NAT, which is doing it right.

In the wild world NAT boxes will be anything. I can't require use of some concrete equipment (it is illegal to do so). I only can require it to do NAT of GRE to hubs and back correctly (which is minimum to run DMVPN).

If I block direct traffic between spoke3-gw and spoke4-gw (on internet system, by installing access-rules denying from 172.31.4.2 to 172.31.3.2 and other way around), I expect the system will see that and forward everyting between spoke3 and spoke4 via hub. However, it doesn't work.

sh ip nhrp on spoke3 this time shows it is unable to resolve 10.100.200.17:

10.100.200.12/30 via 10.100.255.3
   Tunnel0 created 00:00:39, expire 00:04:33
   Type: dynamic, Flags: router unique local 
   NBMA address: 192.168.1.2 
    (no-socket) 
10.100.200.17/32
   Tunnel0 created 00:00:39, expire 00:02:25
   Type: incomplete, Flags: negative 
   Cache hits: 2
10.100.255.3/32 via 10.100.255.3
   Tunnel0 created 00:01:34, expire 00:04:20
   Type: dynamic, Flags: router unique local 
   NBMA address: 192.168.1.2 
    (no-socket) 
10.100.255.4/32 via 10.100.255.4
   Tunnel0 created 00:00:39, expire 00:04:33
   Type: dynamic, Flags: router implicit used nhop 
   NBMA address: 172.31.4.2 
    (Claimed NBMA address: 192.168.1.2) 
10.100.255.253/32 via 10.100.255.253
   Tunnel0 created 03:24:10, never expire 
   Type: static, Flags: used 
   NBMA address: 172.31.0.2 
10.100.255.254/32 via 10.100.255.254
   Tunnel0 created 03:24:10, never expire 
   Type: static, Flags: used 
   NBMA address: 172.31.0.3 

Why it doesn't direct packets via hub in this case? This way is still open.