Cisco – AWS direct connect BGP issue

awsbgpciscorouting

enter image description here

This is out current setup and everything working great!!! here, now i want backup link for AWS so i have request for one more secondary link and terminated on Cisco ASA on Port-channel1.8 interface. 

asa/pri/act# sh run int po1.8
!
interface Port-channel1.8
 description ### AWS-DX-2 ###
 vlan 8
 nameif aws_dx_2
 security-level 0
 ip address 169.254.8.1 255.255.255.248 standby 169.254.8.3

asa/pri/act# sh run int po1.9
!
interface Port-channel1.9
 description ### AWS-DX-1 ###
 vlan 9
 nameif aws_dx
 security-level 0
 ip address 169.254.9.1 255.255.255.248 standby 169.254.9.3

This is my BGP config

router bgp 65501
 bgp log-neighbor-changes
 timers bgp 10 30 0
 address-family ipv4 unicast
  neighbor 169.254.8.2 remote-as 7224
  neighbor 169.254.8.2 password *****
  neighbor 169.254.8.2 activate
  neighbor 169.254.8.2 route-map BGP-LOCAL-PREF in
  neighbor 169.254.8.2 route-map BGP-AS-PREPEND out
  neighbor 169.254.9.2 remote-as 7224
  neighbor 169.254.9.2 password *****
  neighbor 169.254.9.2 activate
  network 10.10.0.0 mask 255.255.0.0
  network 10.20.0.0 mask 255.255.0.0
  network 10.30.0.0 mask 255.255.0.0
  distribute-list ACL-BGP-IN in
  no auto-summary
  no synchronization
 exit-address-family
!

My route-map

asa/pri/act# sh run route-map
!
route-map BGP-LOCAL-PREF permit 10

 set local-preference 90
!
route-map BGP-AS-PREPEND permit 10

 set as-path prepend 65501 65501 65501
!

So far so good till here in BGP summery 

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
169.254.8.2     4         7224 76      61            15    0    0 00:10:49  4
169.254.9.2     4         7224 286     225           15    0    0 00:41:45  4

Now problem start here, from my LAN i can't ping AWS any instance.

If i go ahead and shutdown one of link then my LAN hosts can ping AWS instance, I am trying to create redendency here and somehow its not working, Did i miss something here?

As soon as i do following it works..

neighbor 169.254.8.2 shutdown

I heard we shouldn't use 169.254/16 in local-link, do you think that could be the issue here?

UPDATE -1

  Gateway of last resort is 74.XX.XX.107 to network 0.0.0.0

    S*       0.0.0.0 0.0.0.0 [1/0] via 74.XX.XX.107, outside
    S        10.0.10.0 255.255.255.0 [1/0] via 10.5.0.1, site1
    C        10.5.0.0 255.255.248.0 is directly connected, site1
    L        10.5.3.1 255.255.255.255 is directly connected, site1
    S        10.5.8.0 255.255.248.0 [1/0] via 10.5.0.1, site1
    S        10.5.16.0 255.255.248.0 [1/0] via 10.5.0.1, site1
    S        10.5.255.0 255.255.255.0 [1/0] via 10.5.0.1, site1
    C        10.10.0.0 255.255.0.0 is directly connected, dmz_ext
    L        10.10.0.1 255.255.255.255 is directly connected, dmz_ext
    C        10.20.0.0 255.255.0.0 is directly connected, dmz_int
    L        10.20.0.1 255.255.255.255 is directly connected, dmz_int
    C        10.30.0.0 255.255.0.0 is directly connected, inside
    L        10.30.0.1 255.255.255.255 is directly connected, inside
    C        10.40.0.0 255.255.0.0 is directly connected, pxe
    L        10.40.0.1 255.255.255.255 is directly connected, pxe
    B        10.100.4.0 255.255.252.0 [20/0] via 169.254.9.2, 00:50:35
    C        74.XX.XX.0 255.255.254.0 is directly connected, public
    L        74.XX.XX.10 255.255.255.255 is directly connected, public
    C        74.XX.XX.104 255.255.255.248 is directly connected, outside
    L        74.XX.XX.110 255.255.255.255 is directly connected, outside
    C        169.254.8.0 255.255.255.248 is directly connected, aws_dx_2
    L        169.254.8.1 255.255.255.255 is directly connected, aws_dx_2
    C        169.254.9.0 255.255.255.248 is directly connected, aws_dx
    L        169.254.9.1 255.255.255.255 is directly connected, aws_dx
    C        172.29.0.0 255.255.0.0 is directly connected, ilo
    L        172.29.0.1 255.255.255.255 is directly connected, ilo
    C        192.168.100.0 255.255.255.0 is directly connected, FailoverLink
    L        192.168.100.1 255.255.255.255 is directly connected, FailoverLink

If you notice i have only single BGP route in my table.

B        10.100.4.0 255.255.252.0 [20/0] via 169.254.9.2, 00:58:40

Update -2

I did clear asp drop and now seeing this.

Frame drop:
  No valid adjacency (no-adjacency)                                           12
  Flow is denied by configured rule (acl-drop)                               133
  First TCP packet not SYN (tcp-not-syn)                                      54
  TCP RST/FIN out of order (tcp-rstfin-ooo)                                    4
  Slowpath security checks failed (sp-security-failed)                         1
  ICMP Inspect seq num not matched (inspect-icmp-seq-num-not-matched)         31
  FP L2 rule drop (l2_acl)                                                   270

Best Answer

I would check to see where return traffic from AWS is going. If having both links turned up causes traffic to stop flowing though your ASA then there is a good chance that you have asymmetric routing. Depending on the logging level on your ASA there should be a bunch of half formed connection errors in logging as well.

Related Topic