Traffic not routed back to GKE cluster from remote network via VPN

google-kubernetes-enginevpn

(Followup on GKE pod connecting via VPN?)

I am trying to connect a GKE cluster to a remote network using a GCE VPN to a Cisco ASA 5510. Ping from GKE pod 10.248.0.26 -> remote node 10.99.193.115 arrives at 10.99.193.115 and the ASA says that the echo reply goes back through the tunnel to GKE. However, tcpdump on 10.248.0.26 shows no replies coming in.

Firewall and routing as reported by Google Cloud Console:

Name    Source tag / IP range   Allowed protocols / ports   Target tags

default-allow-icmp  0.0.0.0/0   icmp    Apply to all targets
default-allow-internal  10.240.0.0/16   tcp:1-65535; udp:1-65535; icmp  Apply to all targets
default-allow-ssh   0.0.0.0/0   tcp:22  Apply to all targets
gke-zecluster-d6cc7a55-all  10.248.0.0/14   tcp; udp; icmp;     Apply to all targets
gke-zecluster-d6cc7a55-ssh  <public_ip>/32  tcp:22  gke-zecluster-d6cc7a55-node
gke-zecluster-d6cc7a55-vms  10.240.0.0/16   tcp:1-65535; udp:1-65535; icmp  gke-zecluster-d6cc7a55-node
k8s-fw-a1a92183fb18e11e5be3442010af0001     0.0.0.0/0   tcp:80,443  gke-zecluster-d6cc7a55-node
k8s-fw-a1aa3fe95b18e11e5be3442010af0001     0.0.0.0/0   tcp:2003    gke-zecluster-d6cc7a55-node

Name    Destination IP ranges   Priority    Instance tags   Next hop

default-route-3eed071cad0670e8  0.0.0.0/0   1000    None    Default internet gateway
default-route-7a9ddc4457c714a0  10.240.0.0/16   1000    None    Virtual network
gke-zecluster-d6cc7a55-7b61213c-b187-11e5-be34-42010af00015     10.248.0.0/24   1000    None    gke-zecluster-d6cc7a55-node-j4jx (Zone ze-zone-1)
gke-zecluster-d6cc7a55-7ec5f7a9-b187-11e5-be34-42010af00015     10.248.1.0/24   1000    None    gke-zecluster-d6cc7a55-node-rluf (Zone ze-zone-1)
vpn-1-tunnel-1-route-1  10.99.0.0/16    1000    None    

Is there some logging I can turn on to see what goes on? As far as I can see, the VPN says nothing pertinent about this traffic, only:

15:24:51.058 sending DPD request
15:24:51.058 generating INFORMATIONAL_V1 request 3069408857 [ HASH N(DPD) ]
15:24:51.058 sending packet: from <gce-vpn-ip>[500] to <asa-ip>[500] (92 bytes)
15:24:51.092 received packet: from <asa-ip>[500] to <gce-vpn-ip>[500] (92 bytes)
15:24:51.092 parsed INFORMATIONAL_V1 request 146600869 [ HASH N(DPD_ACK) ]

If I modify the VPN tunnel (GCE VPN, ASA) to have the default net 10.240.0.0/16 at the GCE end traffic passes correctly in both directions.

I assume this is a routing issue, but what? Should not the route 10.248.0.0/24 send the traffic back to the GKE node? Or do I have to somehow declare the GKE network as a network?

Best Answer

If IP address 10.248.0.26 belongs to a GKE node, then for doing ping between the GKE node and your remote node you will need to add a firewall rule on 10.248.0.26/24 network to allow incoming traffic to GKE node or all targets in that network from your remote source.