I am getting Could not resolve host metadata.google.internal or Could not resolve host metadata error when I try to access Metadata Server.
Although I use my own VPC with custom firewall, I don't think this is the issue of firewall because according to Google Metadata Server traffic never leaves the Instance. Besides, there's an entry of it in my host file which proves it:
# Google Compute Engine metadata server
169.254.169.254 metadata.google.internal metadata
I tried pinging 169.254.169.254, but it returns General Failure (but, it proves nothing.. I am not sure if Metadata Server allows ICMP).
My Stackdriver Logging Agents are also failing because of this. Here's a line from its log:
Failed to access metadata service: error_class=Errno::ENETUNREACH error="Failed to open TCP connection to 169.254.169.254:80 (A socket operation was attempted to an unreachable network. - connect(2) for \"169.254.169.254\" port 80)"
In case if it matters, the internal traffic in my VPC network is limited to icmp and tcp:22 (ssh) only.
What's happening and how to fix this issue?
Best Answer
GCP metadata traffic never leaves the physical host running the instance. Such requests go out the guest's interfaces, but never get forwarded.
A host level firewall in the instance can prevent traffic from reaching the metadata server. As it drops packets in the instance's network stack before they leave the guest.
Normally the metadata service responds to http, DNS, and ICMP echo, and all of them are not working for you. Only one hop away so not like it can be routed incorrectly. Sure sounds like a firewall.