Random failed_to_connect_to_backend errors on GCE LB

health checks managed instance groups VS health checks load balancing

The health checks used by managed instance groups are the same health checks used by load balancing, with some differences in behavior. Health checks that you apply to load balancing services help a load balancer determine where to direct network traffic. These health checks do not cause Compute Engine to recreate instances. Health checks that you apply to managed instance groups will proactively signal to the managed instance group to delete and recreate instances if they become UNHEALTHY.

For the majority of scenarios, use separate health checks for load balancing and for monitoring managed instance groups. Health checking for load balancing can and should be more aggressive since these health checks determine whether an instance receives user traffic. Since customers might rely on your services, you want to catch non-responsive instances quickly so you can redirect traffic if necessary. In contrast, health checking for instance groups will cause Compute Engine proactively replace failing instances so you could create health checks that are more conservative than health checks for a load balancer.

https://cloud.google.com/compute/docs/instance-groups/creating-groups-of-managed-instances#monitoring_groups

Traffic from the load balancer to your instance is not enabled by default. Unfortunately this is not well documented, and really, when you create a load balancer this should happen automatically.

Try adding this firewall rule the network that your load balancer and VMs are on:

130.211.0.0/22   tcp:1-5000   Apply to all targets