NGINX vs. GCE Kubernetes ingress classes

google-cloud-platformgoogle-kubernetes-enginekubernetesload balancingnginx

When setting up a Kubernetes ingress on Google Container Engine, you can choose the ingress class (gce or nginx). I realize that the GCE class provisions a load balancer on Google's Cloud Platform, which costs about $20/mo each.

After some research, I couldn't find any prevailing reason why Google's load balancer is any better than using the NGINX ingress class—at least not before hitting a very large scale.

In fact, it appears that the GCE class does not support:

External authentication with ingress.kubernetes.io/auth-url
Basic authentication
A few other small features built into the NGINX ingress

Are there any benefits I'm not aware of for using the GCE class vs. the NGINX class for ingresses?

Best Answer

After some further reading and testing, I did find a few benefits. I also realized that using the NGINX controller would still provision a load balancer... thereby not avoiding the ~$20/mo cost.

Differences

The GCE controller causes an HTTP(S) load balancer to be provisioned
The NGINX controller requires a service to be designated as type: LoadBalancer
- Doing this causes a network load balancer to be provisioned

HTTP(S) load balancer

Network load balancer

Benefits:

As shown in the diagrams above, the HTTP(S) load balancer can load balance across regions, whereas the network load balancer can only load balance across zones in the same region
The GCE controller is built specifically for Google's Cloud Platform, so I assume it works more reliably
The default GCE controller requires no extra effort compared to maintaining and specifying the YAML files required for the NGINX controller to function

Related Solutions

Docker – How to choose the external IP address of a Kubernetes load balancer in Google Kubernetes Engine

TL;DR Google Container Engine running Kubernetes v1.1 supports loadBalancerIP just mark the auto-assigned IP as static first.

Kubernetes v1.1 supports externalIPs:

apiVersion: v1
kind: Service
spec:
  type: LoadBalancer
  loadBalancerIP: 10.10.10.10
  ...

So far there isn't a really good consistent documentation on how to use it on GCE. What is sure is that this IP must first be one of your pre-allocated static IPs.

The cross-region load balancing documentation is mostly for Compute Engine and not Kubernetes/Container Engine, but it's still useful especially the part "Configure the load balancing service".

If you just create a Kubernetes LoadBalancer on GCE, it will create a network Compute Engine > Network > Network load balancing > Forwarding Rule pointing to a target pool made of your machines on your cluster (normally only those running the Pods matching the service selector). It looks like deleting a namespace doesn't nicely clean-up the those created rules.

Update

It is actually now supported (even though under documented):

Check that you're running Kubernetes 1.1 or later (under GKE edit your cluster and check "Node version")
Under Networking > External IP addresses you should have already some Ephemeral marked as pointing to your cluster's VM instance (if not or unsure, deploy once without loadBalancerIP, wait until you've an external IP allocated when you run kubectl get svc, and look up that IP in the list on that page). Mark one of them as static, let's say it External Address is 10.10.10.10.
Edit your LoadBalancer to have loadBalancerIP=10.10.10.10 as above (adapt to the IP that was given to you by Google).

Now if you delete your LoadBalancer or even your namespace, it should preserve that IP address upon re-reploying on that cluster. If you need to change the cluster, some manual fiddling should be possible:

Under “Network load balancing” section, “Target pools” tab, click “Create target pool” button:
- Name: cluster-pool (or any other name)
- Region: Select the region of one of your clusters
- Health Check: Optional, if you wish
- Select existing instance groups: Your Kubernetes cluster
Under “Network load balancing” section, “Forwarding rules” tab, click “Create forwarding rule” button:
- Name: http-cross-region-gfr (or any other name)
- Region: Select the region of one of your clusters
- External IP: Select loadbalancer-ip-crossregion you just reserved
- Target pool: Select cluster-pool you just created

Expose port 80 and 443 on Google Container Engine without load balancer

Yep, through externalIPs on the service. Example service I've used:

apiVersion: v1
kind: Service
metadata:
  name: bind
  labels:
    app: bind
    version: 3.0.0
spec:
  ports:
    - port: 53
      protocol: UDP
  selector:
    app: bind
    version: 3.0.0
  externalIPs:
    - a.b.c.d
    - a.b.c.e

Please be aware that the IPs listed in the config file must be the internal IP on GCE.