#7-2020-Mar-“gRPC: Establish a secure connection to backend within Kubernetes cluster. SSL until Kubernetes-cluster, safe within cluster”

Deploying anything to my Kubernetes cluster have become more and more a trivial task. But, as soon as I don’t do the normal python web API with a Deployment, Headless service and Ingress I get nervous. Why? Getting SSL, DNS and load balancing is as easy as using some annotations on the ingress:

    certmanager.k8s.io/acme-challenge-type: http01 # SSL
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod # SSL
    dns.alpha.kubernetes.io/external: {{ $url }} #DNS
    kubernetes.io/ingress.class: nginx #Load balancing

I don’t know how every component works but putting these annotations makes my life easy.

Today is my day for getting uncomfortable and diving into gRPC.

  1. I want to have a grpc-client (on my localhost) talking to a server running in the Kubernetes cluster.

  2. I want to avoid generating self signed certificates, I want to reuse the ones I have from letsencrypt

  3. I want the grpc-server to run in insecure mode


  1. grpc use HTTP2, how will the load balancer and nginx-ingress deal with that?

  2. What is tls-termination and how do I specify the certificates on the client side?


After a few hours of head-bagging and frustration I got it working:



  1. Additional annotations on the ingress:

        nginx.ingress.kubernetes.io/ssl-redirect: "true"
        nginx.ingress.kubernetes.io/backend-protocol: GRPC
  2. Use default root certificates on the grpc client

    import certifi
    import grpc
    url = "dns:grpc-example.espen.wheelme-web.com"
    with open(certifi.where(), "rb") as c:
        cert = c.read()
    credentials = grpc.ssl_channel_credentials(root_certificates=cert)
    channel = grpc.secure_channel(url, credentials)
  3. Ensure nginx-ingress is running with image version > 0.29.0. I used the helm chart with commit=bbd5a0c and chart.yaml:

    version: 1.31.0
    appVersion: 0.29.0

Final thoughts

  1. I love how deployments are made easier by using annotations on ingresses

  2. I hate banging my head against the wall. It took my a long time to figure out step 2. I thought being explicit in the url would magically understand https and make a secure connection.

  3. Reusing tls from trusted CAs is not the most common use case. Maybe there is a security reason I am unaware of?

  4. If you can use some extra time to understand exactly what you are doing or why something is failing. It is definitely worth it. Monkey mind, changing one parameter at a time and expecting a different result is never a good approach.

  5. I am excited to start using gRPC and hope to use nginx and oauth2 to perform the authentication by simply adding annotations.

    nginx.ingress.kubernetes.io/auth-url: https://$host/oauth2/auth
    nginx.ingress.kubernetes.io/auth-signin: https://$host/oauth2/start?rd=$request_uri


  1. https://github.com/kubernetes/ingress-nginx/tree/master/docs/examples/grpc -> Good guides for the manifests (yaml) files. Also, recommend installing grpcurl

  2. https://github.com/kubernetes/ingress-nginx/issues/4622 -> Dangerous issue thread but useful