Authenticated gRPC service on Kubernetes

In our blog series on Kubernetes, we talked about buildings scalable MLOps on Kubernetes, architecture for MLOps, and solving application development. In this blog, we will talk about hosting a GRPC service on AWS EKS cluster. The process is roughly going to be the same for every Kubernetes cluster — however, we had to do some specific settings on the AWS Load balancer for this to work.

What is gRPC & our usecase

gRPC is an open-source RPC framework that can run in any environment. It is capable of efficiently connecting services within and between data centers, with the ability to plug in support for load balancing, tracing, health checking, and authentication.

Our usecase:  Hosting Tensorflow models as APIs that accepted a payload of around 100MB in size. GRPC performs much better for larger payloads — so we exposed the GRPC port on port 5000.

Hosting the service

We hosted the service on Kubernetes using the Deployment YAML below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-api
  namespace: ml-services
spec:
  replicas: 1
  selector:
    matchLabels:
      truefoundry.com/component: ml-api
  template:
    metadata:
      labels:
        truefoundry.com/application: ml-api
    spec:
      containers:
        - name: ml-api
          image: >-
            XXXX.dkr.ecr.us-east-1.amazonaws.com/ml-services-ml-api:latest
          ports:
            - name: port-8500
              containerPort: 8500
              protocol: TCP
          resources:
            limits:
              cpu: '4'
              ephemeral-storage: 2G
              memory: 4G
            requests:
              cpu: '1'
              ephemeral-storage: 1G
              memory: 500M
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      imagePullSecrets:
        - name: ml-api-image-pull-secret
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 0

This will bring up the pod. We need to create the Service object using the YAML below:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  labels:
    argocd.argoproj.io/instance: tfy-istio-ingress
  name: tfy-wildcard
  namespace: istio-system
spec:
  selector:
    istio: tfy-istio-ingress
  servers:
    - hosts:
        - 'ml.example.com'
      port:
        name: http-tfy-wildcard
        number: 80
        protocol: HTTP
      tls:
        httpsRedirect: true
    - hosts:
        - 'ml.example.com'
      port:
        name: https-tfy-wildcard
        number: 443
        protocol: HTTP

Exposing the service

We are using Istio as the ingress layer in Kubernetes. Istio provisions a Load Balancer when the istio-ingress is installed. The load balancer configuration can be customized using annotations on the istio gateway. The spec for creating the Istio Gateway is as follows:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  labels:
    argocd.argoproj.io/instance: tfy-istio-ingress
  name: tfy-wildcard
  namespace: istio-system
spec:
  selector:
    istio: tfy-istio-ingress
  servers:
    - hosts:
        - 'ml.example.com'
      port:
        name: http-tfy-wildcard
        number: 80
        protocol: HTTP
      tls:
        httpsRedirect: true
    - hosts:
        - 'ml.example.com'
      port:
        name: https-tfy-wildcard
        number: 443
        protocol: HTTP

We are doing the SSL termination on the AWS Load Balancer. For this we have to attach the certificate to the Load Balancer. This can be achieved using the annotations below to the istio gateway chart (https://istio-release.storage.googleapis.com/charts).

"service.beta.kubernetes.io/aws-load-balancer-type": "nlb"
"service.beta.kubernetes.io/aws-load-balancer-backend-protocol": "tcp"
"service.beta.kubernetes.io/aws-load-balancer-ssl-cert": "<certificate-arn>"
"service.beta.kubernetes.io/aws-load-balancer-ssl-ports": "https"
"service.beta.kubernetes.io/aws-load-balancer-alpn-policy": "HTTP2Preferred"

The alpn-policy is important to specify to allow GRPC traffic. Our service ml-api can be exposed by creating a VirtualService pointing to the Kubernetes service. The YAML for the Virtual Service is as follows:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  labels:
    argocd.argoproj.io/instance: ml-services_ml-api
  name: ml-apiport-8500-vs
  namespace: ml-services
spec:
  gateways:
    - istio-system/tfy-wildcard
  hosts:
    - ml.example.com
  http:
    - route:
        - destination:
            host: ml-api
            port:
              number: 8500

Once the virtual service is exposed, we can make requests to our service at ml.example.com. We then wanted to add an authentication to the API so that everybody cannot call the API. We could have added the authentication in the code, but we decided to add it at the istio layer so that it can be a unified layer across all services.

To add authentication at the istio-ingress layer, we decided to go ahead with a IstioWasm Plugin. The yaml for the plugin looks something like:

apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
  name: ml-services-ml-api-0
  namespace: istio-system
spec:
  phase: AUTHN
  pluginConfig:
    basic_auth_rules:
      - credentials:
          - username:password
        hosts:
          - ml.example.com
        prefix: /
        request_methods:
          - GET
          - PUT
          - POST
          - PATCH
          - DELETE
  selector:
    matchLabels:
      istio: tfy-istio-ingress
  url: oci://ghcr.io/istio-ecosystem/wasm-extensions/basic_auth:1.12.0

Once you have applied the above spec to the cluster, the app will ask for the username and password once you open it in the browser.

We have made it super easy on TrueFoundry

To make the above process much easier, we decide to make it really easy on the Truefoundry platform.