AKS with istio and cert-manager Part 1

2019-11-01

In this particular article, I shall share my experience of deploying services on azure aks and exposing the services with istio gateway and TLS with cert-manager.

I would assume that you already have a RBAC kubernetes cluster running on azure.

Firstly install the istio component on the kubernetes cluster

This installs all the necessary istio crds.

helm repo add istio.io https://storage.googleapis.com/istio-release/releases/1.3.3/charts/ && helm repo update
helm upgrade istio-init istio.io/istio-init - install - namespace istio-system - debug

Install istio components with the sds enabled. For now enable it via helm parameters, later below I will explain the need of this feature.

helm upgrade istio-system istio.io/istio-system - namespace istio-system - set gateways.istio-ingressgateway.sds.enabled=true - debug - install

This also creates a ingressgateway service with the type LoadBalancer. Azure takes care of assigning a public IP for the service( You may need to wait for few minutes to get the Public IP assigned)

I am using azuredns, but if you are using any other providers such as godaddy. ! Update your dns zone with the IP above as a A record with the domain you have( You may need to check the support of valid DNS01 providers with Cert Manager). For Eg. lets consider here IP as 1.2.3.4 and domain as test.domain.edu.(just remember this domain since it is needed further for DNS challenge for cert manager)

Install the cert manager for the official site with setup. For simplicity I have copied it below. However keep track of updated versions on the official site.

# Install the CustomResourceDefinition resources separately
kubectl apply --validate=false -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.11/deploy/manifests/00-crds.yaml

# Create the namespace for cert-manager
kubectl create namespace cert-manager

# Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io

# Update your local Helm chart repository cache
helm repo update

# Install the cert-manager Helm chart
helm install \
  --name cert-manager \
  --namespace cert-manager \
  --version v0.11.0 \
  jetstack/cert-manager

Before configuring the Cert manager Issuer and Certificate We need to know few things on what Issuer does. The Issuer basically is a ACME handler which talks to the LetsEncrypt Server with necessary details mentioned below. I am choosing DNS01 challenge to generate certificate for my domain. In order to create DNS challenge we need to provide Issuer a role (In azure it is called a service principal) who can create a txt record in DNS zone and verify the Domain belongs to me. In order to facilitate this we will create azure service principal with a DNS Zone contributor role. Note that a secret (azuredns-config) is created with the client secret .

WARN: To create a role you need to be the owner of the Subscription at least.

#!/bin/bash
AZURE_CERT_MANAGER_SP_NAME=SOME_SERVICE_PRINCIPAL_NAME
AZURE_CERT_MANAGER_DNS_RESOURCE_GROUP=SOME_RESOURCE_GROUP
AZURE_CERT_MANAGER_DNS_NAME=SOME_DNS_ZONE

DNS_SP=$(az ad sp create-for-rbac --name $AZURE_CERT_MANAGER_SP_NAME)
AZURE_CERT_MANAGER_SP_APP_ID=$(echo $DNS_SP | jq -r '.appId')
AZURE_CERT_MANAGER_SP_PASSWORD=$(echo $DNS_SP | jq -r '.password')

# Lower the Permissions of the SP
az role assignment delete --assignee $AZURE_CERT_MANAGER_SP_APP_ID --role Contributor

# Give Access to DNS Zone
DNS_ID=$(az network dns zone show --name $AZURE_CERT_MANAGER_DNS_NAME --resource-group $AZURE_CERT_MANAGER_DNS_RESOURCE_GROUP --query "id" --output tsv)

az role assignment create --assignee $AZURE_CERT_MANAGER_SP_APP_ID --role "DNS Zone Contributor" --scope $DNS_ID

# Check Permissions
az role assignment list --assignee $AZURE_CERT_MANAGER_SP_APP_ID

# Create Secret
kubectl create secret generic azuredns-config \
  --from-literal=CLIENT_SECRET=$AZURE_CERT_MANAGER_SP_PASSWORD

# Get the Service Principal App ID for configuration
echo "Principal: $AZURE_CERT_MANAGER_SP_APP_ID"
echo "Password: $AZURE_CERT_MANAGER_SP_PASSWORD"

Reference: https://docs.cert-manager.io/en/latest/tasks/issuers/setup-acme/dns01/azuredns.html

Now create Issuer with above generated client id and client secret( secret created above as azuredns-config).

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: myname@example.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the DNS-01 challenge provider
    solvers:
    - dns01:
        azuredns:
          # Service principal clientId (also called appId)
          clientID: <CLIENT_ID>
          # A secretKeyRef to a service principal ClientSecret (password)
          # ref: https://docs.microsoft.com/en-us/azure/container-service/kubernetes/container-service-kubernetes-service-principal
          clientSecretSecretRef:
            name: azuredns-config
            key: CLIENT_SECRET
          # Azure subscription Id
          subscriptionID: <AZURE_SUBSCRIPTION_ID>
          # Azure AD tenant Id
          tenantID: <AZURE_TENANT_ID>
          # ResourceGroup name where dns zone is provisioned
          resourceGroupName: <RESOURCE_GRP_OF_DNS_ZONE>
          hostedZoneName: domain.edu
          # Azure Cloud Environment, default to AzurePublicCloud
          environment: AzurePublicCloud

Create a Certificate. Note that the secret which we created(azuredns-config) is referred in the Certificate below.

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: ingress-cert
spec:
  secretName: azuredns-config
  issuerRef:
    name: letsencrypt-prod
    kind: Issuer
  commonName: domain.edu
  dnsNames:
  - domain.edu
  acme:
    config:
    - dns01:
        provider: azuredns
      domains:
      - domain.edu

The Issuer performs the DNS challenge with the Client id and Secret and on successful verification it updates the secret(azuredns-config) with the private key(tls.key) and the certificate(tls.crt)

You can check the events by describing the cert-manager controller pod. In short cert-manager does following steps shown as Kubernetes Resources.

Certificate > CertificateRequest > Order > Challenge This generates certificate

Now the certificate is generated and stored in the secret, we may want the Istio Gateway to use this certificate but still one final nail in the coffin is missing, that is updating the gateway with the secret. The credentialName is equivalent to the [.spec.tls.host.secretName] in Ingress Resource. We need to create a Gateway Resource and configure to use the Istio Gateway with the selector as shown below.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: http-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
  - hosts:
    - '*'
    port:
      name: https-443
      number: 443
      protocol: HTTPS
    tls:
      credentialName: azuredns-config
      mode: SIMPLE

For this above, remember we have enabled SDS on istio via helm charts. This creates 2 containers in the istio ingress gateway pod. 1 is ingress-sds and 2 is istio-proxy. To focus on the former one, in short the sds will push the generated certificate as secret to the envoy instance and in our case its istio gateway.(For more details please read https://www.envoyproxy.io/docs/envoy/latest/configuration/security/secret)

Just check the logs of the ingress-sds pod and below log should ensure that the certificate is successfully pushed to our gateway

2019-11-08T16:01:59.181820Z     info    citadel agent monitor has started.
2019-11-08T16:01:59.181824Z     info    sdsServiceLog   Start SDS grpc server for ingress gateway proxy
2019-11-08T16:01:59.181936Z     info    monitor Monitor server started.
2019-11-08T16:02:04.978761Z     info    sdsServiceLog   CONNECTION ID: router~15.0.0.26~istio-ingressgateway-6b554c75-4pbb5.istio-system~istio-system.svc.cluster.local-1, RESOURCE NAME: azuredns-config, EVENT: pushed key/cert pair to proxy

After this just restart your istio ingress gateway pod if the changes don't show up soon!( this is little flaky since I have seen it takes few minutes to get this reflected)

Next Article has details of deploying simple service to make the mashup work.!

Credits: Envoy SDS: Fortifying Istio Security - Yonggang Liu & Quanjie Lin, Google (https://www.youtube.com/watch?v=QlQyqCaTOh0)

Note: the domain mentioned above is just an example!#