Secure your workloads with cert-manager in Istio’s ambient mode using istio-csr

Many enterprises use cert-manager to manage the lifecycle of their Kubernetes certificates. istio-csr is an agent provided by the cert-manager project to allow management of Istio's control plane and data plane certificates 

Istio’s ambient mode, which has recently become generally available, has a shared proxy on each node called the ztunnel. A ztunnel acts on behalf of all the workloads running on its node, which means it needs certificates for each workload in order to be able to participate in mTLS. When fetching certificates, ztunnel will authenticate to the certificate authority (CA) with its own identity, but request the identity of another workload. Critically, the CA will enforce that the ztunnel has permission to request that identity. Requests for identities not running on the node are rejected.

The new v0.12 of istio-csr supports the ability to set trusted CA node service accounts, which allows it to correctly serve certificates to ztunnel in this fashion.

In this article, you will walk through how to install Istio in ambient mode and integrate with cert-manager.

Prepare a cluster

You will need an empty cluster for this exercise. Use your preferred Kubernetes provider — cloud or local — and don't install anything on it yet. One node will suffice, but be aware that ztunnel is a DaemonSet, and on larger clusters you will need to identify the node which is running a specific workload to query the logs for the specific ztunnel hosting it.

Install cert-manager

Install cert-manager using Helm:

helm repo add jetstack https://charts.jetstack.io --force-update
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.16.1 \
  --set crds.enabled=true

Create Issuer and Issuing Certificate

The issuer used by cert-manager will be installed in the istio-system namespace.  Create it first:

kubectl create namespace istio-system

Now, create the issuer that istio-csr will use, as well as the issuing certificate:

kubectl apply -f- << EOF
# Self sSigned issuers are useful for creating root certificates
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned
  namespace: istio-system
spec:
  selfSigned: {}
---
# Request a self-signed certificate from our Issuer; this will function as our
# issuing root certificate when we pass it into a CA Issuer.

# It's generally fine to issue root certificates like this one with long lifespans;
# the certificates which istio-csr issues will be much shorter lived.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: istio-ca
  namespace: istio-system
spec:
  isCA: true
  duration: 87600h # 10 years
  secretName: istio-ca
  commonName: istio-ca
  privateKey:
    algorithm: ECDSA
    size: 256
  subject:
    organizations:
    - cluster.local
    - cert-manager
  issuerRef:
    name: selfsigned
    kind: Issuer
    group: cert-manager.io
---
# Create a CA issuer using our root. This will be the Issuer which istio-csr will use.
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: istio-ca
  namespace: istio-system
spec:
  ca:
    secretName: istio-ca
EOF

Export the root CA to a local file

Extract the cert from the istio-ca issuer to create the istio-root-ca needed by cert-manager:

# Export our cert from the secret it's stored in, and base64 decode to get the PEM data.
kubectl get -n istio-system secret istio-ca -ogo-template='{{index .data "tls.crt"}}' | base64 -d > ca.pem

# For interest, you can check out what our CA certificate looks like
openssl x509 -in ca.pem -noout -text

# Add our CA to a secret
kubectl create secret generic -n cert-manager istio-root-ca --from-file=ca.pem=ca.pem

Install istio-csr

Add the values to the Helm installation to point to our static root CA:

helm upgrade cert-manager-istio-csr jetstack/cert-manager-istio-csr \
  --install \
  --version=0.12.0 \
  --namespace cert-manager \
  --wait \
  --set "app.tls.rootCAFile=/var/run/secrets/istio-csr/ca.pem" \
  --set "app.tls.certificateDNSNames={cert-manager-istio-csr.cert-manager.svc,istio-csr.cert-manager.svc}" \
  --set "app.server.caTrustedNodeAccounts=istio-system/ztunnel" \
  --set "volumeMounts[0].name=root-ca" \
  --set "volumeMounts[0].mountPath=/var/run/secrets/istio-csr" \
  --set "volumes[0].name=root-ca" \
  --set "volumes[0].secret.secretName=istio-root-ca"

Note that when you install istio-csr, you must define the caTrustedNodeAccounts property with the ztunnel service account for the impersonation to work. 

Double check all the pods are running and that you have a istio-csr pod in state Running:

# Check to see that the istio-csr pod is running and ready
kubectl get pods -n cert-manager | grep csr
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-istio-csr-7cbc554bfc-8cd7z    1/1     Running   0          115s

Install Istio in ambient mode

When you install Istio, you need to disable the default provided CA server, and pass the address where Istio needs to communicate with cert-manager.

helm repo add istio https://istio-release.storage.googleapis.com/charts
helm repo update

helm install istio-base istio/base \
     --namespace istio-system      \
     --version=1.24.0 \
     --wait

helm install istiod istio/istiod \
     --namespace istio-system \
     --version=1.24.0 \
     --set profile=ambient \
     --set "pilot.env.ENABLE_CA_SERVER=false" \
     --set "global.caAddress=cert-manager-istio-csr.cert-manager.svc:443" \
     --set "global.meshID=cluster.local" \
     --wait

helm install istio-cni istio/cni \
     --namespace istio-system \
     --version=1.24.0 \
     --set profile=ambient \
     --wait

helm install ztunnel istio/ztunnel \
     --namespace istio-system \
     --version=1.24.0 \
     --set "caAddress=cert-manager-istio-csr.cert-manager.svc:443" \
     --wait

kubectl get crd gateways.gateway.networking.k8s.io &> /dev/null || \
  { kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml; }

Confirm the Istio components are running:

kubectl get pods -n istio-system
NAME                                   READY   STATUS    RESTARTS   AGE
istio-cni-node-mp8cd                   1/1     Running   0          3h14m
istio-ingressgateway-c6f8fdf96-cmrfr   1/1     Running   0          3h9m
istiod-574b657c8b-bknrz                1/1     Running   0          3h17m
ztunnel-lmnhk                          1/1     Running   0          3h13m

Run a sample workload

In order to test your CA setup, deploy a sample workload.  the httpbin sample and label the namespace to be included in the ambient mesh:

kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/httpbin/httpbin.yaml

In another terminal, start watching the certificate signing requests. 

kubectl get certificaterequests.cert-manager.io -n istio-system -w
NAME         APPROVED   DENIED   READY   ISSUER       REQUESTOR                                         AGE
istio-ca-1   True                True    selfsigned   system:serviceaccount:cert-manager:cert-manager   35m
istiod-2     True                True    istio-ca     system:serviceaccount:cert-manager:cert-manager   4m46s

You will see the certificate that was issued to the control plane. 

Now, enable ambient mode:

kubectl label namespace default istio.io/dataplane-mode=ambient

In your other terminal, you will see a request was created, quickly became Approved, and then became Ready.

NAME         APPROVED   DENIED   READY   ISSUER       REQUESTOR                                         AGE
istio-ca-1   True                True    selfsigned   system:serviceaccount:cert-manager:cert-manager   35m
istiod-2     True                True    istio-ca     system:serviceaccount:cert-manager:cert-manager   4m46s
istio-csr-rmzpp                               istio-ca     system:serviceaccount:cert-manager:cert-manager-istio-csr   0s
istio-csr-rmzpp   True                        istio-ca     system:serviceaccount:cert-manager:cert-manager-istio-csr   0s
istio-csr-rmzpp   True                True    istio-ca     system:serviceaccount:cert-manager:cert-manager-istio-csr   0s
istio-csr-rmzpp   True                True    istio-ca     system:serviceaccount:cert-manager:cert-manager-istio-csr   0s

You can also correlate this with cert-manager’s logs:

kubectl logs -n cert-manager $(kubectl get pods -n cert-manager -o jsonpath='{.items..metadata.name}' --selector app=cert-manager) --since 2m -f
I1115 00:24:11.861423       1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "istio-csr-rmzpp" condition "Approved" to 2024-11-15 00:24:11.86139125 +0000 UTC m=+1410.576942267
I1115 00:24:11.869135       1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "istio-csr-rmzpp" condition "Ready" to 2024-11-15 00:24:11.869126875 +0000 UTC m=+1410.584677850

The logs from istio-csr should reflect the trust domain you defined in the certificate:

kubectl logs -n cert-manager deployments/cert-manager-istio-csr -f
I0926 20:19:28.154889       1 tls.go:247] "renewing serving certificate" logger="tls-provider"
I0926 20:19:28.243664       1 tls.go:389] "serving certificate ready" logger="tls-provider"
2024-09-26T20:19:28.243708Z     info    spiffe  Added 1 certs to trust domain cluster.local in peer cert verifier
I0926 20:19:28.243800       1 tls.go:249] "fetched new serving certificate" logger="tls-provider" expiry-time="2024-09-26 21:19:28 +0000 UTC"
I0926 20:19:28.243825       1 tls.go:226] "waiting to renew certificate" logger="tls-provider" renewal-time="2024-09-26 20:59:28.081275292 +0000 UTC m=+4799.974134549"

Validate certificates

The ztunnel logs will report the establishment of a proxy for the httpbin pod at the time you labeled the default namespace to use the ambient data plane.

Tip: If running a cluster with more than one node, find the node which the httpbin workload is running by running kubectl get pods -n default -l app=httpbin -o wide. Query the ztunnel app in the  istio-system namespace to find out which ztunnel is running on that same node, and view only the logs from that one. In this example, our ztunnel pod is ztunnel-lmnhk.

kubectl logs -n istio-system ztunnel-lmnhk
2024-11-15T00:24:11.073994Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2024-11-15T00:24:11.074120Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2024-11-15T00:24:11.074149Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2024-11-15T00:24:11.074157Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2024-11-15T00:24:11.838900Z	info	xds::client:xds{id=1}	received response	type_url="type.googleapis.com/istio.workload.Address" size=1 removes=0

If ztunnel were to fail the CSR, the logs will reflect the request authenticate failure for the httpbin workload:

2024-11-15T00:28:39.550909Z     error   cert_fetcher    unable to prefetch cert for "spiffe://cluster.local/ns/default/sa/httpbin", skipping, SigningRequest(Status { code: Unauthenticated, message: "request authenticate failure", metadata: MetadataMap { headers: {"content-type": "application/grpc"} }, source: None })


istioctl includes the ztunnel-config command which you can use to validate ambient mode configuration. You can retrieve the certificates for the specified ztunnel pod:

istioctl ztunnel-config certificate ztunnel-lmnhk.istio-system
CERTIFICATE NAME                                 TYPE     STATUS        VALID CERT     SERIAL NUMBER                        NOT AFTER                NOT BEFORE
spiffe://cluster.local/ns/default/sa/httpbin     Leaf     Available     true           67682361356572e8433ce715e9f0dd85     2024-11-15T01:24:11Z     2024-11-15T00:24:11Z
spiffe://cluster.local/ns/default/sa/httpbin     Root     Available     true           a20262d6b78916602c50259a44a9e5b3     2034-11-13T00:01:19Z     2024-11-15T00:01:19Z

Finally we can get the certificate, and confirm it was issued by cert-manager:

istioctl ztunnel-config certificate ztunnel-lmnhk.istio-system -o json | jq -r '.[0].certChain[0].pem' | base64 -d | openssl x509 -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            67:68:23:61:35:65:72:e8:43:3c:e7:15:e9:f0:dd:85
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: O=cert-manager + O=cluster.local, CN=istio-ca
        Validity
            Not Before: Nov 15 00:24:11 2024 GMT
            Not After : Nov 15 01:24:11 2024 GMT
        Subject: 
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:a7:57:a8:34:11:49:f5:ae:87:fd:62:74:1e:f8:
                    cd:51:58:41:eb:cb:8e:8a:81:c2:5c:4d:11:01:01:
                    af:0e:a1:ba:91:f3:3b:4e:5b:b7:6b:32:a4:9f:be:
                    59:95:26:07:b8:42:2c:c6:b4:d6:75:cb:e3:8d:c3:
                    ae:87:0b:73:e9
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Extended Key Usage: 
                TLS Web Client Authentication, TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier: 
                78:36:C1:72:FF:31:6F:BB:26:26:40:EF:02:64:D1:06:FF:11:2E:57
            X509v3 Subject Alternative Name: critical
                URI:spiffe://cluster.local/ns/default/sa/httpbin
    Signature Algorithm: ecdsa-with-SHA256
    Signature Value:
        30:45:02:20:74:bc:f6:70:8d:5f:fd:ce:4b:81:51:89:cd:bc:
        02:ca:38:1f:2e:1d:7d:1f:64:a5:13:58:e5:9b:01:5c:5a:66:
        02:21:00:ad:87:cb:ce:a6:06:ad:be:4e:7c:16:8a:c6:5b:eb:
        11:0c:98:f5:71:fd:19:9e:69:68:f4:6d:7a:ae:fa:a6:0e

In conclusion

cert-manager is a popular certificate authority for Kubernetes, and using istio-csr, you can now integrate it with Istio’s ambient mode.

At Solo.io we work with customers deploying ambient mesh to consistently manage application traffic in production. We are the leading contributors to ambient mesh, and provide enhancements to ztunnel in Gloo Mesh  to help you further simplify your mesh adoption.

Explore ambient mesh today, and reach out to us for any questions.