How to integrate Istio with AWS Private Certificate Authority
February 24, 2023
Arka Bhattacharya
When users install Istio, by default a certificate authority (CA) is created and a CA certificate is self-signed. The CA certificate is then used to manage the certificate lifecycle for applications running on the Istio service mesh. This enables easy setup of out-of-the-box mTLS communication between applications running on the Istio service mesh.
As we move toward a production implementation, Istio also provides an option to bring your own CA certificate instead of using the default self-signed CA certificate. Enterprises prefer automating their public/private key infrastructure, known as PKI, to secure, manage, and create certificates. As a general practice, Enterprises would use a private CA certificate signed by their root certificate or another intermediate to set up the certificate authority for their service mesh.
Out of the many available options, AWS Private CA is a managed CA that helps organizations secure their applications and devices using private certificates in AWS. Additionally, Cert Manager is another popular choice for simplifying the process of obtaining, renewing, and using those certificates. Cert Manager and AWS Private CA issuer plugins are commonly used together to set up the custom CA for the Istio service mesh.
Some of the key benefits of this integration are:
Automation of certificate lifecycle management, ensuring that certificate(s) are renewed before expiry to avoid any downtime.
Easy auditability of private key infrastructure.
Flexible configuration options via the Certificate CRD to meet the needs of a wider audience.
Visual representation of the different components involved
In the following sections, we share the steps on how to set up a custom CA for your Istio service mesh, with AWS Private CA as the private key infrastructure.
Install Cert Manager
Cert Manager simplifies the process of obtaining, renewing, and using certificates. It supports integration with a wide range of private key infrastructures including AWS Private CA. In this section we will install Cert Manager with Helm. Alternate installation options are present here.
# verify
kubectl -n cert-manager rollout status deploy/cert-manager;
kubectl -n cert-manager rollout status deploy/cert-manager-cainjector;
kubectl -n cert-manager rollout status deploy/cert-manager-webhook;
Create the CA hierarchy in AWS
In this step, we create a root certificate authority and a subordinate certificate authority in the AWS Private CA. Here is related documentation from AWS, which talks about designing a CA hierarchy.
Please note: The following script creates a root CA and a subordinate CA in the AWS Private CA. You can skip running the following script if you already have a CA hierarchy present in AWS Private CA, and would like to reuse that hierarchy instead of creating a new root CA and a new subordinate CA from that root CA.
If you wish to re-use an existing AWS Private CA object, kindly save the ARN of the same in the environment variable ISTIO_CA_ARN. We will be using this variable in future steps.
Note, if you wish to use this script to create your private CA hierarchy, kindly edit the COUNTRY, ORGANIZATION, ORGANIZATIONAL_UNIT, STATE, LOCALITY as per your requirement.
#!/bin/bash
set -e
# Purpose:
# Creates one Root CA and a Subordinate CA in AWS Private Certificate Authority(PCA).
# - Subordinate CA is meant to be used to setup the custom CA for Istio.
# Note: Please feel free to edit the following section as per your need for the CA Subject details.
export COUNTRY="US"
export ORGANIZATION="Solo.io"
export ORGANIZATIONAL_UNIT="Consulting"
export STATE="MA"
export LOCALITY="Boston"
# Note: Please feel free to edit the values for Validity time of the Root cert and the Subordinate cert.
export ROOT_CERT_VALIDITY_IN_DAYS=3650
export SUBORDINATE_CERT_VALIDITY_IN_DAYS=1825
##
# Process Flow for AWS Private CA setup
# - Create config json file with details for the Certificate Authority.
# - Create CA (Root or Subordinate) in AWS Private CA.
# - Download CSR file corresponding to the newly created Private CA.
# - Issue certificate for the CA with the help of the downloaded CSR and the Private CA ARN.
# > Note: Use "--certificate-authority-arn" parameter for issuing a cert for Subordinate/Intermediate CA)
# - Import this certificate in AWS Private CA.
##
echo
echo "[INFO] Creates the root private certificate authority (CA)."
# https://docs.aws.amazon.com/cli/latest/reference/acm-pca/create-certificate-authority.html
ROOT_CAARN=$(aws acm-pca create-certificate-authority \
--certificate-authority-configuration file://ca_config_root_ca.json \
--certificate-authority-type "ROOT" \
--idempotency-token 01234567 \
--output json \
--tags Key=Name,Value=RootCA | jq -r '.CertificateAuthorityArn')
echo "[INFO] Sleeping for 15 seconds for CA creation to be completed..."
sleep 15
echo "[DEBUG] ARN of Root CA=${ROOT_CAARN}"
echo "[INFO] download Root CA CSR from AWS"
# https://docs.aws.amazon.com/cli/latest/reference/acm-pca/get-certificate-authority-csr.html
aws acm-pca get-certificate-authority-csr \
--certificate-authority-arn "${ROOT_CAARN}" \
--output text > root-ca.csr
echo "[INFO] Issue Root Certificate. Valid for ${ROOT_CERT_VALIDITY_IN_DAYS} days"
# https://docs.aws.amazon.com/cli/latest/reference/acm-pca/issue-certificate.html
ROOT_CERTARN=$(aws acm-pca issue-certificate \
--certificate-authority-arn "${ROOT_CAARN}" \
--csr fileb://root-ca.csr \
--signing-algorithm "SHA256WITHRSA" \
--template-arn arn:aws:acm-pca:::template/RootCACertificate/V1 \
--validity Value=${ROOT_CERT_VALIDITY_IN_DAYS},Type="DAYS" \
--idempotency-token 1234567 \
--output json | jq -r '.CertificateArn')
echo "[INFO] Sleeping for 15 seconds for cert issuance to be completed..."
sleep 15
echo "[DEBUG] ARN of Root Certificate=${ROOT_CERTARN}"
echo "[INFO] Retrieves root certificate from private CA and save locally as root-ca.pem"
# https://docs.aws.amazon.com/cli/latest/reference/acm-pca/get-certificate.html
aws acm-pca get-certificate \
--certificate-authority-arn "${ROOT_CAARN}" \
--certificate-arn "${ROOT_CERTARN}" \
--output text > root-ca.pem
echo "[INFO] Import the signed Private CA certificate for the CA specified by the ARN into ACM PCA"
# https://docs.aws.amazon.com/cli/latest/reference/acm-pca/import-certificate-authority-certificate.html
aws acm-pca import-certificate-authority-certificate \
--certificate-authority-arn "${ROOT_CAARN}" \
--certificate fileb://root-ca.pem
echo "-----------------------------------------------------------"
echo "ARN of Root CA is ${ROOT_CAARN}"
echo "-----------------------------------------------------------"
##
# Intermediate CA setup section
# Note: If you wish to create more than 1 subordinate CAs please add more constants in the for loop
##
for CA_FOR_COMPONENT in "Istio"
do
echo
echo "###########################################################"
echo " Create Intermediate CA for ${CA_FOR_COMPONENT}"
echo " Generated and managed by ACM, signed by Root CA"
echo "###########################################################"
cat <<EOF > "ca_config_intermediate_ca_${CA_FOR_COMPONENT}.json"
{
"KeyAlgorithm":"RSA_2048",
"SigningAlgorithm":"SHA256WITHRSA",
"Subject":{
"Country":"${COUNTRY}",
"Organization":"${ORGANIZATION}",
"OrganizationalUnit":"${ORGANIZATIONAL_UNIT}",
"State":"${STATE}",
"Locality":"${LOCALITY}",
"CommonName":"Intermediate CA ${CA_FOR_COMPONENT}"
}
}
EOF
echo "[INFO] Create the Subordinate/Intermediate private certificate authority (CA) for ${CA_FOR_COMPONENT}"
# https://docs.aws.amazon.com/cli/latest/reference/acm-pca/create-certificate-authority.html
SUBORDINATE_CAARN=$(aws acm-pca create-certificate-authority \
--certificate-authority-configuration file://ca_config_intermediate_ca_${CA_FOR_COMPONENT}.json \
--certificate-authority-type "SUBORDINATE" \
--idempotency-token 01234567 \
--tags Key=Name,Value="SubordinateCA-${CA_FOR_COMPONENT}" | jq -r '.CertificateAuthorityArn')
echo "[INFO] Sleeping for 15 seconds for CA creation to be completed..."
sleep 15
echo "[DEBUG] ARN of Subordinate CA for ${CA_FOR_COMPONENT}=${SUBORDINATE_CAARN}"
echo "[INFO] Download Intermediate CA CSR from AWS"
# https://docs.aws.amazon.com/cli/latest/reference/acm-pca/get-certificate-authority-csr.html
aws acm-pca get-certificate-authority-csr \
--certificate-authority-arn "${SUBORDINATE_CAARN}" \
--output text > "intermediate_ca_${CA_FOR_COMPONENT}.csr"
echo "[INFO] Issue Intermediate Certificate for ${CA_FOR_COMPONENT}. Valid for ${SUBORDINATE_CERT_VALIDITY_IN_DAYS} days."
SUBORDINATE_CERTARN=$(aws acm-pca issue-certificate \
--certificate-authority-arn "${ROOT_CAARN}" \
--csr fileb://intermediate_ca_${CA_FOR_COMPONENT}.csr \
--signing-algorithm "SHA256WITHRSA" \
--template-arn arn:aws:acm-pca:::template/SubordinateCACertificate_PathLen1/V1 \
--validity Value=${SUBORDINATE_CERT_VALIDITY_IN_DAYS},Type="DAYS" \
--idempotency-token 1234567 \
--output json | jq -r '.CertificateArn')
echo "[INFO] Sleeping for 15 seconds for cert issuance to be completed..."
sleep 15
echo "[DEBUG] ARN of Subordinate CA Certificate for ${CA_FOR_COMPONENT}=${SUBORDINATE_CERTARN}"
echo "[INFO] Retrieve Intermediate certificate from private CA and save locally as intermediate-cert.pem"
aws acm-pca get-certificate \
--certificate-authority-arn "${ROOT_CAARN}" \
--certificate-arn "${SUBORDINATE_CERTARN}" \
--output json | jq -r '.Certificate' > "intermediate-cert-${CA_FOR_COMPONENT}.pem"
echo "[INFO] Retrieve Intermediate certificate chain from private CA and save locally as intermediate-cert-chain.pem"
aws acm-pca get-certificate \
--certificate-authority-arn "${ROOT_CAARN}" \
--certificate-arn "${SUBORDINATE_CERTARN}" \
--output json | jq -r '.CertificateChain' > "intermediate-cert-chain-${CA_FOR_COMPONENT}.pem"
echo "[INFO] Import the certificate into ACM PCA"
aws acm-pca import-certificate-authority-certificate \
--certificate-authority-arn "${SUBORDINATE_CAARN}" \
--certificate fileb://intermediate-cert-${CA_FOR_COMPONENT}.pem \
--certificate-chain fileb://intermediate-cert-chain-${CA_FOR_COMPONENT}.pem
# cleanup
rm "ca_config_intermediate_ca_${CA_FOR_COMPONENT}.json" "intermediate_ca_${CA_FOR_COMPONENT}.csr" "intermediate-cert-${CA_FOR_COMPONENT}.pem" "intermediate-cert-chain-${CA_FOR_COMPONENT}.pem"
echo "-----------------------------------------------------------"
echo "ARN of ${CA_FOR_COMPONENT} CA is ${SUBORDINATE_CAARN}"
echo "-----------------------------------------------------------"
done
After running the above script, kindly save the ARN in ISTIO_CA_ARNfor future steps:
To access any AWS resource, we need the right level of permissions defined. In this section, we define a new IAM policy to only grant the required level of access.
We use “aws-privateca-issuer” Cert Manager plugin to interact with the AWS Private CA. The plugin needs the right access to interact with the AWS Private CA. It needs to use the IAM Policy that we created in the previous step. In this step, we create a ServiceAccount for the “aws-privateca-issuer” plugin, and associate the ServiceAccount with a newly created IAM Role, which has the IAM Policy attached.
Scenario: cacerts Kubernetes secret is not getting created despite creating the required custom resources:
Checking AWSPCAIssuer object status should be helpful.
Reviewing AWSPCAIssuer spec is suggested.
When we create a custom Certificate object, a related CertificateRequest object gets created. Status of the CertificateRequest object contains helpful debug information.
Checking logs of cert manager and AWS PCA Issuer plugin is also recommended.
How Solo.io can help
You don’t have to go it alone. At Solo.io we offer consulting services so your team and your business will thrive on your journey.
At Solo.io we live and breathe enterprise Istio Service Mesh, API gateway, API management, and GraphQL, including best practices implementing CI/CD. We build solutions on AWS, Azure, GCP, OpenShift, VMs, and local Kubernetes deployments. Contact us here to learn more.
