GitOps is becoming increasingly popular approach to manage Kubernetes components. It works by using Git as a single source of truth for declarative infrastructure and applications, allowing your application definitions, configurations, and environments to be declarative and version controlled. This helps to make these workflows automated, auditable, and easy to understand.
Purpose of This Tutorial
The main goal of this tutorial is to showcase how Gloo Mesh components can seamlessly integrate into a GitOps workflow, with Argo CD being our tool of choice. We’ll guide you through the installation of Argo CD, Gloo Platform, Istio, and we’ll explore the Gloo Mesh Dashboard.
Beyond the installation walkthroughs, we’ll engage in practical exercises specifically designed to showcase the robustness of the system. These exercises serve as a technical demonstration of how the pull/sync mechanism offered by Argo CD enhances the reliability and resilience of your applications and service mesh.
Where This Has Been Tested
Kubernetes versions 1.25-1.28
Google Kubernetes Engine using 2x n2-standard-4 instance types
AWS EKS using 2x m5.xlarge instance types
Microsoft AKS using 2x Standard_DS3_v2 instance types
Locally using K3D and Colima+K3S on Macbook Pro M1
High Level Architecture
Request a Trial
If you are interested in walking through the following steps hands-on, request a trial license of Gloo Mesh!
Prerequisites
This tutorial assumes a single Kubernetes cluster for demonstration. Instructions have been validated on k3d, as well as in EKS and GKE. Please note that the setup and installation of Kubernetes are beyond the scope of this guide. Ensure that your cluster contexts are named gloo by running:
kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* gloo k3d-gloo admin@k3d-gloo
Set Environment Variables
To get started, set the MY_CLUSTER_CONTEXT and MY_CLUSTER_NAME vars for our cluster.
If your local clusters have a different context name, you will want to have it match the expected context name(s). In this example, we are setting the context name as gloo.
Reminder if you want a specific version of Istio or to use the officially supported images provided by Solo.io, get the Hub value from the Solo support page for Istio Solo images. The value is present within the Solo.io Istio Versioning Repo key section.
Otherwise, we can use the 1.20.2-solo/ Istio image provided in our Solo Workshops.
Provide the Gloo Mesh version:
export GLOO_MESH_VERSION=2.5.0
Installing Argo CD
Let’s start by deploying Argo CD to our gloo cluster context.
Now that we have Gloo Mesh installed, we can use the IstioLifecycleManager CRD to deploy and manage Istio across all of our workload clusters. This allows us to treat our Istio deployment like any other CRD in the Kubernetes cluster, rather than a separate deployment mechanism such as using helm install to deploy a chart. While there is nothing wrong the Helm-based approach (and in the next section of this blog we will also review using Helm + Argo CD to install Istio) the IstioLifecycleManager API was designed to provide value at scale, such as being able to target multiple clusters at a time to handle day 2 operations such as updating Istio to a new image for a security patch, or upgrading to a new release.
The following instructions will configure the lifecycle manager CRDs directly, but afterwards we will explore how to automate this with an Argo CD Application.
Let’s deploy the Istio control plane using the IstioLifecycleManager:
kubectl apply --context "${MY_CLUSTER_CONTEXT}" -f - <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: IstioLifecycleManager
metadata:
name: istiod-control-plane
namespace: gloo-mesh
spec:
installations:
# The revision for this installation, such as 1-20
- revision: ${ISTIO_REVISION}
# List all workload clusters to install Istio into
clusters:
- name: ${MY_CLUSTER_NAME}
# If set to true, the spec for this revision is applied in the cluster
defaultRevision: true
# When set to true, the lifecycle manager allows you to perform in-place upgrades by skipping checks that are required for canary upgrades
skipUpgradeValidation: true
istioOperatorSpec:
# Only the control plane components are installed
# (https://istio.io/latest/docs/setup/additional-setup/config-profiles/)
profile: minimal
# Solo.io Istio distribution repository; required for Gloo Istio.
# You get the repo key from your Solo Account Representative.
hub: ${HUB}
# Any Solo.io Gloo Istio tag
tag: ${ISTIO_VERSION}
namespace: istio-system
# Mesh configuration
meshConfig:
# Enable access logging only if using.
accessLogFile: /dev/stdout
# Encoding for the proxy access log (TEXT or JSON). Default value is TEXT.
accessLogEncoding: JSON
# Enable span tracing only if using.
enableTracing: true
defaultConfig:
# Wait for the istio-proxy to start before starting application pods
holdApplicationUntilProxyStarts: true
proxyMetadata:
# Enable Istio agent to handle DNS requests for known hosts
# Unknown hosts are automatically resolved using upstream DNS servers
# in resolv.conf (for proxy-dns)
ISTIO_META_DNS_CAPTURE: "true"
# Enable automatic address allocation (for proxy-dns)
ISTIO_META_DNS_AUTO_ALLOCATE: "true"
# Set the default behavior of the sidecar for handling outbound traffic
# from the application
outboundTrafficPolicy:
mode: ALLOW_ANY
# The administrative root namespace for Istio configuration
rootNamespace: istio-system
# Traffic management
values:
global:
meshID: gloo-mesh
network: ${MY_CLUSTER_NAME}
multiCluster:
clusterName: ${MY_CLUSTER_NAME}
# Traffic management
components:
pilot:
k8s:
env:
# Disable selecting workload entries for local service routing.
# Required for Gloo VirtualDestinaton functionality.
- name: PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES
value: "false"
# Reload cacerts when cert-manager changes it
- name: AUTO_RELOAD_PLUGIN_CERTS
value: "true"
EOF
You can check to see that istiod has been deployed.
kubectl get pods -n istio-system --context "${MY_CLUSTER_CONTEXT}"
Output should look similar to below:
NAME READY STATUS RESTARTS AGE
istiod-1-20-78b54758c5-q852m 1/1 Running 0 2m16s
Next we will configure the istio-gateways namespace and Kubernetes service for the gateway. Separating the Kubernetes Service is recommended because it allows us to manage the lifecycle of the load balancer in front of the Istio ingressgateway separate from the lifecycle of the deployment. For example, having full control over the revision selector of the Service and when to make the traffic switchover when doing a canary upgrade.
kubectl apply --context "${MY_CLUSTER_CONTEXT}" -f - <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: GatewayLifecycleManager
metadata:
name: istio-ingressgateway
namespace: gloo-mesh
spec:
installations:
# The revision for this installation, such as 1-20
- gatewayRevision: ${ISTIO_REVISION}
# List all workload clusters to install Istio into
clusters:
- name: ${MY_CLUSTER_NAME}
# If set to true, the spec for this revision is applied in the cluster
activeGateway: true
istioOperatorSpec:
# No control plane components are installed
profile: empty
# Solo.io Istio distribution repository; required for Gloo Istio.
# You get the repo key from your Solo Account Representative.
hub: ${HUB}
# The Solo.io Gloo Istio tag
tag: ${ISTIO_VERSION}
values:
gateways:
istio-ingressgateway:
customService: true
components:
ingressGateways:
- name: istio-ingressgateway
namespace: istio-gateways
enabled: true
label:
istio: ingressgateway
EOF
You can check to see that the Istio ingressgateway has been deployed.
kubectl get pods -n istio-gateways --context "${MY_CLUSTER_CONTEXT}"
Output should look similar to below:
NAME READY STATUS RESTARTS AGE
istio-ingressgateway-1-20-5bc944987-882ls 1/1 Running 0 55s
Configuring IstioLifecycleManager or GatewayLifecycleManager with Argo CD
Since we can treat the IstioLifecycleManager and GatewayLifecycleManager the same as any other Kubernetes CRD, we can deploy Istio on our cluster by using an Argo Application that is configured to deploy any valid YAML configuration in the /lifecyclemanagerdirectory in this repo. This will allow us to configure all of the steps we just did with one Argo CD Application resource.
Note the use of the annotation argocd.argoproj.io/sync-wave in each manifest, which can help with the deployment ordering of each component. The order of operations would go least to greatest. More on Argo CD sync-waves.
The corresponding Argo Application would look like this, but note that the example here is set to use a cluster named gloo, Istio version 1.20.2-solo and Istio revision set to 1-20 and would require a PR to this repo to be re-configured.
In the future, we can commit more Istio configuration to this directory to continue building out our cluster, or even to deploy Istio onto other newly onboarded workload clusters.
Installing Istio with Helm
NOTE: If you have already completed the installation using IstioLifecycleManager, you can skip this step completely.
Let’s set our environment variables for this exercise.
Reminder if you want a specific version of Istio or to use the officially supported images provided by Solo.io, get the Hub value from the Solo support page for Istio Solo images. The value is present within the Solo.io Istio Versioning Repo key section.
Otherwise, we can use the 1.20.2 Istio image provided in our Solo Workshops.
kubectl apply --context "${MY_CLUSTER_CONTEXT}" -f- <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: istio-ingressgateway
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
annotations:
argocd.argoproj.io/sync-wave: "-1"
spec:
destination:
server: https://kubernetes.default.svc
namespace: istio-gateways
project: default
source:
chart: gateway
repoURL: https://istio-release.storage.googleapis.com/charts
targetRevision: "${ISTIO_VERSION}"
helm:
values: |
# Name allows overriding the release name. Generally this should not be set
name: "istio-ingressgateway-${ISTIO_REVISION}"
# revision declares which revision this gateway is a part of
revision: "${ISTIO_REVISION}"
replicaCount: 1
service:
# Type of service. Set to "None" to disable the service entirely
type: LoadBalancer
ports:
- name: http2
port: 80
protocol: TCP
targetPort: 80
- name: https
port: 443
protocol: TCP
targetPort: 443
annotations:
# AWS NLB Annotation
service.beta.kubernetes.io/aws-load-balancer-type: "nlb-ip"
loadBalancerIP: ""
loadBalancerSourceRanges: []
externalTrafficPolicy: ""
# Pod environment variables
env: {}
annotations:
proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'
# Labels to apply to all resources
labels:
istio.io/rev: ${ISTIO_REVISION}
istio: ingressgateway
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
EOF
You can check to see that istiod and the istio ingressgateways have been deployed.
kubectl get pods -n istio-system --context "${MY_CLUSTER_CONTEXT}" && \
kubectl get pods -n istio-gateways --context "${MY_CLUSTER_CONTEXT}"
Output should look similar to below:
NAME READY STATUS RESTARTS AGE
istiod-1-20-6-86499c5945-bbsfl 1/1 Running 0 38m
NAME READY STATUS RESTARTS AGE
istio-ingressgateway-1-20-6-6575484979-5fbn7 1/1 Running 0 36m
Additionally you can use meshctl to do the port-forward for you.
meshctl dashboard
At this point, you should have ArgoCD, Gloo Mesh, and Istio installed on the cluster!
Let's Deploy Our First App!
Let’s deploy an Application using Argo CD! Similar to before, we will confgure an Argo Application that is configured to deploy any valid YAML configuration in the easybutton/workloads directory onto the cluster.
In the easybutton/workloads directory is the Bookinfo application, the corresponding RouteTable, and a VirtualGateway. In the future, we can commit more mesh configuration to this directory to continue deploying new applications to this cluster.
Confirm that the virtual gateway was configured:
kubectl get virtualgateway -n istio-gateways --context "${MY_CLUSTER_CONTEXT}"
Output should look similar to below.
NAME AGE
north-south-gw 100s
Confirm that a route table was created.
kubectl get routetables -A --context "${MY_CLUSTER_CONTEXT}"
Output should look similar to below.
NAMESPACE NAME AGE
bookinfo-frontends bookinfo-routetable 51s
You can check to see that the bookinfo application has been deployed.
kubectl get pods -n bookinfo-frontends --context "${MY_CLUSTER_CONTEXT}" && \
kubectl get pods -n bookinfo-backends --context "${MY_CLUSTER_CONTEXT}"
Output should look similar to below:
NAME READY STATUS RESTARTS AGE
productpage-v1-b8f647485-cnzqj 2/2 Running 0 3m8s
NAME READY STATUS RESTARTS AGE
ratings-v1-85b5789856-ppzzg 2/2 Running 0 3m8s
reviews-v2-56bc64445f-c54fn 2/2 Running 0 3m8s
reviews-v1-749bbb44c5-bp8z4 2/2 Running 0 3m8s
details-v1-984bdbc4d-7zjjp 2/2 Running 0 3m8s
The configured virtual gateway and route table exposes the Bookinfo application on port 80 of the Istio Ingress Gateway we deployed earlier. You should be able to access it with the following command:
echo "access the Bookinfo application at http://$(kubectl --context "${MY_CLUSTER_CONTEXT}" get svc -n istio-gateways --selector=istio=ingressgateway -o jsonpath='{.items[*].status.loadBalancer.ingress[0].*}')/productpage"
Observe All the Things!
By seamlessly integrating with popular tools like Grafana and Prometheus, implementing Istio allows organizations to gain valuable insights into the health, performance, and overall behavior of their distributed applications within the service mesh.
Similar to before, we will confgure an Argo Application that is configured to deploy any valid YAML configuration in the easybutton/observability directory onto the cluster. This application will manage the deployment of Prometheus and Grafana, configure default dashboards, and expose the Grafana UI to us through the gateway with another route table.
You can check to see that the Grafana and Prometheus applications have been deployed.
kubectl get pods -n grafana --context "${MY_CLUSTER_CONTEXT}"
kubectl get pods -n istio-system --context "${MY_CLUSTER_CONTEXT}"
Output should look similar to below:
NAME READY STATUS RESTARTS AGE
grafana-576f6587c5-v4bjr 2/2 Running 0 13m
NAME READY STATUS RESTARTS AGE
istiod-1-20-6b99fdd545-c8m5v 1/1 Running 0 47m
prometheus-6cf7c7cb94-bpn5v 2/2 Running 0 23m
You can see that a few default dashboards configured using configmaps have been created.
kubectl get configmap -n grafana --context "${MY_CLUSTER_CONTEXT}"
Output should look similar to below:
NAME DATA AGE
solo-istio-dashboard 1 34m
kubernetes-dashboard 1 34m
istio-grafana-dashboards 6 34m
Confirm that an additional route table was created as well.
kubectl get routetables -A --context "${MY_CLUSTER_CONTEXT}"
Output should look similar to below.
NAMESPACE NAME AGE
bookinfo-frontends bookinfo-routetable 16m
gloo-mesh grafana-routetable 119s
The configured virtual gateway and route table exposes the Grafana application on port 80 at the /grafana path of the Istio Ingress Gateway we deployed earlier. You should be able to access it with the following command:
echo "access the Grafana UI at http://$(kubectl --context "${MY_CLUSTER_CONTEXT}" get svc -n istio-gateways --selector=istio=ingressgateway -o jsonpath='{.items[*].status.loadBalancer.ingress[0].*}')/grafana"
If you navigate to the available dashboards, we should see a kubernetes dashboard, the upstream community Istio dashboards, as well as the Solo Istio Performance Dashboard!
Resiliency Testing
From a resiliency perspective, deploying Gloo Mesh with Argo CD or any other GitOps tool provides a few clear benefits to a manual or traditional push based approach.
Declarative Configuration: Argo CD enables the definition of the desired state of your system through declarative configuration stored in Git, ensuring a clear and version-controlled source of truth.
Automated Synchronization: The pull-based approach of Argo CD ensures automatic synchronization between the desired state in Git and the actual state in the Kubernetes cluster, minimizing manual intervention and reducing the risk of configuration drift.
Self-Healing Mechanisms: Argo CD can automatically detect discrepancies between the desired and actual states and initiate corrective actions, contributing to a self-healing mechanism that enhances the overall resilience of the deployed components.
We have already demonstrated using declarative configuration and Git as our source of truth when deploying and installing Gloo Mesh, but let’s take a look at auto sync and self-healing.
Scenario 1 – Infrastructure Resiliency
For Kubernetes Pods, in the event of a pod failure the scheduler will automatically start a new pod. But what happens if the Deployment is deleted?
In Kubernetes, removing a Deployment such as the gloo-mesh-mgmt-server is assumed to be part of a controlled process. Kubernetes does not automatically create replacement Pods in this case. Deleting a Deployment is often done when you want to intentionally scale down or update the application.
Let’s take a look at the Gloo Mesh Deployment.
kubectl get deployments -n gloo-mesh --context "${MY_CLUSTER_CONTEXT}"
With Argo CD, in the Application resource we can set selfHeal: true and prune: true options so that if a Deployment in an application controlled by Argo CD is deleted, it will trigger Argo CD’s reconciliation process. This includes self-healing mechanisms to ensure the desired state of the Kubernetes cluster is declared state in Git and automatically takes corrective actions to make the system to manual changes.
Since we deployed Gloo Mesh using Argo CD, let’s create some real chaos and delete all of the deployments in the gloo-mesh namespace and see what happens.
In a typical installation, all of the deployments would be deleted permanently. Let’s take a look at what happens when we are using the self-healing capabilities of Argo CD.
kubectl get pods -n gloo-mesh --context "${MY_CLUSTER_CONTEXT}"
kubectl get deployments -n gloo-mesh --context "${MY_CLUSTER_CONTEXT}"
kubectl get secrets -n gloo-mesh --context "${MY_CLUSTER_CONTEXT}"
You can see almost immediately that all of the deployments are re-synced to the cluster.
Another important detail to note is that Gloo Mesh is not in the data path! This means that although the entire deployment was deleted and restarted, the traffic routing to our applications remains unaffected.
Scenario 2 – Configuration Resiliency
Similarly to keeping the health of our Applications and Infra alive, we can benefit from self-healing capabilities by syncing our application networking configuration as well. Let’s test another scenario in which a piece of our service mesh configuration is mistakenly removed from the cluster and see what happens.
Let’s take a look at our virtual gateways and route tables.
kubectl get routetables -A --context "${MY_CLUSTER_CONTEXT}" && \
kubectl get virtualgateway -A --context "${MY_CLUSTER_CONTEXT}"
Output should look similar to below.
NAMESPACE NAME AGE
bookinfo-frontends bookinfo-routetable 2m41s
NAMESPACE NAME AGE
istio-gateways north-south-gw 2m41s
Now let’s take down both our virtual gateway and route table. In our current environment this would just be the Bookinfo application, but a similar scenario in a large environment could have significant consequences affecting multiple teams.
Again we see Argo CD to the rescue! Instead of suffering a total outage, Argo CD detected a drift in the desired state and reconciled the virtual gateway and route table.
Scenario 3 – Application Resiliency
To show another example of how Argo CD resiliency capabilities can help, let’s explore a scenario where a user can intentionally/unintentionally modify the deployment of a running application such as the image of a running Pod. In this scenario, without Argo CD the image update would be treated as an intentional action — however with Argo CD in place we should observe a syncing event to reconfigure to the source of truth in Git.
First, in a separate terminal we can do a watch on the image. Note, you may need to set your MY_CLUSTER_CONTEXT variable again.
Now we can update the image of the productpage-v1 deployment to bogus-container:hack and see what happens.
kubectl --context "${MY_CLUSTER_CONTEXT}" set image deployment/productpage-v1 -n bookinfo-frontends productpage=bogus-container:hack
What we should see is that even though the image was updated, it was automatically reconciled back to the desired state in Git. The output of the watch command should reflect this.
To streamline deployment, utilize the Argo app-of-apps pattern again to manage and orchestrate the entire setup in one Application!
However, since the state is committed to Git, we have a few requirements:
Argo CD installed on the cluster (preferably fresh install)
Cluster context named gloo
Renaming Cluster Context
If your local clusters have a different context name, you will want to have it match the expected context name(s). In this example, we are setting the context name as gloo.
In case you haven’t configured it already, Gloo Mesh Enterprise requires a Trial License Key.
To avoid storing the license key in Git, we can enhance security by utilizing a licenseSecretKeyRef in the Helm values, necessitating the creation of the license key as a secret.
GLOO_MESH_LICENSE_KEY=<<input_license_key_here>
The license is stored as a base64 encoded secret, set your the following variable based on your OS:
For Linux:
# Linux
BASE64_LICENSE_KEY=$(echo -n "${GLOO_MESH_LICENSE_KEY}" | base64 -w 0)
Additionally you can use meshctl to do the port-forward for you.
meshctl dashboard
Access the Bookinfo Application
The configured route table exposes the Bookinfo application on port 80 of the Istio Ingress Gateway we deployed earlier. You should be able to access it with the following command:
echo "access the Bookinfo application at http://$(kubectl --context "${MY_CLUSTER_CONTEXT}" get svc -n istio-gateways --selector=istio=ingressgateway -o jsonpath='{.items[*].status.loadBalancer.ingress[0].*}')/productpage"
Access the Grafana UI
The configured virtual gateway and route table exposes the Grafana application on port 80 at the /grafana path of the Istio Ingress Gateway we deployed earlier. You should be able to access it with the following command:
echo "access the Grafana UI at http://$(kubectl --context "${MY_CLUSTER_CONTEXT}" get svc -n istio-gateways --selector=istio=ingressgateway -o jsonpath='{.items[*].status.loadBalancer.ingress[0].*}')/grafana"
Cleanup
To uninstall, you can delete the easybutton Argo Application.
Gloo Mesh delivers a more powerful service mesh solution, powered by Istio and Cilium, for your cloud-native workloads. Boost security, resiliency, and observability at your organization. Learn more about Gloo Mesh today.
