Migrate from Legacy Stackdriver Monitoring in GKE

In this post, I will show how to upgrade the existing GKE cluster to Stackdriver Kubernetes Engine Monitoring using Terraform.

As per Google Documentation, Legacy Stackdriver Monitoring and Logging has been deprecated and recommends to migrate to Stackdriver Kubernetes Engine Monitoring.

What’s the difference?

Stackdriver Kubernetes Engine Monitoring uses a different models to store and serve GKE metrics, logs, and metadata. Notable changes:

• Resource name has changed from gce_instance to k8s_node.
• Metrics prefix has changed from container.googleapis.com/ to kubernetes.io/.
  

More details can be found on official  documentation

Before starting migration

As the metrics prefix and resource name has changed, existing Dashboard, Alert Policies will get affected after migration and needs configuration update.
To identify the affected Stackdriver configuration, GCP Console provides a Single-pane window(named Kubernetes Migration Status) to view the resource that needs an update.

To view Kubernetes Migration Status, go to

• Monitoring in Google Cloud Console
• Click Settings and then Kubernetes Migration Status

This will show a Dashboard as below and gives you all the information about the resources that need an update after migrating to Stackdriver Kubernetes Engine Monitoring.

Below is the screenshot of my demo project. It shows I have one Dashboard left to update.

Enabling new Stackdriver Monitoring & Logging

This blog assumes that GKE cluster has been created using Terraform. I will focus on properties to change to enable new Stackdriver.

Sample Terraform code of GKE cluster running on Legacy Stackdriver looks something like below:

resource "google_container_cluster" "my_cluster" {
  name     = "app-cluster"
  location = "europe-west1"

  ...
  ...

  monitoring_service = "monitoring.googleapis.com"
  logging_service    = "logging.googleapis.com"
}

In above code, monitoring.googleapis.com and logging.googleapis.com refers to the Legacy Stackdriver Monitoring & Logging.

To upgrade the GKE Cluster, we just need to update these values as below:

resource "google_container_cluster" "my_cluster" {
  name     = "app-cluster"
  location = "europe-west1"

  ...
  ...

  monitoring_service = "monitoring.googleapis.com/kubernetes"
  logging_service    = "logging.googleapis.com/kubernetes"
}

After updating the code, run terraform plan and terraform apply.

Note: It might take several minutes for GKE to update the cluster and reflect the new values.
Note: It’s not clear in the Terraform resource documentation that monitoring.googleapis.com/kubernetes is the new Stackdriver Monitoring & Logging so I have raised a PR. Hopefully, it gets merged and documentation becomes clearer.

Post Migration

Once the GKE cluster reflects the new Monitoring & Logging mode, update your affected resources (Dashboards, Alerts, etc.)

Conclusion

It’s really easy to turn on the new mode of Stackdriver Monitoring and Logging using Terraform.
If you have feedback or questions, please reach out to me on LinkedIn or Twitter