Define Google Cloud Managed Service for Monitoring

You may have seen this notice when opening SLOs Overview in Cloud Console.

This notice announces a recent change in the way of defining services for Cloud Monitoring. Before the change, Cloud Monitoring automatically discovered services that were provisioned in AppEngine, Cloud Run or GKE. These services were automatically populated in the Services Overview dashboard. After the change, all services in the Services Overview dashboard have to be created explicitly. To simplify this task, when defining a new service in UI you are presented with a list of candidates that is built based on the auto-discovered services. The full list of the auto-discovered services includes managed services from AppEngine, Cloud Run and Istio as well as GKE workloads and services. Besides UI you can add managed services to Cloud Monitoring using the services.create API or using the Terraform google_monitoring_service resource.

For example, if you have a GKE cluster named cluster-001 provisioned in the us-central1 region that has a service frontend in the default namespace, the following command in Cloud Shell will define this service for Cloud Monitoring:

curl -X POST \
  <https://monitoring.googleapis.com/v3/\>
  projects/${GOOGLE_CLOUD_PROJECT}/services?service_id=frontend \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d \
'
{
  "gkeService": {
    "clusterName": "cluster-001",
    "location": "us-central1",
    "namespaceName": "default",
    "serviceName": "frontend"
  }
}'

When using the Terraform resource, the keys for the service_labels argument should be converted from the camel case notation (in documentation) to the snake case notation. For example, the command above will look in Terraform like the following:

resource "google_monitoring_service" "frontend" {
  service_id = "frontend"
  basic_service {
    service_type = "GKE_SERVICE"
    service_labels = {
      location : "us-central1",
      cluster_name : "cluster-001",
      service_namespace : "default",
      service_name : "frontend"
    }
  }
}

When your definition of the service does not match one to one with one of the managed services you can add it to Cloud Monitoring by defining a custom service. You will use the same API request:

curl -X POST \
  <https://monitoring.googleapis.com/v3/\>
  projects/${GOOGLE_CLOUD_PROJECT}/services?service_id=custom_svc \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d \
'
{
  "displayName": "custom sevice"
  "custom": {}
}'

Or you will use a designated Terraform resource, google_monitoring_custom_service:

resource "google_monitoring_custom_service" "custom_svc" {
  service_id = "custom_svc"
  display_name = "custom service"
}

Compared to a custom service, the auto-detected services come with two predefined SLIs for availability and latency. These SLIs utilize the metrics of the managed services that are automatically captured such as request processing time or HTTP request status. For custom services these SLIs have to be defined explicitly using request-based or window-based SLIs.

Check creating SLOs and SLO-based alerts to find more information about tracking your service SLO and error budgets. Check what predefined SLIs are used in avaiability and latency SLOs.