ADK’s Built-in Observability

Have you ever found yourself grappling with observability instrumentation, even with tools like OpenTelemetry? This challenge becomes even more pronounced when developing Agentic applications, where understanding internal workflows is paramount. There are many frameworks that help developers to create an Agentic application. I’d like to review what observability data is generated when developers use the Agent Development Kit (ADK) from Google. ADK framework comes with logging and tracing instrumentation out-of-the-box.

Logging in ADK

Logging in ADK is implemented using standard logging packages: vanilla logging in Python and Simple Logging Facade for Java (SLF4J) in Java. ADK implementation in all languages adhere to the same core principles:

• Standard Library: Use the language standard logging library, so any configuration or handler that works with it will work with ADK
• Hierarchical Loggers: Use predictable logger identities (names) hierarchically based on the module path (e.g., google_adk.google.adk.agents.llm_agent), allowing for fine-grained control over which parts of the framework produce logs.
• User-Configured: Do not configure logging. Allow configuring the standard logging at application level so the developer using the framework can set up the desired logging configuration in their application’s entry point.

This means you can easily align ADK’s log output format with your existing application’s setup. ADK writes logs as plain text by default. You can customize the log format to be a structured payload. For example, to configure your Python application to output logs in Google Cloud’s structured logging format (including trace correlation), you can use a custom formatter like this:

from datetime import datetime, date
import json, logging
from opentelemetry import trace

...

def json_serial(obj):
    if isinstance(obj, (datetime, date)):
        return obj.isoformat()
    raise TypeError(f"Object of type {obj.__class__.__name__} is not JSON serializable")

class JsonFormatter(logging.Formatter):
    def format(self, record):
        span = trace.get_current_span()
        json_log_object = {
            'severity': record.levelname,
            'message': record.getMessage(),
            'logging.googleapis.com/trace': google_trace_id_format(span.get_span_context().trace_id),
            'logging.googleapis.com/spanId': trace.format_span_id(span.get_span_context().span_id),            
        }
        return json.dumps(json_log_object, default=json_serial)

sh = logging.StreamHandler(sys.stdout)
sh.setFormatter(JsonFormatter())

This code formats logs that are printed to stdout, as a stringified JSON including fields for correlating logs with trace spans if available. You can look at additional examples for standard logging configuration in other languages.

Tracing in ADK

Shifting focus to tracing, ADK provides robust, built-in capabilities. The tracing generation is implemented using OpenTelemetry, an open-source standard widely recognized for supporting numerous languages and ingestion methods. Tracing in ADK follows the same principles as logging. It uses standard for spanning traces, does not limiting developers with ingestion methods and allow clear troubleshooting of hierarchical trace spans. For local development and debugging, you can leverage ADK’s built-in tracing with the convenient Trace View feature of the ADK Web UI. When you deploy to Agent Engine or Cloud Run, you can enable tracing ingestion to Cloud Trace with a single argument.

Agent Engine Deployment - from ADK CLI

adk deploy agent_engine \
    --project=$GOOGLE_CLOUD_PROJECT \
    --region=$GOOGLE_CLOUD_LOCATION \
    --staging_bucket=$STAGING_BUCKET \
    --trace_to_cloud \
    $AGENT_PATH

The same argument is used when you use ADK CLI to deploy to Cloud Run.

Agent Engine Deployment - from Python SDK

from vertexai.preview import reasoning_engines
...
adk_app = reasoning_engines.AdkApp(
    agent=root_agent,
    enable_tracing=True,
)
...

Cloud Run Deployment - from Python SDK

import os
from google.adk.cli.fast_api import get_fast_api_app
from fastapi import FastAPI
...
AGENT_DIR = os.path.dirname(os.path.abspath(__file__))

app: FastAPI = get_fast_api_app(
    agents_dir=AGENT_DIR,
    web=True,
    trace_to_cloud=True,
)
...

If you plan to host the application on a different cloud platform or ingest your logs to another destination than Cloud Trace, you can customize OpenTelemetry configuration using environment variables or configuring trace provider and exporter instances in code. See this ADK example for configuring a custom trace exporter with ADK Runner instance or OpenTelemetry documentation about OTel exporters. You can find other examples for enabling and customizing tracing in ADK documentation.

Monitoring metrics

ADK does not generate metrics. In my opinion some metrics such as the size of the session or context might come in handy. However, my survey of developer community forums showed that there is no common ground about what metrics are useful for Agentic workflow. There is an agreement about LLM-based metric (e.g. number of tokens for LLM inference) which is out of scope of ADK. Developers can use ADK tracing capabilities to explore performance (timing) of the agents and tools of the agentic application.

Wrapping up

While manual observability instrumentation is always an option, it invariably comes with development and maintenance overhead. The ADK framework significantly reduces this burden by providing built-in observability capabilities. ADK is specifically designed to offer developers granular control over log and trace formatting and ingestion. This allows for seamless integration of ADK’s observability data into your existing workflows, providing a comprehensive view of your application’s performance with minimal additional code. Crucially, when deploying your agentic application to Google Cloud, you benefit from out-of-the-box tracing and logging ingestion and correlation, further simplifying your observability strategy.