AI Agent Observability: Multi-Cloud Monitoring with AgentCore and OpenTelemetry

AI agents now run on AWS, on-premises, in Kubernetes, and with other cloud providers. That makes monitoring harder. Amazon Bedrock AgentCore Observability offers tracing and analysis tools. But by default, it only supports agents running on the AgentCore runtime in AWS. The AWS blog post shows a solution: use the AWS Distro for OpenTelemetry (ADOT) to connect external agents to AgentCore’s dashboard. We analyze that solution, explain the technical parts, and provide example prompts.

Overview

The architecture forwards telemetry data from AI agents running outside AWS – e.g., on GCP, Azure, or on-premises – to the AgentCore Observability dashboard. ADOT is integrated into the agent’s Python process. It automatically instruments the agent framework (in the example Strands Agents) and exports spans, metrics, and logs via the OpenTelemetry exporter to the CloudWatch OTLP endpoint. Authentication uses IAM with SigV4 signing.

The data flow has three components: ADOT auto-instrumentation, IAM credentials, and environment variables. The environment variables control routing and authentication. Without them, telemetry data ends up in general CloudWatch logs instead of the dedicated Generative AI Observability dashboard.

Prompt Analysis

The article doesn’t contain classic prompts. It offers technical configuration steps and code examples. For practical implementation, we present two example prompts: one for configuring the observability environment, one for querying agent metrics.

The Prompt

# Prompt 1: Configuration of Observability for an External AI Agent
"You are a DevOps expert for AWS infrastructure. Explain step by step how I can connect an AI agent running with Strands Agents on a GCP server to Amazon Bedrock AgentCore Observability. Cover the installation of the packages 'aws-opentelemetry-distro' and 'strands-agents[otel]', explain the necessary IAM permissions and environment variables (AGENT_OBSERVABILITY_ENABLED, OTEL_PYTHON_DISTRO, OTEL_RESOURCE_ATTRIBUTES, OTEL_EXPORTER_OTLP_LOGS_HEADERS), and show how to start the agent with 'opentelemetry-instrument'."

Components

Role/Persona: The prompt assigns the model a DevOps expert role. This keeps the answer technical and professional.

Context: The context mentions GCP server and Strands Agents. This prevents generic answers and forces a solution tailored to the scenario.

Task: The task is explicit: step-by-step explanation of the integration. Mentioning packages, IAM permissions, and environment variables structures the answer and covers relevant aspects.

Output format: A step-by-step guide is expected. That means numbered, clearly separated presentation with commands and explanations.

Constraints: The specific variables and packages limit the solution space. The model must focus on the approach described in the AWS article and not suggest alternative tools.

# Prompt 2: Query and Interpretation of Observability Data
"You are an AI operations analyst. I have set up observability for my agent 'gcp-hosted-agent' and see multiple traces in the CloudWatch dashboard. Describe which spans typically appear in a successful agent trace (e.g., 'invoke_agent', 'chat', 'execute_event_loop_cycle', 'chat.us.anthropic.claude-haiku') and how to recognize behavioral patterns of the agent using latency and token metrics. Summarize which metrics may indicate hallucinations or inefficient tool calls."

Components

Role/Persona: The role of the AI operations analyst emphasizes the ability to interpret telemetry data and derive business relevance.

Context: The user refers to specific spans from the AWS article, so the answer must address real observations.

Task: Two subtasks: describing typical spans and deriving behavioral patterns from metrics. The prompt requires an analytical, not purely descriptive, answer.

Output format: The answer should summarize – a textual explanation with examples, not just a list.

Constraints: The focus is exclusively on the mentioned spans and metrics. It also links to quality issues like hallucinations, steering the answer toward AI system responsibility.

Frequently Asked Questions

Which environment variables are essential for connecting an external agent to AgentCore Observability?

Needed are AGENT_OBSERVABILITY_ENABLED=true, OTEL_PYTHON_DISTRO=aws_distro, OTEL_PYTHON_CONFIGURATOR=aws_configurator, OTEL_RESOURCE_ATTRIBUTES (with aws.log.group.names) and OTEL_EXPORTER_OTLP_LOGS_HEADERS (with x-aws-metric-namespace=bedrock-agentcore). These control routing and authentication.

Which IAM permissions are required?

The IAM user needs permissions for bedrock:InvokeModel, logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents, xray:PutTraceSegments, xray:PutTelemetryRecords, xray:GetSamplingRules, xray:GetSamplingTargets and cloudwatch:PutMetricData.

Why doesn’t observability work without ADOT in non-AWS environments?

AgentCore Observability natively supports only agents running on the AgentCore runtime. For external environments, automatic instrumentation and SigV4 signing are missing. ADOT takes over these tasks and exports telemetry via the CloudWatch OTLP endpoint.

Can other agent frameworks like LangGraph or CrewAI also be connected?

Yes, the AWS article explicitly states that the ADOT-based pattern also applies to other OpenTelemetry-compatible frameworks. One just needs to ensure that the framework emits OTel spans.

How long does it take for telemetry data to appear in the dashboard?

Typically, data appears within two to three minutes after running the agent. This is the usual latency for processing and indexing in CloudWatch.

Is using IAM access keys recommended for production environments?

The article recommends IAM Roles Anywhere for production, which issues temporary credentials via X.509 certificates. Long-lived access keys should only be used in development environments.

Source

Based on this article.