AI groups constructing manufacturing brokers face a irritating asymmetry: the variety of agent frameworks retains rising, however analysis tooling has not saved tempo. Most analysis programs assume you constructed your agent in a particular approach: a particular SDK, a particular giant language mannequin (LLM) shopper, a particular tracing sample. The second you step outdoors that slender compatibility zone, the analysis pipeline breaks.<\/p>\n
Groups construct on LangGraph for its workflow orchestration mannequin, on LlamaIndex for its tight integration with retrieval pipelines, and on the OpenAI Brokers SDK when their group standardizes on GPT fashions. They use Google ADK for multi-agent coordination, or the Claude Agent SDK for native Anthropic functionality. They attain for Strands Brokers as a result of its model-driven loop will get a working agent working on Amazon Bedrock AgentCore in minutes quite than days. And more and more, they deploy all of those on Amazon Bedrock AgentCore runtime<\/a>, a functionality of Amazon Bedrock AgentCore. It handles the internet hosting, scaling, reminiscence, and observability infrastructure they’d in any other case rebuild for every mission.<\/p>\n Amazon Bedrock AgentCore evaluations<\/a>\u00a0solves this fragmentation by decoupling analysis from the framework alternative. Each main framework helps OpenTelemetry, both natively or by means of a group instrumentation library. So long as an agent\u2019s telemetry flows by means of OpenTelemetry, the analysis service can rating it, no matter what SDK sits beneath. This publish explains how that works: what telemetry the service reads, the way it decides how one can learn your spans, which attributes carry the analysis information, and the way protection extends to frameworks past the named record.<\/p>\n OpenTelemetry is a vendor-neutral instrumentation framework that standardizes how distributed programs emit traces, metrics, and logs. A hint is a tree of spans, and every span represents a single step inside a request: one unit of labor with its title, timestamps, a set of typed attributes, and optionally available span occasions. Spans are exported over the OpenTelemetry Protocol (OTLP) and picked up by a telemetry backend. On AgentCore runtime, that backend is the AWS Distro for OpenTelemetry (ADOT), which routes spans and occasion data to Amazon CloudWatch.<\/p>\n An agent\u2019s execution produces many sorts of span, as a result of an agent does many sorts of labor. A single consumer flip can generate spans for mannequin calls, device calls, doc retrieval from a vector retailer or database, reranking of retrieved outcomes, embedding era, guardrail checks, prompt-template rendering, reminiscence reads and writes, and the orchestration steps that tie them collectively. The 2 conventions title these explicitly: the OpenTelemetry GenAI conventions<\/a> outline operations corresponding to Out of that full set, the analysis service wants three span roles to reconstruct what occurred in a session and rating it:<\/p>\n These three are what the evaluators function on. The service classifies each span it receives, reads the values it wants from these three roles, and passes over the remainder. Richer traces that additionally carry retrieval, reranking, guardrail, or reminiscence spans are dealt with with none particular configuration. These spans merely add context the evaluators don\u2019t require. As frameworks and the conventions add new span sorts over time, this stays forward-compatible: an unfamiliar span type is context the service skips, not an error.<\/p>\n The service reads the three marked span roles and treats the others as extra context.<\/p>\n Completely different frameworks and instrumentation libraries document these three roles utilizing totally different attribute names, nesting constructions, and span-naming conventions. The OpenTelemetry GenAI semantic conventions and the OpenInference specification each outline schemas for recording these three span roles, however they use totally different attribute keys and totally different span-kind vocabularies. AgentCore Evaluations bridges each schemas to the identical outcome.<\/p>\n While you run an analysis, on demand or by means of a web-based analysis config, the service fetches your agent\u2019s spans and occasion data from CloudWatch and reconstructs the session. A session<\/strong> is grouped by The service classifies every span, extracts the values it wants, and fingers the reconstructed session to the evaluators. From that time on, analysis is solely framework-agnostic: the identical evaluators, GoalSuccessRate, Correctness, Helpfulness, and customized LLM-as-a-judge, rating each framework identically. That is proven visually within the following determine.<\/p>\n Determine 1: Knowledge circulation from AgentCore Runtime by means of Amazon CloudWatch to AgentCore Evaluations<\/p>\n<\/p><\/div>\n You don\u2019t configure any of this.<\/strong> Each OpenTelemetry instrumentation library stamps a Protection reaches past the named frameworks. Any library whose scope title falls beneath Getting a session evaluated finish to finish comes down to 2 necessities. The primary is grouping: your agent\u2019s spans should carry a The second requirement is that your information supply consists of the message content material, not solely the spans. For brokers with unified observability<\/a> (the default for newly created brokers), that is automated: message content material lives in the identical per-agent log group because the spans, so a single log group is adequate. For current brokers nonetheless working the pre-unified configuration, nonetheless, spans land within the shared The OpenTelemetry GenAI semantic conventions outline a schema for spans produced by LLM frameworks. AgentCore Evaluations reads a particular subset of those attributes to categorise spans and extract analysis information. Figuring out which attributes carry that information is beneficial when you find yourself debugging an analysis or writing customized instrumentation.<\/p>\n Span classification begins with When Device identification comes from Message content material lives in one among two locations, relying on how telemetry was collected. When telemetry is cut up, the content material is within the correlated occasion document physique ( OpenInference is an open specification maintained by Arize AI and extensively adopted throughout the LlamaIndex, Haystack, and Phoenix ecosystems. It represents LLM operations with totally different attribute names from the OpenTelemetry GenAI conventions. A number of frameworks, together with the OpenAI Brokers SDK with Span classification in OpenInference makes use of Inference-span message content material makes use of a flat, listed attribute conference. Enter messages comply with the sample Device schemas on inference spans use Execute device spans use three attributes: The named frameworks are coated straight, however the design is intentionally open-ended. For any framework outdoors that record, the 2 generic paths launched earlier present broad protection at any time when the instrumentation follows a documented conference:<\/p>\n The scope-name prefix is what routes a library to one among these paths. For groups writing customized instrumentation that must be evaluated generically, naming your scope beneath To make this concrete, the AgentCore samples repository<\/a> supplies full working examples, as an example an HR assistant carried out with the OpenAI Brokers SDK, Google ADK, LlamaIndex, and the Claude Agent SDK. Every deploys to AgentCore runtime and is evaluated with the identical built-in and customized evaluators.<\/p>\n Instrumentation setup requires no express code in both agent.<\/strong> On AgentCore runtime, AWS Distro for OpenTelemetry (ADOT) discovers the put in instrumentation bundle at startup and prompts it routinely. For the OpenAI Brokers SDK pattern, including For LlamaIndex particularly, agent construction issues.<\/strong> The agent have to be constructed as a To reliably ship telemetry to CloudWatch, flush on the finish of the invocation handler earlier than returning the response.<\/strong> AgentCore runtime suspends the execution atmosphere after the handler returns, whereas the OTel SDK buffers telemetry in client-side batch processors that export on a periodic timer. Because of this, unexported information should still be sitting within the buffers at return time, and there’s no assure that an export completes earlier than suspension. In apply, lacking telemetry from an omitted flush is the most typical supply of analysis failures. The OpenAI Brokers SDK and LlamaIndex samples implement this with the next sample:<\/p>\n The flush should cowl each the tracer supplier (spans) and the logger supplier (occasion data): spans and occasion data undergo separate client-side batch processors, so flushing solely one among them leaves the opposite\u2019s buffer untouched.<\/p>\n AgentCore Evaluations makes use of The GoalSuccessRate evaluator operates on the session degree and checks whether or not the agent\u2019s tool-call historical past matched the anticipated trajectory throughout all turns. Correctness and Helpfulness are trace-level evaluators that rating every particular person flip. Constructed-in evaluators<\/a> are configured for the session, hint, device ranges and out there to make use of as templates. For customized evaluators<\/a> that you simply configure, you outline if it really works on the session, hint or device degree. It’s attainable to mix evaluators at varied ranges right into a single analysis name. The analysis code for the totally different framework samples is structurally equivalent. The one variations are the agent names and useful resource IDs.<\/p>\n The pattern repository demonstrates each analysis modes, which serve distinct functions in a manufacturing agent workflow.<\/p>\n On-demand analysis<\/strong> is the pure match for steady integration and steady supply (CI\/CD) pipelines and regression testing. You invoke the agent to generate a session, then name On-line analysis<\/strong> screens reside agent visitors repeatedly. You create an On-line analysis can solely use evaluators that don’t require floor fact, since reside visitors has no related anticipated responses. The built-in evaluators work in each modes. Customized LLM-as-a-judge evaluators that reference On-line analysis outcomes land in a CloudWatch log group at AgentCore Evaluations is framework-agnostic. Whether or not you construct with LangGraph, the OpenAI Brokers SDK, LlamaIndex, Google ADK, the Claude Agent SDK, or Strands Brokers, the identical evaluators apply uniformly in on-demand and on-line modes, as a result of the mixing contract is constructed on OpenTelemetry requirements. In case your agent emits spans beneath a acknowledged scope title, consists of device schemas on inference spans, and surfaces message content material by means of span attributes or occasion data, the analysis service produces scores with none framework-specific code in your facet. For frameworks past the named record, following the OpenTelemetry GenAI conventions or the OpenInference specification with the fitting scope prefix is all it takes. With one analysis suite, throughout each agent that emits telemetry beneath a acknowledged scope, now you can assess high quality persistently.<\/p>\n Able to attempt it your self? Get began with the AgentCore Evaluations function samples<\/a>, and consult with the Supported agent frameworks<\/a> documentation for full particulars on connecting your framework of alternative.<\/p>\n Thanks to the AgentCore evaluations engineering crew led by Shoaib Javed and Qiaoxuan Xue, and the engineers who contributed to this work: Swarnim Singhal, Lefan Zhang, Athul Iddya, Irene Wang, Ritvika Pillai, Aditya Arepalli.<\/p>\nOpenTelemetry because the widespread language<\/h2>\n
chat<\/code>, embeddings<\/code>, retrieval<\/code>, execute_tool<\/code>, invoke_agent<\/code>, create_agent<\/code>, plan<\/code>, and a household of reminiscence operations. OpenInference<\/a> defines span sorts corresponding to LLM<\/code>, TOOL<\/code>, RETRIEVER<\/code>, RERANKER<\/code>, EMBEDDING<\/code>, AGENT<\/code>, CHAIN<\/code>, GUARDRAIL<\/code>, EVALUATOR<\/code>, and PROMPT<\/code>. A manufacturing hint routinely mixes a number of of those.<\/p>\n\n
Session (one runtimeSessionId)\n\u2514\u2500\u2500 Hint (one consumer flip, one trace_id)\n \u251c\u2500\u2500 invoke agent span \u2190 learn: consumer immediate + closing agent response\n \u251c\u2500\u2500 inference span \u2190 learn: messages to mannequin + mannequin reply\n \u251c\u2500\u2500 execute device span \u2190 learn: device title + parameters + outcome\n \u251c\u2500\u2500 retriever span (context; not required by evaluators)\n \u251c\u2500\u2500 inference span \u2190 learn: subsequent mannequin name with device lead to historical past\n \u2514\u2500\u2500 ... (guardrail, reminiscence, reranker, orchestration spans, and extra)<\/code><\/pre>\n<\/p><\/div>\nHow AgentCore Evaluations reads your traces<\/h2>\n
session.id<\/code>. Inside it, every hint<\/strong> (one trace_id<\/code>) is one consumer flip. Each flip is made up of the three span sorts described earlier.<\/p>\n![\"AgentCore](\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2026\/08\/24\/ML-21721-1.png\")
<\/p>\nscope.title<\/code> on the spans and occasion data it produces, and the analysis service makes use of that worth to resolve how one can learn them. The proper dealing with prompts routinely from the instrumentation bundle you put in, with no adjustments to your agent code. Every supported framework and its scope title is listed within the Supported agent frameworks documentation, which right now covers Strands Brokers, LangGraph, the OpenAI Brokers SDK, LlamaIndex, Google ADK, and the Claude Agent SDK, most with each OpenTelemetry and OpenInference instrumentation.<\/p>\nopentelemetry.instrumentation.*<\/code> (following the OpenTelemetry GenAI semantic conventions) or openinference.instrumentation.*<\/code> (following the OpenInference specification) is learn by means of a generic path. In apply, supporting a brand new framework is normally solely a matter of putting in a compliant instrumentation bundle: the scope-name prefix is how a library opts in. A scope named mycompany.agent.tracing<\/code> won’t be picked up, even when its spans comply with the conventions precisely. The prefix is the sign that the instrumentation writer intentionally conformed to a documented schema.<\/p>\nsession.id<\/code> attribute that matches the runtimeSessionId<\/code> you invoked the agent with. That attribute is what lets the service assemble spans into traces and traces right into a session. On AgentCore runtime, ADOT injects this attribute routinely, so no agent code adjustments are wanted.<\/p>\naws\/spans<\/code> log group. Message content material is saved individually as correlated occasion data within the agent\u2019s log group. In that configuration, in case your information supply covers solely aws\/spans<\/code>, span classification nonetheless succeeds, however message content material comes again empty, and any evaluator that scores response high quality returns an error.<\/p>\nOpenTelemetry GenAI semantic conventions: Attributes that drive analysis<\/h2>\n
gen_ai.operation.title<\/code>:<\/p>\n\n
invoke_agent<\/code> marks the invoke agent span, the top-level span per flip, carrying the consumer immediate and closing response.<\/li>\nchat<\/code> marks an inference span. The service reads the message historical past and the mannequin\u2019s reply.<\/li>\nexecute_tool<\/code> marks an execute device span. The service reads the device title, enter parameters, and outcome.<\/li>\n<\/ul>\ngen_ai.operation.title<\/code> is absent, widespread in some LlamaIndex traces, the service falls again to traceloop.span.type<\/code>, the place workflow<\/code> maps to the invoke agent span, device<\/code> to an execute device span, and llm<\/code> to an inference span.<\/p>\ngen_ai.device.title<\/code>, and gen_ai.device.name.id<\/code> is the correlation ID that hyperlinks a device name requested on an inference span to its outcome on the corresponding execute device span. The gen_ai.device.definitions<\/code> attribute on inference spans carries a JSON-encoded array of device schemas, which the GoalSuccessRate evaluator makes use of to confirm that the instruments the agent known as had been amongst these it declared out there.<\/p>\nphysique.enter.messages<\/code> and physique.output.messages<\/code>). When it’s not cut up, the content material stays on the span, as gen_ai.enter.messages<\/code> or gen_ai.output.messages<\/code> attributes or as inline span occasions. The service reads from whichever location holds the content material, so that you don\u2019t have to know which path a given deployment took. For the complete image of the place content material lives, see Spans, occasion data, and telemetry indicators.<\/p>\nOpenInference conventions: An alternate semantic layer<\/h2>\n
openinference-instrumentation-openai-agents<\/code>, Google ADK, and the Claude Agent SDK, produce OpenInference spans.<\/p>\nopeninference.span.type<\/code>. The values LLM<\/code>, TOOL<\/code>, AGENT<\/code>, and CHAIN<\/code> correspond to inference, execute device, invoke agent, and structural container spans respectively.<\/p>\nllm.input_messages.{i}.message.position<\/code> and llm.input_messages.{i}.message.content material<\/code>, the place i<\/code> begins at zero and increments for every message within the historical past. Device outcome messages carry a further llm.input_messages.{i}.message.tool_call_id<\/code> to hyperlink them to the corresponding device name. Output messages comply with the identical listed sample, with device calls nested additional:<\/p>\nllm.output_messages.0.message.position = \"assistant\"\nllm.output_messages.0.message.tool_calls.0.tool_call.perform.title = \"get_pto_balance\"\nllm.output_messages.0.message.tool_calls.0.tool_call.perform.arguments = \"{\"employee_id\":\"EMP-001\"}\"\nllm.output_messages.0.message.tool_calls.0.tool_call.id = \"call_abc123\"<\/code><\/pre>\n<\/p><\/div>\nllm.instruments.{i}.device.json_schema<\/code>, the place every worth is a JSON string encoding both a plain perform schema or an OpenAI-style {\"kind\": \"perform\", \"perform\": {...}}<\/code> wrapper. The service handles each and reads the title<\/code>, description<\/code>, and parameters<\/code> fields.<\/p>\ndevice.title<\/code> for the ID, enter.worth<\/code> for the parameters (a JSON-encoded object for multi-argument instruments, or a plain string for single-argument instruments), and output.worth<\/code> for the outcome. The output.worth<\/code> format has developed throughout variations of the OpenInference instrumentation library, and the service reads each the older and newer shapes, so you aren’t tied to a particular model.<\/p>\nEvaluating any compliant framework<\/h2>\n
\n
openinference.instrumentation.*<\/code>. It classifies spans with openinference.span.type<\/code>, reads inference content material from the listed llm.input_messages<\/code> and llm.output_messages<\/code> attributes, and reads device.title<\/code>, enter.worth<\/code>, and output.worth<\/code> for device spans. This covers most frameworks that undertake the OpenInference specification, together with ones launched after a given framework was named explicitly.<\/li>\nopentelemetry.instrumentation.*<\/code>. It classifies spans with gen_ai.operation.title<\/code>, reads message content material from the occasion document physique or span, and makes use of gen_ai.device.title<\/code> for device identification.<\/li>\n<\/ul>\nopentelemetry.instrumentation.*<\/code> or openinference.instrumentation.*<\/code> is the best way to decide in.<\/p>\nFrom agent code to analysis rating: A walkthrough<\/h2>\n
opentelemetry-instrumentation-openai-agents<\/code> to necessities.txt<\/code> is adequate. For the LlamaIndex pattern, opentelemetry-instrumentation-llamaindex<\/code> does the identical job. The one factor the OpenAI Brokers SDK agent should keep away from is looking set_tracing_disabled(True)<\/code>: the instrumentation hooks into the SDK\u2019s personal tracing pipeline, so disabling SDK tracing silences the analysis spans.<\/p>\nFunctionAgent<\/code> or ReActAgent<\/code> (workflow brokers), not as a plain AgentExecutor<\/code>. Workflow brokers emit a top-level invoke agent span, which anchors the hint. With out it, there isn’t any top-level span to reconstruct the flip from, and the session can’t be assembled from inference spans alone.<\/p>\ndef _flush_telemetry():\n from opentelemetry import hint as _trace\n from opentelemetry._logs import get_logger_provider as _get_lp\n for supplier in (_trace.get_tracer_provider(), _get_lp()):\n flush = getattr(supplier, \"force_flush\", None)\n if flush:\n flush()\n\n@app.entrypoint\nasync def invoke(payload, context):\n immediate = payload.get(\"immediate\", \"\")\n attempt:\n outcome = await run_agent(immediate)\n lastly:\n _flush_telemetry()\n return str(outcome)<\/code><\/pre>\n<\/p><\/div>\nEvaluationClient<\/code> identically for each frameworks<\/strong>, after a CloudWatch ingestion wait of 90\u2013150 seconds:<\/p>\nfrom bedrock_agentcore.analysis import EvaluationClient\nfrom bedrock_agentcore.analysis.shopper import ReferenceInputs\nfrom datetime import timedelta\n\nec = EvaluationClient(region_name=REGION)\n# The shopper resolves every evaluator's degree (SESSION, TRACE, or TOOL_CALL)\n# routinely, so no degree configuration is required right here.\noutcomes = ec.run(\n evaluator_ids=[\n \"Builtin.GoalSuccessRate\",\n \"Builtin.Correctness\",\n \"Builtin.Helpfulness\",\n CUSTOM_RESPONSE_QUALITY_ID,\n CUSTOM_SESSION_COMPLETENESS_ID,\n ],\n agent_id=AGENT_ID,\n session_id=SESSION_ID,\n look_back_time=timedelta(hours=1),\n reference_inputs=ReferenceInputs(\n assertions=ASSERTIONS,\n expected_trajectory=EXPECTED_TRAJECTORY,\n expected_response=EXPECTED_RESPONSES[-1],\n ),\n)<\/code><\/pre>\n<\/p><\/div>\nOn-demand and on-line analysis modes<\/h2>\n
EvaluationClient.run()<\/code> in the identical script to attain it. Since you management the invocation, you may provide floor fact by means of ReferenceInputs<\/code>: anticipated responses, anticipated device trajectories, and behavioral assertions. A pipeline that runs on each pull request can invoke a hard and fast set of take a look at prompts, rating them towards floor fact, and fail the construct if any evaluator drops under threshold.<\/p>\nOnlineEvaluationConfig<\/code> as soon as, pointing it on the agent\u2019s CloudWatch log group and specifying a sampling fee. The service picks up new classes routinely as they arrive:<\/p>\n_cp.create_online_evaluation_config(\n onlineEvaluationConfigName=\"hr_agent_online_eval\",\n rule={\"samplingConfig\": {\"samplingPercentage\": 25.0}},\n dataSourceConfig={\n \"cloudWatchLogs\": {\n \"logGroupNames\": [CW_LOG_GROUP],\n \"serviceNames\": [OTEL_SERVICE_NAME],\n }\n },\n evaluators=[\n {\"evaluatorId\": \"Builtin.GoalSuccessRate\"},\n {\"evaluatorId\": \"Builtin.Correctness\"},\n {\"evaluatorId\": \"Builtin.Helpfulness\"},\n ],\n evaluationExecutionRoleArn=ONLINE_EVAL_ROLE_ARN,\n enableOnCreate=True,\n)<\/code><\/pre>\n<\/p><\/div>\n{expected_response}<\/code> or {assertions}<\/code> placeholders of their directions are on-demand solely.<\/p>\n\/aws\/bedrock-agentcore\/evaluations\/outcomes\/{config_id}<\/code>. Constructing a CloudWatch alarm or dashboard from that information provides you steady visibility into agent high quality in manufacturing, throughout all frameworks, by means of a single constant interface. As a result of the identical analysis pipeline processes on-demand and on-line visitors identically, scores from a CI take a look at run and scores from manufacturing visitors are straight comparable when the agent\u2019s habits is steady.<\/p>\nConclusion<\/h2>\n
Acknowledgements<\/h3>\n
Concerning the authors<\/h2>\n