Consider any agent framework with Amazon Bedrock AgentCore Evaluations

AI groups constructing manufacturing brokers face a irritating asymmetry: the variety of agent frameworks retains rising, however analysis tooling has not saved tempo. Most analysis programs assume you constructed your agent in a particular approach: a particular SDK, a particular giant language mannequin (LLM) shopper, a particular tracing sample. The second you step outdoors that slender compatibility zone, the analysis pipeline breaks.

Groups construct on LangGraph for its workflow orchestration mannequin, on LlamaIndex for its tight integration with retrieval pipelines, and on the OpenAI Brokers SDK when their group standardizes on GPT fashions. They use Google ADK for multi-agent coordination, or the Claude Agent SDK for native Anthropic functionality. They attain for Strands Brokers as a result of its model-driven loop will get a working agent working on Amazon Bedrock AgentCore in minutes quite than days. And more and more, they deploy all of those on Amazon Bedrock AgentCore runtime, a functionality of Amazon Bedrock AgentCore. It handles the internet hosting, scaling, reminiscence, and observability infrastructure they’d in any other case rebuild for every mission.

Amazon Bedrock AgentCore evaluations solves this fragmentation by decoupling analysis from the framework alternative. Each main framework helps OpenTelemetry, both natively or by means of a group instrumentation library. So long as an agent’s telemetry flows by means of OpenTelemetry, the analysis service can rating it, no matter what SDK sits beneath. This publish explains how that works: what telemetry the service reads, the way it decides how one can learn your spans, which attributes carry the analysis information, and the way protection extends to frameworks past the named record.

OpenTelemetry because the widespread language

OpenTelemetry is a vendor-neutral instrumentation framework that standardizes how distributed programs emit traces, metrics, and logs. A hint is a tree of spans, and every span represents a single step inside a request: one unit of labor with its title, timestamps, a set of typed attributes, and optionally available span occasions. Spans are exported over the OpenTelemetry Protocol (OTLP) and picked up by a telemetry backend. On AgentCore runtime, that backend is the AWS Distro for OpenTelemetry (ADOT), which routes spans and occasion data to Amazon CloudWatch.

An agent’s execution produces many sorts of span, as a result of an agent does many sorts of labor. A single consumer flip can generate spans for mannequin calls, device calls, doc retrieval from a vector retailer or database, reranking of retrieved outcomes, embedding era, guardrail checks, prompt-template rendering, reminiscence reads and writes, and the orchestration steps that tie them collectively. The 2 conventions title these explicitly: the OpenTelemetry GenAI conventions outline operations corresponding to chat, embeddings, retrieval, execute_tool, invoke_agent, create_agent, plan, and a household of reminiscence operations. OpenInference defines span sorts corresponding to LLM, TOOL, RETRIEVER, RERANKER, EMBEDDING, AGENT, CHAIN, GUARDRAIL, EVALUATOR, and PROMPT. A manufacturing hint routinely mixes a number of of those.

Out of that full set, the analysis service wants three span roles to reconstruct what occurred in a session and rating it:

• An invoke agent span represents the top-level request-response cycle, which is one consumer flip in a dialog. It carries the consumer immediate and the ultimate agent response.
• Inference spans symbolize particular person mannequin calls, every carrying the message historical past handed to the mannequin and the mannequin’s reply.
• Execute device spans symbolize every device the agent known as, carrying the device title, enter parameters, and outcome.

These three are what the evaluators function on. The service classifies each span it receives, reads the values it wants from these three roles, and passes over the remainder. Richer traces that additionally carry retrieval, reranking, guardrail, or reminiscence spans are dealt with with none particular configuration. These spans merely add context the evaluators don’t require. As frameworks and the conventions add new span sorts over time, this stays forward-compatible: an unfamiliar span type is context the service skips, not an error.

Session (one runtimeSessionId)
└── Hint (one consumer flip, one trace_id)
    ├── invoke agent span ← learn: consumer immediate + closing agent response
    ├── inference span ← learn: messages to mannequin + mannequin reply
    ├── execute device span ← learn: device title + parameters + outcome
    ├── retriever span (context; not required by evaluators)
    ├── inference span ← learn: subsequent mannequin name with device lead to historical past
    └── ... (guardrail, reminiscence, reranker, orchestration spans, and extra)

The service reads the three marked span roles and treats the others as extra context.

Completely different frameworks and instrumentation libraries document these three roles utilizing totally different attribute names, nesting constructions, and span-naming conventions. The OpenTelemetry GenAI semantic conventions and the OpenInference specification each outline schemas for recording these three span roles, however they use totally different attribute keys and totally different span-kind vocabularies. AgentCore Evaluations bridges each schemas to the identical outcome.

How AgentCore Evaluations reads your traces

While you run an analysis, on demand or by means of a web-based analysis config, the service fetches your agent’s spans and occasion data from CloudWatch and reconstructs the session. A session is grouped by session.id. Inside it, every hint (one trace_id) is one consumer flip. Each flip is made up of the three span sorts described earlier.

The service classifies every span, extracts the values it wants, and fingers the reconstructed session to the evaluators. From that time on, analysis is solely framework-agnostic: the identical evaluators, GoalSuccessRate, Correctness, Helpfulness, and customized LLM-as-a-judge, rating each framework identically. That is proven visually within the following determine.

Determine 1: Knowledge circulation from AgentCore Runtime by means of Amazon CloudWatch to AgentCore Evaluations

You don’t configure any of this. Each OpenTelemetry instrumentation library stamps a scope.title on the spans and occasion data it produces, and the analysis service makes use of that worth to resolve how one can learn them. The proper dealing with prompts routinely from the instrumentation bundle you put in, with no adjustments to your agent code. Every supported framework and its scope title is listed within the Supported agent frameworks documentation, which right now covers Strands Brokers, LangGraph, the OpenAI Brokers SDK, LlamaIndex, Google ADK, and the Claude Agent SDK, most with each OpenTelemetry and OpenInference instrumentation.

Protection reaches past the named frameworks. Any library whose scope title falls beneath opentelemetry.instrumentation.* (following the OpenTelemetry GenAI semantic conventions) or openinference.instrumentation.* (following the OpenInference specification) is learn by means of a generic path. In apply, supporting a brand new framework is normally solely a matter of putting in a compliant instrumentation bundle: the scope-name prefix is how a library opts in. A scope named mycompany.agent.tracing won’t be picked up, even when its spans comply with the conventions precisely. The prefix is the sign that the instrumentation writer intentionally conformed to a documented schema.

Getting a session evaluated finish to finish comes down to 2 necessities. The primary is grouping: your agent’s spans should carry a session.id attribute that matches the runtimeSessionId you invoked the agent with. That attribute is what lets the service assemble spans into traces and traces right into a session. On AgentCore runtime, ADOT injects this attribute routinely, so no agent code adjustments are wanted.

The second requirement is that your information supply consists of the message content material, not solely the spans. For brokers with unified observability (the default for newly created brokers), that is automated: message content material lives in the identical per-agent log group because the spans, so a single log group is adequate. For current brokers nonetheless working the pre-unified configuration, nonetheless, spans land within the shared aws/spans log group. Message content material is saved individually as correlated occasion data within the agent’s log group. In that configuration, in case your information supply covers solely aws/spans, span classification nonetheless succeeds, however message content material comes again empty, and any evaluator that scores response high quality returns an error.

OpenTelemetry GenAI semantic conventions: Attributes that drive analysis

The OpenTelemetry GenAI semantic conventions outline a schema for spans produced by LLM frameworks. AgentCore Evaluations reads a particular subset of those attributes to categorise spans and extract analysis information. Figuring out which attributes carry that information is beneficial when you find yourself debugging an analysis or writing customized instrumentation.

Span classification begins with gen_ai.operation.title:

• invoke_agent marks the invoke agent span, the top-level span per flip, carrying the consumer immediate and closing response.
• chat marks an inference span. The service reads the message historical past and the mannequin’s reply.
• execute_tool marks an execute device span. The service reads the device title, enter parameters, and outcome.

When gen_ai.operation.title is absent, widespread in some LlamaIndex traces, the service falls again to traceloop.span.type, the place workflow maps to the invoke agent span, device to an execute device span, and llm to an inference span.

Device identification comes from gen_ai.device.title, and gen_ai.device.name.id is the correlation ID that hyperlinks a device name requested on an inference span to its outcome on the corresponding execute device span. The gen_ai.device.definitions attribute on inference spans carries a JSON-encoded array of device schemas, which the GoalSuccessRate evaluator makes use of to confirm that the instruments the agent known as had been amongst these it declared out there.

Message content material lives in one among two locations, relying on how telemetry was collected. When telemetry is cut up, the content material is within the correlated occasion document physique (physique.enter.messages and physique.output.messages). When it’s not cut up, the content material stays on the span, as gen_ai.enter.messages or gen_ai.output.messages attributes or as inline span occasions. The service reads from whichever location holds the content material, so that you don’t have to know which path a given deployment took. For the complete image of the place content material lives, see Spans, occasion data, and telemetry indicators.

OpenInference conventions: An alternate semantic layer

OpenInference is an open specification maintained by Arize AI and extensively adopted throughout the LlamaIndex, Haystack, and Phoenix ecosystems. It represents LLM operations with totally different attribute names from the OpenTelemetry GenAI conventions. A number of frameworks, together with the OpenAI Brokers SDK with openinference-instrumentation-openai-agents, Google ADK, and the Claude Agent SDK, produce OpenInference spans.

Span classification in OpenInference makes use of openinference.span.type. The values LLM, TOOL, AGENT, and CHAIN correspond to inference, execute device, invoke agent, and structural container spans respectively.

Inference-span message content material makes use of a flat, listed attribute conference. Enter messages comply with the sample llm.input_messages.{i}.message.position and llm.input_messages.{i}.message.content material, the place i begins at zero and increments for every message within the historical past. Device outcome messages carry a further llm.input_messages.{i}.message.tool_call_id to hyperlink them to the corresponding device name. Output messages comply with the identical listed sample, with device calls nested additional:

llm.output_messages.0.message.position = "assistant"
llm.output_messages.0.message.tool_calls.0.tool_call.perform.title = "get_pto_balance"
llm.output_messages.0.message.tool_calls.0.tool_call.perform.arguments = "{"employee_id":"EMP-001"}"
llm.output_messages.0.message.tool_calls.0.tool_call.id = "call_abc123"

Device schemas on inference spans use llm.instruments.{i}.device.json_schema, the place every worth is a JSON string encoding both a plain perform schema or an OpenAI-style {"kind": "perform", "perform": {...}} wrapper. The service handles each and reads the title, description, and parameters fields.

Execute device spans use three attributes: device.title for the ID, enter.worth for the parameters (a JSON-encoded object for multi-argument instruments, or a plain string for single-argument instruments), and output.worth for the outcome. The output.worth format has developed throughout variations of the OpenInference instrumentation library, and the service reads each the older and newer shapes, so you aren’t tied to a particular model.

Evaluating any compliant framework

The named frameworks are coated straight, however the design is intentionally open-ended. For any framework outdoors that record, the 2 generic paths launched earlier present broad protection at any time when the instrumentation follows a documented conference:

• The OpenInference path handles any scope beneath openinference.instrumentation.*. It classifies spans with openinference.span.type, reads inference content material from the listed llm.input_messages and llm.output_messages attributes, and reads device.title, enter.worth, and output.worth for device spans. This covers most frameworks that undertake the OpenInference specification, together with ones launched after a given framework was named explicitly.
• The OpenTelemetry path handles any scope beneath opentelemetry.instrumentation.*. It classifies spans with gen_ai.operation.title, reads message content material from the occasion document physique or span, and makes use of gen_ai.device.title for device identification.

The scope-name prefix is what routes a library to one among these paths. For groups writing customized instrumentation that must be evaluated generically, naming your scope beneath opentelemetry.instrumentation.* or openinference.instrumentation.* is the best way to decide in.

From agent code to analysis rating: A walkthrough

To make this concrete, the AgentCore samples repository supplies full working examples, as an example an HR assistant carried out with the OpenAI Brokers SDK, Google ADK, LlamaIndex, and the Claude Agent SDK. Every deploys to AgentCore runtime and is evaluated with the identical built-in and customized evaluators.

Instrumentation setup requires no express code in both agent. On AgentCore runtime, AWS Distro for OpenTelemetry (ADOT) discovers the put in instrumentation bundle at startup and prompts it routinely. For the OpenAI Brokers SDK pattern, including opentelemetry-instrumentation-openai-agents to necessities.txt is adequate. For the LlamaIndex pattern, opentelemetry-instrumentation-llamaindex does the identical job. The one factor the OpenAI Brokers SDK agent should keep away from is looking set_tracing_disabled(True): the instrumentation hooks into the SDK’s personal tracing pipeline, so disabling SDK tracing silences the analysis spans.

For LlamaIndex particularly, agent construction issues. The agent have to be constructed as a FunctionAgent or ReActAgent (workflow brokers), not as a plain AgentExecutor. Workflow brokers emit a top-level invoke agent span, which anchors the hint. With out it, there isn’t any top-level span to reconstruct the flip from, and the session can’t be assembled from inference spans alone.

To reliably ship telemetry to CloudWatch, flush on the finish of the invocation handler earlier than returning the response. AgentCore runtime suspends the execution atmosphere after the handler returns, whereas the OTel SDK buffers telemetry in client-side batch processors that export on a periodic timer. Because of this, unexported information should still be sitting within the buffers at return time, and there’s no assure that an export completes earlier than suspension. In apply, lacking telemetry from an omitted flush is the most typical supply of analysis failures. The OpenAI Brokers SDK and LlamaIndex samples implement this with the next sample:

def _flush_telemetry():
    from opentelemetry import hint as _trace
    from opentelemetry._logs import get_logger_provider as _get_lp
    for supplier in (_trace.get_tracer_provider(), _get_lp()):
        flush = getattr(supplier, "force_flush", None)
        if flush:
            flush()

@app.entrypoint
async def invoke(payload, context):
    immediate = payload.get("immediate", "")
    attempt:
        outcome = await run_agent(immediate)
    lastly:
        _flush_telemetry()
    return str(outcome)

The flush should cowl each the tracer supplier (spans) and the logger supplier (occasion data): spans and occasion data undergo separate client-side batch processors, so flushing solely one among them leaves the opposite’s buffer untouched.

AgentCore Evaluations makes use of EvaluationClient identically for each frameworks, after a CloudWatch ingestion wait of 90–150 seconds:

from bedrock_agentcore.analysis import EvaluationClient
from bedrock_agentcore.analysis.shopper import ReferenceInputs
from datetime import timedelta

ec = EvaluationClient(region_name=REGION)
# The shopper resolves every evaluator's degree (SESSION, TRACE, or TOOL_CALL)
# routinely, so no degree configuration is required right here.
outcomes = ec.run(
    evaluator_ids=[
        "Builtin.GoalSuccessRate",
        "Builtin.Correctness",
        "Builtin.Helpfulness",
        CUSTOM_RESPONSE_QUALITY_ID,
        CUSTOM_SESSION_COMPLETENESS_ID,
    ],
    agent_id=AGENT_ID,
    session_id=SESSION_ID,
    look_back_time=timedelta(hours=1),
    reference_inputs=ReferenceInputs(
        assertions=ASSERTIONS,
        expected_trajectory=EXPECTED_TRAJECTORY,
        expected_response=EXPECTED_RESPONSES[-1],
    ),
)

The GoalSuccessRate evaluator operates on the session degree and checks whether or not the agent’s tool-call historical past matched the anticipated trajectory throughout all turns. Correctness and Helpfulness are trace-level evaluators that rating every particular person flip. Constructed-in evaluators are configured for the session, hint, device ranges and out there to make use of as templates. For customized evaluators that you simply configure, you outline if it really works on the session, hint or device degree. It’s attainable to mix evaluators at varied ranges right into a single analysis name. The analysis code for the totally different framework samples is structurally equivalent. The one variations are the agent names and useful resource IDs.

On-demand and on-line analysis modes

The pattern repository demonstrates each analysis modes, which serve distinct functions in a manufacturing agent workflow.

On-demand analysis is the pure match for steady integration and steady supply (CI/CD) pipelines and regression testing. You invoke the agent to generate a session, then name EvaluationClient.run() in the identical script to attain it. Since you management the invocation, you may provide floor fact by means of ReferenceInputs: anticipated responses, anticipated device trajectories, and behavioral assertions. A pipeline that runs on each pull request can invoke a hard and fast set of take a look at prompts, rating them towards floor fact, and fail the construct if any evaluator drops under threshold.

On-line analysis screens reside agent visitors repeatedly. You create an OnlineEvaluationConfig as soon as, pointing it on the agent’s CloudWatch log group and specifying a sampling fee. The service picks up new classes routinely as they arrive:

_cp.create_online_evaluation_config(
    onlineEvaluationConfigName="hr_agent_online_eval",
    rule={"samplingConfig": {"samplingPercentage": 25.0}},
    dataSourceConfig={
        "cloudWatchLogs": {
            "logGroupNames": [CW_LOG_GROUP],
            "serviceNames": [OTEL_SERVICE_NAME],
        }
    },
    evaluators=[
        {"evaluatorId": "Builtin.GoalSuccessRate"},
        {"evaluatorId": "Builtin.Correctness"},
        {"evaluatorId": "Builtin.Helpfulness"},
    ],
    evaluationExecutionRoleArn=ONLINE_EVAL_ROLE_ARN,
    enableOnCreate=True,
)

On-line analysis can solely use evaluators that don’t require floor fact, since reside visitors has no related anticipated responses. The built-in evaluators work in each modes. Customized LLM-as-a-judge evaluators that reference {expected_response} or {assertions} placeholders of their directions are on-demand solely.

On-line analysis outcomes land in a CloudWatch log group at /aws/bedrock-agentcore/evaluations/outcomes/{config_id}. Constructing a CloudWatch alarm or dashboard from that information provides you steady visibility into agent high quality in manufacturing, throughout all frameworks, by means of a single constant interface. As a result of the identical analysis pipeline processes on-demand and on-line visitors identically, scores from a CI take a look at run and scores from manufacturing visitors are straight comparable when the agent’s habits is steady.

Conclusion

AgentCore Evaluations is framework-agnostic. Whether or not you construct with LangGraph, the OpenAI Brokers SDK, LlamaIndex, Google ADK, the Claude Agent SDK, or Strands Brokers, the identical evaluators apply uniformly in on-demand and on-line modes, as a result of the mixing contract is constructed on OpenTelemetry requirements. In case your agent emits spans beneath a acknowledged scope title, consists of device schemas on inference spans, and surfaces message content material by means of span attributes or occasion data, the analysis service produces scores with none framework-specific code in your facet. For frameworks past the named record, following the OpenTelemetry GenAI conventions or the OpenInference specification with the fitting scope prefix is all it takes. With one analysis suite, throughout each agent that emits telemetry beneath a acknowledged scope, now you can assess high quality persistently.

Able to attempt it your self? Get began with the AgentCore Evaluations function samples, and consult with the Supported agent frameworks documentation for full particulars on connecting your framework of alternative.

Acknowledgements

Thanks to the AgentCore evaluations engineering crew led by Shoaib Javed and Qiaoxuan Xue, and the engineers who contributed to this work: Swarnim Singhal, Lefan Zhang, Athul Iddya, Irene Wang, Ritvika Pillai, Aditya Arepalli.

Concerning the authors