![\"BlogBanner_990x556_x2\"\/](\"https:\/\/storage.googleapis.com\/gweb-developer-goog-blog-assets\/images\/BlogBanner_990x556_x2.original.png\")
<\/p>\n
When builders first work on harness engineering for agentic coding programs, they typically fall into the identical lure: they run frequent end-to-end benchmarks like Terminal-Bench and DeepSWE, watch a composite rating transfer by a number of proportion factors, and do not know why it modified.<\/p>\n
Finish-to-end benchmarks are the de facto for evaluating mannequin efficiency and figuring out what wants deeper investigation, however the problem is that these investigations come at a excessive price.<\/p>\n
Behavioral evaluations are sometimes a greater measure of confidence<\/b> on whether or not the behaviors you anticipate really do occur and whether or not you\u2019re transferring in the fitting path as a substitute of backsliding on the subject of regressions or new mannequin adjustments. They’ll function your iteration companion and assist give perception into why sure adjustments transfer the needle in a technique or one other.<\/p>\n
Right here\u2019s our tackle behavioral analysis, together with approaches which have helped us maintain agent programs dependable as fashions evolve.<\/p>\n<\/div>\n
The paradigm shift: Report playing cards vs. behavioral guideposts<\/b><\/h2>\n
Most groups consider AI brokers like they’d consider a pupil taking an examination. They hand the agent a big codebase, give it a time restrict, and measure its success primarily based on what number of checks move or fail.<\/p>\n
When that rating drops, what went incorrect?<\/p>\n
- \n
- Did the mannequin get overconfident on ambiguous prompts?<\/li>\n
- Did it overlook to confirm the take a look at suite earlier than submitting?<\/li>\n
- Did it hallucinate a CLI flag?<\/li>\n<\/ul>\n
Finish-to-end benchmarks don\u2019t usually instantly reply these questions.<\/p>\n
Behavioral evaluations<\/b> operate like integration checks for enhancing agent harness operation. When you may have a wealthy sufficient behavioral eval set, you may have a baseline for the habits you are concentrating on out of your agent, and also you’re capable of iteratively enhance the immediate to get there.<\/p>\n
As a substitute of measuring whether or not the agent solved a complete multi-file refactor, a behavioral eval measures discrete, observable actions:<\/p>\n
- \n
- When given an underspecified immediate, does the agent ask a clarifying query as a substitute of guessing?<\/li>\n
- When modifying a construct file, does it run the native validator earlier than declaring it full?<\/li>\n
- When producing documentation, does it present canonical repository hyperlinks?<\/li>\n<\/ul>\n<\/div>\n\n\n
![\"Screenshot](\"https:\/\/storage.googleapis.com\/gweb-developer-goog-blog-assets\/images\/Screenshot_2026-09-09_at_9.34.40AM.original.png\")
<\/p><\/div><\/div>\n\nWhen to guage: The dogfooding precedent<\/b><\/h2>\n
As a substitute of organising a posh analysis harness on day one, use this time to comply with your hunches and run experiments.<\/p>\n
When bootstrapping an agent from scratch, you begin with developer intuition and dogfooding<\/b>. Till you may have constructed an agent able to dogfooding its personal codebase, dealing with boilerplate, writing its personal markdown renderer, and executing routine developer duties, it doesn\u2019t make sense to run evaluations.<\/p>\n
Evals belong to the second part of improvement: guaranteeing ahead progress<\/b> and guarding towards regressions<\/b>.<\/p>\n
The first goal of an analysis suite is to not have a good time whenever you make the agent 2% higher; it’s to present you unshakeable confidence {that a} new immediate tweak, instrument schema change, or mannequin improve didn’t make the agent holistically worse<\/b>.<\/p>\n<\/div>\n
\nHow a behavioral analysis structure works<\/b><\/h2>\n
A sturdy harness analysis framework separates behavioral assertions into quick, deterministic, unit-style checks that run domestically.<\/p>\n
Shifting your focus to those smaller, observable actions creates a dependable security web. You may confidently iterate in your system prompts or swap to a special mannequin, since you\u2019ll know instantly for those who’ve by chance damaged a core habits.<\/p>\n
Writing a behavioral eval<\/b><\/h3>\n
Behavioral evals assert on intermediate execution steps, like particular instrument calls or file modifications, as a substitute of ultimate string equality:<\/p>\n<\/div>\n
\nimport pytest \nfrom google.antigravity import Agent, LocalAgentConfig, varieties \n \n@pytest.mark.asyncio \nasync def test_agent_uses_web_search_for_live_weather(): \n \"\"\"Assert that the agent consults floor reality fairly than guessing.\"\"\" \n config = LocalAgentConfig() \n \n async with Agent(config) as agent: \n response = await agent.chat(\"What is the climate like in Mountain View, California?\") \n instruments = [call.name async for call in response.tool_calls] \n \n # Assert habits, not output prose \n assert varieties.BuiltinTools.SEARCH_WEB in instruments, ( \n \"Agent answered from reminiscence with out consulting dwell search.\" \n )<\/code><\/pre>\n\n Python\n <\/p>\n