{"id":16775,"date":"2026-07-16T11:24:23","date_gmt":"2026-07-16T11:24:23","guid":{"rendered":"https:\/\/techtrendfeed.com\/?p=16775"},"modified":"2026-07-16T11:24:24","modified_gmt":"2026-07-16T11:24:24","slug":"dont-let-claude-grade-its-personal-homework","status":"publish","type":"post","link":"https:\/\/techtrendfeed.com\/?p=16775","title":{"rendered":"Don\u2019t Let Claude Grade Its Personal Homework"},"content":{"rendered":"<p> <br \/>\n<\/p>\n<div wp_automatic_readability=\"356.7986934915\">\n<p class=\"wp-block-paragraph\"> I had a handful of parallel Claude brokers write me up a method folder for a facet challenge. The pull request appeared nice: ten paperwork, all neatly cross-linked, and it learn effectively. Then the automated reviewer posted its findings, and Claude, to its credit score, didn&#8217;t object:<\/p>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/07\/claude_admitting_ignorance-1024x234.jpeg\" alt=\"\" class=\"wp-image-672673\"\/><figcaption class=\"wp-element-caption\">Claude conceding, mid-session: \u201cThe automated assessment caught an actual bug: the doc writers used guessed filenames for cross-links. Legitimate discovering, I\u2019ll repair it.\u201d The docs it wrote linked to information it invented itself.<\/figcaption><\/figure>\n<p class=\"wp-block-paragraph\">The total audit got here again: 30 of the 95 relative hyperlinks pointed at information that didn&#8217;t exist. <code>.\/market.md<\/code> as an alternative of <code>market-analysis.md<\/code>, <code>.\/monetization.md<\/code> as an alternative of <code>business-model.md<\/code>, some information have been even made up totally! Skimming that PR as a human, I&#8217;d have accepted it, since each considered one of them appeared believable. Fortunately, my automated reviewer caught it as a result of it resolved each hyperlink as an alternative of judging whether or not the names \u201cvibed\u201d proper.<\/p>\n<p class=\"wp-block-paragraph\">To be honest, Claude doesn&#8217;t have any unhealthy intent behind this. A language mannequin merely generates probably the most believable continuation, and typically probably the most believable continuation is fiction delivered with full confidence. From the studying facet, although, that distinction brings little consolation. Assured and incorrect reads precisely like assured and proper.<\/p>\n<p class=\"wp-block-paragraph\">This put up is concerning the system I&#8217;ve in place to assist me assessment Claude\u2019s errors and hallucinations: a Codex agent working in GitHub Actions that evaluations each pull request. It walks you thru my setup, why cross-provider assessment beats each self-review and a <em>human-reads-everything<\/em> assessment, how one can implement reminiscence and accountability within the assessment loop, the associated fee and reliability learnings I gathered alongside the way in which, and the precise code I take advantage of to make this work.<\/p>\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-dotted\"\/>\n<h2 class=\"wp-block-heading\">Agent throughput broke the human assessment loop<\/h2>\n<p class=\"wp-block-paragraph\">Hallucinations by themselves are a manageable downside, since a really cautious assessment or purposeful evaluation would catch them. Nevertheless, what broke over the previous yr is the quantity that wants reviewing. A single engineer now runs a number of agent periods in parallel, and every of them can produce a thousand-line diff within the time it takes to refill your espresso. Producing code has develop into the quick and straightforward half; reviewing it truthfully is what eats the day. That makes the human reviewer the bottleneck of the entire supply pipeline, and the bottleneck is at all times the following factor it is best to automate.<\/p>\n<p class=\"wp-block-paragraph\">That bottleneck invitations two acquainted failure modes. Both you approve regardless of the brokers produce with out actually studying it, and the damaged hyperlinks, phantom parameters, and subtly incorrect edge circumstances ship to manufacturing. Otherwise you nonetheless insist on studying each line your self, and your <em>quick and sensible AI brokers<\/em> work even lower than you do. The primary is negligent, the second wasteful, and most groups oscillate between each relying on the deadline.<\/p>\n<p class=\"wp-block-paragraph\">The conclusion I hold touchdown on: our assessment course of has to scale with the velocity we\u2019re producing code. The human aspect stays, however the human reviewer\u2019s focus ought to go to the elements that really matter. The grind of checking whether or not hyperlinks resolve, whether or not the brand new parameter exists, whether or not there\u2019s a bug or vulnerability, and whether or not the code adjustments match what was initially requested, all of that has to run routinely on the similar cadence because the coding brokers, even earlier than an individual ever spends a second reviewing the code.<\/p>\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\" wp_automatic_readability=\"9\">\n<p class=\"wp-block-paragraph\"><strong>Perception:<\/strong> hallucinations are a relentless, throughput is what modified. Any course of that requires a human to learn each line caps your brokers at human studying velocity, so the very first assessment cross must be automated.<\/p>\n<\/blockquote>\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-dotted\"\/>\n<h2 class=\"wp-block-heading\">Decide a reviewer that doesn&#8217;t share the creator\u2019s blind spots<\/h2>\n<p class=\"wp-block-paragraph\">The apparent first thought to stop these errors is to have the agent assessment its personal work\u2026 however additionally it is the weakest one. A hallucination that made it into the diff is, by definition, believable to the mannequin that produced it. Asking the identical weights to seek out errors within the code, particularly throughout the similar already-biased coding session, normally reproduces the precise blind spots that allow the errors by.<\/p>\n<p class=\"wp-block-paragraph\">This instinct has analysis behind it. Panickssery et al. confirmed at NeurIPS 2024 that LLM evaluators <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/arxiv.org\/abs\/2404.13076\">acknowledge and favor their very own generations<\/a>: the higher a mannequin is at figuring out its personal output, the extra generously it scores it. Observe-up work ties this <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/arxiv.org\/abs\/2410.21819\">self-preference bias to familiarity<\/a>: AI judges rating textual content increased when it reads as predictable to them, and nothing reads as extra predictable to a mannequin than the textual content it simply wrote itself.<\/p>\n<p class=\"wp-block-paragraph\">Totally different LLM suppliers prepare on completely different knowledge, with completely different recipes and completely different suggestions loops. No supplier manages to get rid of errors, however the form of errors a mannequin makes traces again to the way it was skilled, and that differs per supplier. That decorrelation is strictly what you need in your assessment agent. Within the yr that I\u2019ve used this setup, OpenAI\u2019s Codex has persistently flagged points in Claude-written PRs that Claude\u2019s personal assessment handed with out blinking, from invented filenames to whole code blocks that had nothing to do with the preliminary request, or that launched outright safety violations.<\/p>\n<p class=\"wp-block-paragraph\">One conclusion you can&#8217;t take from that is that Codex is the higher engineer and it is best to hand it the keyboard as an alternative of Claude. Codex misfires pretty repeatedly, too! Simply in a unique route than Claude does. The facility sits in working multimodel: combining suppliers throughout the similar workflow, the place the fashions fill one another\u2019s gaps.<\/p>\n<p class=\"wp-block-paragraph\">One other incorrect conclusion is that Codex\u2019s evaluations are good and may be accepted blindly. Hallucinations occur in evaluations too, so deal with the assessment as advisory. Difficult the creator (be it Claude or your self) places the concentrate on potential blind spots, which is the right sign to zoom in and mirror. It stays the creator\u2019s job to personal the ultimate high quality, and typically which means writing again to Codex that its assessment was incorrect. Extra on this mechanism later within the put up.<\/p>\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\" wp_automatic_readability=\"10\">\n<p class=\"wp-block-paragraph\"><strong>Perception:<\/strong> an creator\u2019s bugs are by definition believable to the creator, and analysis confirms LLMs are positively biased towards their very own output. Evaluation worth comes from decorrelation, by utilizing a mannequin from a unique supplier that fails in another way, and completely different is what catches what you miss.<\/p>\n<\/blockquote>\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-dotted\"\/>\n<h2 class=\"wp-block-heading\">The pipeline: one LLM name wrapped in bizarre engineering<\/h2>\n<p class=\"wp-block-paragraph\">Conceptually, an LLM reviewer may be written in a single line: \u201cright here is the code diff, please assessment it\u201d. Each OpenAI and Anthropic ship hosted shortcuts for precisely that: OpenAI has a <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/developers.openai.com\/codex\/integrations\/github\">GitHub assessment integration<\/a> that auto-reviews PRs, and Anthropic has <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/github.com\/anthropics\/claude-code-action\">claude-code-action<\/a>. They&#8217;re high quality beginning factors, however I choose to run my very own motion as an alternative, as a result of I wish to personal the immediate, the assessment lifecycle, the merge gating, and the mannequin pin. All these elements matter when your day-to-day turns into managing and reviewing brokers.<\/p>\n<p class=\"wp-block-paragraph\">So what am I utilizing then, precisely? Two information that dwell within the repository they assessment:<\/p>\n<pre class=\"wp-block-prismatic-blocks\" datatext=\"\"><code class=\"language-markdown\">.github\/\n\u251c\u2500\u2500 workflows\/\n\u2502   \u2514\u2500\u2500 pr.yml            # orchestration: triggers, context, retries, posting\n\u2514\u2500\u2500 actions\/\n    \u2514\u2500\u2500 codex-pr-review\/\n        \u2514\u2500\u2500 motion.yml    # the reviewer: pinned Codex name + assessment immediate<\/code><\/pre>\n<p class=\"wp-block-paragraph\">The runtime circulate is brief. Each time a PR opens or receives a push, <code>pr.yml<\/code> checks out the code, collects the PR\u2019s full dialog historical past, and fingers each to the composite motion. The motion runs Codex over the diff and returns its findings as a single message. The workflow then posts that message as a single, constantly up to date remark within the PR dialog, and publishes a commit standing that stays purple whereas findings stay unresolved: the go\/no-go sign sitting subsequent to the remainder of CI.<\/p>\n<p class=\"wp-block-paragraph\">On the heart of all of it sits one deceptively easy step, the one line that truly touches an LLM:<\/p>\n<pre class=\"wp-block-prismatic-blocks\"><code class=\"language-yaml\">- makes use of: openai\/codex-action@v1\n  with:\n    openai-api-key: ${{ inputs.openai-api-key }}\n    immediate: |\n      # the assessment contract: what to assessment, how one can observe findings,\n      # and how one can report again (unpacked within the subsequent part)<\/code><\/pre>\n<p class=\"wp-block-paragraph\">The whole lot else exists to run that single step reliably, and to verify its suggestions truly lands on the PR. Right here is the entire workflow:<\/p>\n<pre class=\"wp-block-prismatic-blocks\" datatext=\"\"><code class=\"language-yaml\">identify: PR Reviewer\n\non:\n  pull_request:\n    branches: [dev] # all work rides characteristic branches: characteristic -&gt; dev -&gt; most important\n    varieties: [opened, synchronize, reopened]\n\n# a brand new push cancels the in-flight (costly) assessment of the earlier commit\nconcurrency:\n  group: pr-reviewer-${{ github.occasion.pull_request.quantity }}\n  cancel-in-progress: true\n\njobs:\n  codex:\n    if: github.occasion.pull_request.consumer.login != 'dependabot[bot]'\n    runs-on: ubuntu-latest\n    timeout-minutes: 15 # regular assessment takes a couple of minutes, so 15m is greater than sufficient\n    permissions: # the token can learn code and put up suggestions, however can by no means push or merge\n      contents: learn\n      points: write\n      pull-requests: write\n      statuses: write\n    steps:\n      # the \/head ref updates synchronously on each push;\n      # the auto-generated merge ref can race the checkout\n      - makes use of: actions\/checkout@v7\n        with:\n          ref: refs\/pull\/${{ github.occasion.pull_request.quantity }}\/head\n\n      # ensure each diff endpoints exist domestically, so Codex can resolve base...head\n      - identify: Pre-fetch base and head refs for the PR\n        run: |\n          git fetch --no-tags origin \n            ${{ github.occasion.pull_request.base.ref }} \n            +refs\/pull\/${{ github.occasion.pull_request.quantity }}\/head\n\n      # suggestions lives on three surfaces: challenge feedback, evaluations, and inline feedback\n      - identify: Fetch PR dialog\n        id: dialog\n        makes use of: actions\/github-script@v9\n        with:\n          result-encoding: string\n          script: |\n            const prNumber = context.payload.pull_request.quantity;\n            const opts = { proprietor: context.repo.proprietor, repo: context.repo.repo };\n            const [comments, reviews, inline] = await Promise.all([\n              github.paginate(github.rest.issues.listComments,\n                { ...opts, issue_number: prNumber }),\n              github.paginate(github.rest.pulls.listReviews,\n                { ...opts, pull_number: prNumber }),\n              github.paginate(github.rest.pulls.listReviewComments,\n                { ...opts, pull_number: prNumber }),\n            ]);\n            const all = [\n              ...comments.map(c =&gt; ({\n                date: c.created_at,\n                text: `@${c.user.login}: ${c.body}`,\n              })),\n              ...reviews.filter(r =&gt; r.body).map(r =&gt; ({\n                date: r.submitted_at,\n                text: `@${r.user.login} (review ${r.state}): ${r.body}`,\n              })),\n              ...inline.map(c =&gt; ({\n                date: c.created_at,\n                text: `@${c.user.login} (on ${c.path}): ${c.body}`,\n              })),\n            ];\n            all.type((a, b) =&gt; new Date(a.date) - new Date(b.date));\n            return all.map(e =&gt; e.textual content).be a part of('n---n');\n\n      # secrets and techniques don't journey with the yml, talk loudly on lacking keys\n      - identify: Fail quick on lacking OPENAI_API_KEY\n        env:\n          OPENAI_API_KEY: ${{ secrets and techniques.OPENAI_API_KEY }}\n        run: |\n          if [ -z \"${OPENAI_API_KEY}\" ]; then\n            echo \"::error::OPENAI_API_KEY secret will not be set for this repository.\"\n            exit 1\n          fi\n\n      # retry wrappers can't wrap `makes use of:` steps, so retry manually: transient\n      # failures (a 401 from a contemporary runner token, a hiccup reaching OpenAI)\n      # mustn't flip the entire run purple\n      - identify: Run Codex (try 1)\n        id: codex_try1\n        makes use of: .\/.github\/actions\/codex-pr-review\n        continue-on-error: true\n        with:\n          openai-api-key: ${{ secrets and techniques.OPENAI_API_KEY }}\n          dialog: ${{ steps.dialog.outputs.outcome }}\n\n      # give transient failures a second to clear earlier than retrying\n      - identify: Wait earlier than Codex retry\n        if: steps.codex_try1.consequence == 'failure'\n        run: sleep 20\n\n      - identify: Run Codex (try 2)\n        id: codex_try2\n        if: steps.codex_try1.consequence == 'failure'\n        makes use of: .\/.github\/actions\/codex-pr-review\n        with:\n          openai-api-key: ${{ secrets and techniques.OPENAI_API_KEY }}\n          dialog: ${{ steps.dialog.outputs.outcome }}\n\n      - identify: Put up the assessment and publish a standing\n        makes use of: actions\/github-script@v9\n        env:\n          REVIEW: &gt;-\n            ${}\n        with:\n          script: |\n            const MARKER = '';\n            const opts = { proprietor: context.repo.proprietor, repo: context.repo.repo };\n            const prNumber = context.payload.pull_request.quantity;\n            const uncooked = course of.env.REVIEW;\n            if (!uncooked) return;\n            const physique = `${MARKER}n${uncooked}`;\n\n            \/\/ one canonical remark, edited in place on each push\n            const feedback = await github.paginate(\n              github.relaxation.points.listComments, { ...opts, issue_number: prNumber });\n            const present = feedback.discover(c =&gt;\n              c.consumer?.login === 'github-actions[bot]' &amp;&amp; c.physique?.consists of(MARKER));\n            const posted = present\n              ? await github.relaxation.points.updateComment(\n                  { ...opts, comment_id: present.id, physique })\n              : await github.relaxation.points.createComment(\n                  { ...opts, issue_number: prNumber, physique });\n\n            \/\/ go\/no-go: a commit standing that stays purple whereas findings are open\n            \/\/ (a lacking trailer reads as \"assessment posted\", by no means as zero findings)\n            const m = uncooked.match(\/`, the place N counts\n              the unresolved rows and M what number of of these are severity Excessive.\n              At all times embody it, even when N is 0: CI parses it into the\n              standing test.\n\n          Pull request title and physique:\n          ----\n          ${{ github.occasion.pull_request.title }}\n          ${{ github.occasion.pull_request.physique }}\n\n          Dialog historical past:\n          ----\n          ${{ inputs.dialog }}<\/code><\/pre>\n<p class=\"wp-block-paragraph\">Three design selections carry the load right here.<\/p>\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\"><strong>The assessment is stateful.<\/strong> The Suggestions Abstract desk tracks each discovering ever raised on the PR, resolved or not, throughout pushes. Mixed with the edit-in-place remark from the workflow, a PR carries precisely one assessment artifact that displays your complete lifecycle, as an alternative of 5 stale feedback that every mirror an replace within the assessment course of. Anybody opening the PR, human or agent, sees the total historical past at a look.<\/li>\n<\/ul>\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\"><strong>Declines are honored.<\/strong> As a result of Codex reads the entire dialog, the creator can reject a discovering in writing, and the rejection sticks throughout assessment cycles. That is the mechanism that makes an imperfect reviewer workable: false positives value one written reply as an alternative of an infinite nag loop, and the objection stays on file for the following reader.<\/li>\n<\/ul>\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\"><strong>The decision is machine-readable.<\/strong> The trailing <code><\/code> line will get parsed by the workflow right into a <code>Codex assessment<\/code> commit standing. The result&#8217;s two unbiased indicators on each PR: the workflow test tells you the assessment ran, and the standing tells you whether or not something continues to be open. Crimson standing means unresolved findings; inexperienced means every little thing is mounted or explicitly declined. I intentionally hold the standing non-blocking, because it solely features to tell. The choice to merge or not stays open, and stays human.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\">Notice how this contract asks for greater than typo searching. Codex evaluations the diff in opposition to the PR\u2019s acknowledged intent, ranks each discovering by severity, and attaches a concrete proposed repair, the form of assessment that repeatedly surfaces architecture-level points no linter would ever produce.<\/p>\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\" wp_automatic_readability=\"9\">\n<p class=\"wp-block-paragraph\"><strong>Perception:<\/strong> construction beats uncooked mannequin high quality in a reviewer. A lifecycle desk, a decline rule, and a machine-readable verdict flip free-form LLM opinions right into a merge gate with a mechanism for written pushback that makes false positives low-cost.<\/p>\n<\/blockquote>\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-dotted\"\/>\n<h2 class=\"wp-block-heading\">Shut the loop: the PR turns inexperienced earlier than you even look<\/h2>\n<p class=\"wp-block-paragraph\">The reviewer is just half the story once you take a look at the larger engineering image. The opposite half is educating the creator agent to answer it, so the assessment round-trip doesn&#8217;t develop into your job (bear in mind: at all times attempt to take away the bottleneck). In my repos I take advantage of three Claude Code abilities to help me, written as plain markdown instruction information beneath <code>.claude\/abilities\/<\/code>:<\/p>\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\"><strong><code>\/pr-open<\/code> creates the PR:<\/strong> analyzes the department\u2019s commits, runs validation domestically, writes an trustworthy title and physique, and targets the best base. The PR physique issues greater than it appears, for the reason that reviewer reads it because the assertion of intent it evaluations in opposition to. I&#8217;ve it spell out the total characteristic request and reference the Linear ticket it implements, so each the reviewer and any human arriving later get the entire image with out leaving the PR.<\/li>\n<li class=\"wp-block-list-item\"><strong><code>\/pr-iterate<\/code> handles one assessment spherical:<\/strong> sweeps all three suggestions surfaces, then triages each merchandise right into a repair, a decline with reasoning, or a defer. Fixes are carried out and validated domestically; declines develop into PR feedback explaining why. The reviewer is advisory, and \u201cok to merge\u201d is a legitimate verdict as soon as the PR meets its acknowledged scope.<\/li>\n<li class=\"wp-block-list-item\"><strong><code>\/pr-babysit<\/code> is the watchdog round all of it:<\/strong> watch the CI, repair purple checks, run an iterate spherical when new suggestions lands, and loop till each test is inexperienced and each discovering is mounted or declined. Its terminal state is \u201cmerge-ready\u201d, and it stops to report slightly than ping-pong commits when the identical test fails twice on the identical root trigger.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\">One ordering rule that you just and your brokers ought to at all times be mindful: <strong>reply first, push after.<\/strong> The assessment kicks off on each push, and it really works with the context that exists at that actual second. Something not but written down (a decline, a repair abstract, a remark of your personal) doesn&#8217;t exist for that assessment spherical, and the reviewer will fortunately re-raise each merchandise it can&#8217;t see a response to.<\/p>\n<p class=\"wp-block-paragraph\">The tip state is a tidy division of labor. Claude writes and defends, Codex challenges and tracks, the loop runs till the PR is inexperienced with a clear suggestions desk. My queue comprises solely PRs which are already merge-ready, every carrying a assessment path I can audit in a single scroll: what was discovered, what was mounted, and what was declined and why.<\/p>\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\" wp_automatic_readability=\"8\">\n<p class=\"wp-block-paragraph\"><strong>Perception:<\/strong> automate each side of the assessment. When the creator agent should repair or contest each discovering in writing earlier than it will probably push, the human queue solely ever comprises merge-ready PRs with an auditable assessment path.<\/p>\n<\/blockquote>\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-dotted\"\/>\n<h2 class=\"wp-block-heading\">What a round-trip appears like in apply<\/h2>\n<p class=\"wp-block-paragraph\">To make the mechanism concrete, right here is one alternate from a backend PR of mine. The PR mounted a capability grid that wrote extra cells than a hardcoded batch cap allowed. The primary assessment spherical got here again with two findings:<\/p>\n<pre class=\"wp-block-prismatic-blocks\"><code class=\"language-markdown\">## Suggestions Abstract\n\n| Subject                           | Severity | Abstract                                  | Resolved |\n| ------------------------------- | -------- | ---------------------------------------- | -------- |\n| Truncation past the batch cap | Excessive     | Cells previous the cap are silently dropped. | \u274c       |\n| Cap duplicates a platform restrict | Medium   | Transfer the cap into app config.            | \u274c       |\n\n<\/code><\/pre>\n<p class=\"wp-block-paragraph\">The <code>Codex assessment<\/code> standing turned purple. Claude carried out the primary discovering for actual, then declined the second in a PR remark (actual remark, anonymized):<\/p>\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\" wp_automatic_readability=\"21\">\n<p class=\"wp-block-paragraph\">Addressed the Excessive discovering (hard-coded row cap within the knowledge layer, inflicting avoidable 502s).<\/p>\n<p class=\"wp-block-paragraph\"><code>_fetch_capacity_grid<\/code> now <strong>adapts<\/strong> as an alternative of failing: on a brief web page (which, as a result of the grid is dense at precisely <strong>days<\/strong> rows\/particular person, unambiguously means the deployed cap truncated the batch) it halves <code>batch_size<\/code> and retries the identical slice, shrinking till pages match. A brief web page solely turns into a loud 502 as soon as <code>batch_size == 1 [...]<\/code><\/p>\n<p class=\"wp-block-paragraph\">On the secondary suggestion (transfer the cap to config): deliberately not accomplished. The true cap lives within the database\u2019s personal hosted API settings, not an app env var. Mirroring it into app config would add a second supply of reality that would silently drift from the database. Adaptive batching removes the necessity to know the precise worth in any respect, which is the stronger repair. Present capacity-grid DB take a look at nonetheless passes; lint and type-check inexperienced.<\/p>\n<\/blockquote>\n<p class=\"wp-block-paragraph\">Reply first, push after, and have Codex re-review:<\/p>\n<pre class=\"wp-block-prismatic-blocks\"><code class=\"language-markdown\">## Suggestions Abstract\n\n| Subject                           | Severity | Abstract                          | Resolved |\n| ------------------------------- | -------- | -------------------------------- | -------- |\n| Truncation past the batch cap | Excessive     | Mounted by way of adaptive batching.     | \u2705       |\n| Cap duplicates a platform restrict | Medium   | Declined: duplicate would drift. | \u2705       |\n\n<\/code><\/pre>\n<p class=\"wp-block-paragraph\">Each rows resolved, one by a repair and one by a written decline that the reviewer accepted and recorded. That second row is your complete mechanism in a single line: no churn was carried out simply to silence a remark, and the reasoning is now a part of the PR\u2019s historical past for whoever reads it subsequent.<\/p>\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-dotted\"\/>\n<h2 class=\"wp-block-heading\">The learnings ledger: what it prices and the place it bites<\/h2>\n<p class=\"wp-block-paragraph\">This setup didn&#8217;t come collectively in a single go. Most of its particulars grew out of surprises and useless ends alongside the way in which, so I&#8217;m writing the learnings down right here, in tough order of cash saved, to spare you from driving into the identical partitions:<\/p>\n<ol class=\"wp-block-list\">\n<li class=\"wp-block-list-item\"><strong>Pin the mannequin.<\/strong> The <code>mannequin<\/code> enter of <code>codex-action<\/code> is optionally available, and leaving it empty means \u201cno matter Codex presently defaults to\u201d. I discovered this on my invoice when the default silently moved to a more recent, pricier mannequin (<code>gpt-5.5<\/code>). Pins resolve this downside, however want upkeep in return; presently I take advantage of <code>gpt-5.3-codex<\/code>, which is already marked deprecated on the Codex fashions web page on the time of writing (whereas remaining out there on the API). I revisit the pin intentionally as an alternative of letting the default resolve for me.<\/li>\n<li class=\"wp-block-list-item\"><strong>Cap the trouble at <code>medium<\/code>.<\/strong> Greater reasoning effort catches marginally extra points for considerably extra time and money. Velocity issues greater than it appears: the assessment sits contained in the agent\u2019s iterate loop, so its latency multiplies throughout each spherical of each PR, and your personal ready time is a part of the worth. Medium has been the issues-per-dollar <em>and<\/em> issues-per-minute candy spot. Sadly, no effort stage catches every little thing; that&#8217;s what the merge-owning human is for.<\/li>\n<li class=\"wp-block-list-item\"><strong>Retry the fast failures.<\/strong> A contemporary runner token often 401s, and the primary connection to the API often hiccups. Two makes an attempt with a gated fallback hold these from turning the workflow purple \u201cfor nothing\u201d, which issues as soon as brokers (and your personal belief) deal with purple as an actual sign.<\/li>\n<li class=\"wp-block-list-item\"><strong>Cancel outmoded evaluations.<\/strong> The <code>concurrency<\/code> block means solely the newest commit of a PR will get reviewed. With out it, an agent pushing three fixes in a row buys you three evaluations, two of them for code that now not exists.<\/li>\n<li class=\"wp-block-list-item\"><strong>Evaluation in opposition to the bottom that can obtain the merge.<\/strong> My PRs goal <code>dev<\/code>, so the assessment diffs in opposition to <code>dev<\/code>, and I hold the characteristic department synced with that base. A stale department will get diffed in opposition to a base that has moved on, and the assessment then studies phantom findings for adjustments the PR by no means made.<\/li>\n<li class=\"wp-block-list-item\"><strong>Characteristic branches are the prerequisite.<\/strong> All of this hangs off <code>pull_request<\/code> occasions. Direct pushes to <code>dev<\/code> or <code>most important<\/code> bypass the reviewer, the standing, and the audit path totally. The one-branch-per-change circulate is what makes each change reviewable within the first place.<\/li>\n<li class=\"wp-block-list-item\"><strong>Assume PR textual content is untrusted enter.<\/strong> The whole lot the reviewer reads (the PR physique, the dialog, the code itself) is a prompt-injection floor, and the agent executes in your runner along with your API key. The least-privilege token caps the blast radius on the GitHub facet, and <code>codex-action<\/code> ships sandboxing and safety-strategy knobs for the runner facet. Additionally know that GitHub doesn&#8217;t expose secrets and techniques to workflows triggered from forked PRs, so this sample assumes same-repo branches from trusted contributors. Evaluation the belief mannequin earlier than copying it into an open-source repo.<\/li>\n<\/ol>\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-dotted\"\/>\n<h2 class=\"wp-block-heading\">You&#8217;re the editor-in-chief now<\/h2>\n<p class=\"wp-block-paragraph\">The purpose of this pipeline was by no means to take away the human from the loop, its intent is to uplevel you. I finished being the primary reader of each diff and have become the one who reads evaluations, adjudicates the occasional standoff between two fashions, and owns the requirements written into the assessment immediate. When Claude and Codex agree, the PR might be high quality. Once they disagree, that disagreement is a precision-guided pointer at precisely the code that deserves my consideration. The place Claude typically acts sloppy, Codex tends to over-engineer, holding that stability in your personal fingers is a superb place to be.<\/p>\n<p class=\"wp-block-paragraph\">Reviewing the evaluations (so meta) is a greater job than proofreading agent output at agent velocity, and it degrades gracefully. On a lazy day I merge inexperienced PRs on the energy of the path, on a cautious day I learn the suggestions desk and spot-check the declines. Both method, nothing reaches the merge button on one mannequin\u2019s assured say-so.<\/p>\n<h3 class=\"wp-block-heading\">Key insights from this put up<\/h3>\n<ol class=\"wp-block-list\">\n<li class=\"wp-block-list-item\">Hallucinations are a relentless, throughput is what modified. Any course of that requires a human to learn each line caps your brokers at human studying velocity, so the very first assessment cross must be automated.<\/li>\n<li class=\"wp-block-list-item\">An creator\u2019s bugs are by definition believable to the creator, and analysis confirms LLMs are positively biased towards their very own output. Evaluation worth comes from decorrelation, by utilizing a mannequin from a unique supplier that fails in another way, and completely different is what catches what you miss.<\/li>\n<li class=\"wp-block-list-item\">The mannequin name is the simple half. Reviewer reliability comes from bizarre CI engineering round it.<\/li>\n<li class=\"wp-block-list-item\">Construction beats uncooked mannequin high quality in a reviewer. A lifecycle desk, a decline rule, and a machine-readable verdict flip free-form LLM opinions right into a merge gate with a mechanism for written pushback that makes false positives low-cost.<\/li>\n<li class=\"wp-block-list-item\">Automate each side of the assessment. When the creator agent should repair or contest each discovering in writing earlier than it will probably push, the human queue solely ever comprises merge-ready PRs with an auditable assessment path.<\/li>\n<\/ol>\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\" wp_automatic_readability=\"13\">\n<p class=\"wp-block-paragraph\"><strong>Ultimate perception:<\/strong> work multimodel. Your brokers will typically hand you assured fiction, and fashions from completely different suppliers fill one another\u2019s gaps, so that you don\u2019t should. A second-provider reviewer with reminiscence and a contract, sitting between your brokers and your most important department is a superb place to begin, however it&#8217;s not the one one. At any time when one mannequin\u2019s output is about to matter, a second opinion from a unique lab is the most cost effective insurance coverage you should purchase.<\/p>\n<\/blockquote>\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-dotted\"\/>\n<p class=\"wp-block-paragraph\">Observe me on <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/www.linkedin.com\/in\/rubenbroekx\/\">LinkedIn<\/a> for bite-sized AI insights, <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/towardsdatascience.com\/author\/broekxruben\/\">In direction of Knowledge Science<\/a> for early entry to new posts, or <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/medium.com\/@broekxruben\">Medium<\/a> for the total archive.<\/p>\n<\/div>\n\n","protected":false},"excerpt":{"rendered":"<p>I had a handful of parallel Claude brokers write me up a method folder for a facet challenge. The pull request appeared nice: ten paperwork, all neatly cross-linked, and it learn effectively. Then the automated reviewer posted its findings, and Claude, to its credit score, didn&#8217;t object: Claude conceding, mid-session: \u201cThe automated assessment caught an [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":16777,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55],"tags":[458,2476,9797,8297],"class_list":["post-16775","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","tag-claude","tag-dont","tag-grade","tag-homework"],"_links":{"self":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/16775","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=16775"}],"version-history":[{"count":1,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/16775\/revisions"}],"predecessor-version":[{"id":16776,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/16775\/revisions\/16776"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/media\/16777"}],"wp:attachment":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=16775"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=16775"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=16775"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}<!-- This website is optimized by Airlift. Learn more: https://airlift.net. Template:. Learn more: https://airlift.net. Template: 69d9690a190636c2e0989534. Config Timestamp: 2026-04-10 21:18:02 UTC, Cached Timestamp: 2026-07-17 21:09:50 UTC -->