{"id":16775,"date":"2026-07-16T11:24:23","date_gmt":"2026-07-16T11:24:23","guid":{"rendered":"https:\/\/techtrendfeed.com\/?p=16775"},"modified":"2026-07-16T11:24:24","modified_gmt":"2026-07-16T11:24:24","slug":"dont-let-claude-grade-its-personal-homework","status":"publish","type":"post","link":"https:\/\/techtrendfeed.com\/?p=16775","title":{"rendered":"Don\u2019t Let Claude Grade Its Personal Homework"},"content":{"rendered":"<p> <br \/>\n<\/p>\n<div wp_automatic_readability=\"356.7986934915\">\n<p class=\"wp-block-paragraph\"> I had a handful of parallel Claude brokers write me up a method folder for a facet challenge. The pull request appeared nice: ten paperwork, all neatly cross-linked, and it learn effectively. Then the automated reviewer posted its findings, and Claude, to its credit score, didn&#8217;t object:<\/p>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/07\/claude_admitting_ignorance-1024x234.jpeg\" alt=\"\" class=\"wp-image-672673\"\/><figcaption class=\"wp-element-caption\">Claude conceding, mid-session: \u201cThe automated assessment caught an actual bug: the doc writers used guessed filenames for cross-links. Legitimate discovering, I\u2019ll repair it.\u201d The docs it wrote linked to information it invented itself.<\/figcaption><\/figure>\n<p class=\"wp-block-paragraph\">The total audit got here again: 30 of the 95 relative hyperlinks pointed at information that didn&#8217;t exist. <code>.\/market.md<\/code> as an alternative of <code>market-analysis.md<\/code>, <code>.\/monetization.md<\/code> as an alternative of <code>business-model.md<\/code>, some information have been even made up totally! Skimming that PR as a human, I&#8217;d have accepted it, since each considered one of them appeared believable. Fortunately, my automated reviewer caught it as a result of it resolved each hyperlink as an alternative of judging whether or not the names \u201cvibed\u201d proper.<\/p>\n<p class=\"wp-block-paragraph\">To be honest, Claude doesn&#8217;t have any unhealthy intent behind this. A language mannequin merely generates probably the most believable continuation, and typically probably the most believable continuation is fiction delivered with full confidence. From the studying facet, although, that distinction brings little consolation. Assured and incorrect reads precisely like assured and proper.<\/p>\n<p class=\"wp-block-paragraph\">This put up is concerning the system I&#8217;ve in place to assist me assessment Claude\u2019s errors and hallucinations: a Codex agent working in GitHub Actions that evaluations each pull request. It walks you thru my setup, why cross-provider assessment beats each self-review and a <em>human-reads-everything<\/em> assessment, how one can implement reminiscence and accountability within the assessment loop, the associated fee and reliability learnings I gathered alongside the way in which, and the precise code I take advantage of to make this work.<\/p>\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-dotted\"\/>\n<h2 class=\"wp-block-heading\">Agent throughput broke the human assessment loop<\/h2>\n<p class=\"wp-block-paragraph\">Hallucinations by themselves are a manageable downside, since a really cautious assessment or purposeful evaluation would catch them. Nevertheless, what broke over the previous yr is the quantity that wants reviewing. A single engineer now runs a number of agent periods in parallel, and every of them can produce a thousand-line diff within the time it takes to refill your espresso. Producing code has develop into the quick and straightforward half; reviewing it truthfully is what eats the day. That makes the human reviewer the bottleneck of the entire supply pipeline, and the bottleneck is at all times the following factor it is best to automate.<\/p>\n<p class=\"wp-block-paragraph\">That bottleneck invitations two acquainted failure modes. Both you approve regardless of the brokers produce with out actually studying it, and the damaged hyperlinks, phantom parameters, and subtly incorrect edge circumstances ship to manufacturing. Otherwise you nonetheless insist on studying each line your self, and your <em>quick and sensible AI brokers<\/em> work even lower than you do. The primary is negligent, the second wasteful, and most groups oscillate between each relying on the deadline.<\/p>\n<p class=\"wp-block-paragraph\">The conclusion I hold touchdown on: our assessment course of has to scale with the velocity we\u2019re producing code. The human aspect stays, however the human reviewer\u2019s focus ought to go to the elements that really matter. The grind of checking whether or not hyperlinks resolve, whether or not the brand new parameter exists, whether or not there\u2019s a bug or vulnerability, and whether or not the code adjustments match what was initially requested, all of that has to run routinely on the similar cadence because the coding brokers, even earlier than an individual ever spends a second reviewing the code.<\/p>\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\" wp_automatic_readability=\"9\">\n<p class=\"wp-block-paragraph\"><strong>Perception:<\/strong> hallucinations are a relentless, throughput is what modified. Any course of that requires a human to learn each line caps your brokers at human studying velocity, so the very first assessment cross must be automated.<\/p>\n<\/blockquote>\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-dotted\"\/>\n<h2 class=\"wp-block-heading\">Decide a reviewer that doesn&#8217;t share the creator\u2019s blind spots<\/h2>\n<p class=\"wp-block-paragraph\">The apparent first thought to stop these errors is to have the agent assessment its personal work\u2026 however additionally it is the weakest one. A hallucination that made it into the diff is, by definition, believable to the mannequin that produced it. Asking the identical weights to seek out errors within the code, particularly throughout the similar already-biased coding session, normally reproduces the precise blind spots that allow the errors by.<\/p>\n<p class=\"wp-block-paragraph\">This instinct has analysis behind it. Panickssery et al. confirmed at NeurIPS 2024 that LLM evaluators <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/arxiv.org\/abs\/2404.13076\">acknowledge and favor their very own generations<\/a>: the higher a mannequin is at figuring out its personal output, the extra generously it scores it. Observe-up work ties this <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/arxiv.org\/abs\/2410.21819\">self-preference bias to familiarity<\/a>: AI judges rating textual content increased when it reads as predictable to them, and nothing reads as extra predictable to a mannequin than the textual content it simply wrote itself.<\/p>\n<p class=\"wp-block-paragraph\">Totally different LLM suppliers prepare on completely different knowledge, with completely different recipes and completely different suggestions loops. No supplier manages to get rid of errors, however the form of errors a mannequin makes traces again to the way it was skilled, and that differs per supplier. That decorrelation is strictly what you need in your assessment agent. Within the yr that I\u2019ve used this setup, OpenAI\u2019s Codex has persistently flagged points in Claude-written PRs that Claude\u2019s personal assessment handed with out blinking, from invented filenames to whole code blocks that had nothing to do with the preliminary request, or that launched outright safety violations.<\/p>\n<p class=\"wp-block-paragraph\">One conclusion you can&#8217;t take from that is that Codex is the higher engineer and it is best to hand it the keyboard as an alternative of Claude. Codex misfires pretty repeatedly, too! Simply in a unique route than Claude does. The facility sits in working multimodel: combining suppliers throughout the similar workflow, the place the fashions fill one another\u2019s gaps.<\/p>\n<p class=\"wp-block-paragraph\">One other incorrect conclusion is that Codex\u2019s evaluations are good and may be accepted blindly. Hallucinations occur in evaluations too, so deal with the assessment as advisory. Difficult the creator (be it Claude or your self) places the concentrate on potential blind spots, which is the right sign to zoom in and mirror. It stays the creator\u2019s job to personal the ultimate high quality, and typically which means writing again to Codex that its assessment was incorrect. Extra on this mechanism later within the put up.<\/p>\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\" wp_automatic_readability=\"10\">\n<p class=\"wp-block-paragraph\"><strong>Perception:<\/strong> an creator\u2019s bugs are by definition believable to the creator, and analysis confirms LLMs are positively biased towards their very own output. Evaluation worth comes from decorrelation, by utilizing a mannequin from a unique supplier that fails in another way, and completely different is what catches what you miss.<\/p>\n<\/blockquote>\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-dotted\"\/>\n<h2 class=\"wp-block-heading\">The pipeline: one LLM name wrapped in bizarre engineering<\/h2>\n<p class=\"wp-block-paragraph\">Conceptually, an LLM reviewer may be written in a single line: \u201cright here is the code diff, please assessment it\u201d. Each OpenAI and Anthropic ship hosted shortcuts for precisely that: OpenAI has a <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/developers.openai.com\/codex\/integrations\/github\">GitHub assessment integration<\/a> that auto-reviews PRs, and Anthropic has <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/github.com\/anthropics\/claude-code-action\">claude-code-action<\/a>. They&#8217;re high quality beginning factors, however I choose to run my very own motion as an alternative, as a result of I wish to personal the immediate, the assessment lifecycle, the merge gating, and the mannequin pin. All these elements matter when your day-to-day turns into managing and reviewing brokers.<\/p>\n<p class=\"wp-block-paragraph\">So what am I utilizing then, precisely? Two information that dwell within the repository they assessment:<\/p>\n<pre class=\"wp-block-prismatic-blocks\" datatext=\"\"><code class=\"language-markdown\">.github\/\n\u251c\u2500\u2500 workflows\/\n\u2502   \u2514\u2500\u2500 pr.yml            # orchestration: triggers, context, retries, posting\n\u2514\u2500\u2500 actions\/\n    \u2514\u2500\u2500 codex-pr-review\/\n        \u2514\u2500\u2500 motion.yml    # the reviewer: pinned Codex name + assessment immediate<\/code><\/pre>\n<p class=\"wp-block-paragraph\">The runtime circulate is brief. Each time a PR opens or receives a push, <code>pr.yml<\/code> checks out the code, collects the PR\u2019s full dialog historical past, and fingers each to the composite motion. The motion runs Codex over the diff and returns its findings as a single message. The workflow then posts that message as a single, constantly up to date remark within the PR dialog, and publishes a commit standing that stays purple whereas findings stay unresolved: the go\/no-go sign sitting subsequent to the remainder of CI.<\/p>\n<p class=\"wp-block-paragraph\">On the heart of all of it sits one deceptively easy step, the one line that truly touches an LLM:<\/p>\n<pre class=\"wp-block-prismatic-blocks\"><code class=\"language-yaml\">- makes use of: openai\/codex-action@v1\n  with:\n    openai-api-key: ${{ inputs.openai-api-key }}\n    immediate: |\n      # the assessment contract: what to assessment, how one can observe findings,\n      # and how one can report again (unpacked within the subsequent part)<\/code><\/pre>\n<p class=\"wp-block-paragraph\">The whole lot else exists to run that single step reliably, and to verify its suggestions truly lands on the PR. Right here is the entire workflow:<\/p>\n<pre class=\"wp-block-prismatic-blocks\" datatext=\"\"><code class=\"language-yaml\">identify: PR Reviewer\n\non:\n  pull_request:\n    branches: [dev] # all work rides characteristic branches: characteristic -&gt; dev -&gt; most important\n    varieties: [opened, synchronize, reopened]\n\n# a brand new push cancels the in-flight (costly) assessment of the earlier commit\nconcurrency:\n  group: pr-reviewer-${{ github.occasion.pull_request.quantity }}\n  cancel-in-progress: true\n\njobs:\n  codex:\n    if: github.occasion.pull_request.consumer.login != 'dependabot[bot]'\n    runs-on: ubuntu-latest\n    timeout-minutes: 15 # regular assessment takes a couple of minutes, so 15m is greater than sufficient\n    permissions: # the token can learn code and put up suggestions, however can by no means push or merge\n      contents: learn\n      points: write\n      pull-requests: write\n      statuses: write\n    steps:\n      # the \/head ref updates synchronously on each push;\n      # the auto-generated merge ref can race the checkout\n      - makes use of: actions\/checkout@v7\n        with:\n          ref: refs\/pull\/${{ github.occasion.pull_request.quantity }}\/head\n\n      # ensure each diff endpoints exist domestically, so Codex can resolve base...head\n      - identify: Pre-fetch base and head refs for the PR\n        run: |\n          git fetch --no-tags origin \n            ${{ github.occasion.pull_request.base.ref }} \n            +refs\/pull\/${{ github.occasion.pull_request.quantity }}\/head\n\n      # suggestions lives on three surfaces: challenge feedback, evaluations, and inline feedback\n      - identify: Fetch PR dialog\n        id: dialog\n        makes use of: actions\/github-script@v9\n        with:\n          result-encoding: string\n          script: |\n            const prNumber = context.payload.pull_request.quantity;\n            const opts = { proprietor: context.repo.proprietor, repo: context.repo.repo };\n            const [comments, reviews, inline] = await Promise.all([\n              github.paginate(github.rest.issues.listComments,\n                { ...opts, issue_number: prNumber }),\n              github.paginate(github.rest.pulls.listReviews,\n                { ...opts, pull_number: prNumber }),\n              github.paginate(github.rest.pulls.listReviewComments,\n                { ...opts, pull_number: prNumber }),\n            ]);\n            const all = [\n              ...comments.map(c =&gt; ({\n                date: c.created_at,\n                text: `@${c.user.login}: ${c.body}`,\n              })),\n              ...reviews.filter(r =&gt; r.body).map(r =&gt; ({\n                date: r.submitted_at,\n                text: `@${r.user.login} (review ${r.state}): ${r.body}`,\n              })),\n              ...inline.map(c =&gt; ({\n                date: c.created_at,\n                text: `@${c.user.login} (on ${c.path}): ${c.body}`,\n              })),\n            ];\n            all.type((a, b) =&gt; new Date(a.date) - new Date(b.date));\n            return all.map(e =&gt; e.textual content).be a part of('n---n');\n\n      # secrets and techniques don't journey with the yml, talk loudly on lacking keys\n      - identify: Fail quick on lacking OPENAI_API_KEY\n        env:\n          OPENAI_API_KEY: ${{ secrets and techniques.OPENAI_API_KEY }}\n        run: |\n          if [ -z \"${OPENAI_API_KEY}\" ]; then\n            echo \"::error::OPENAI_API_KEY secret will not be set for this repository.\"\n            exit 1\n          fi\n\n      # retry wrappers can't wrap `makes use of:` steps, so retry manually: transient\n      # failures (a 401 from a contemporary runner token, a hiccup reaching OpenAI)\n      # mustn't flip the entire run purple\n      - identify: Run Codex (try 1)\n        id: codex_try1\n        makes use of: .\/.github\/actions\/codex-pr-review\n        continue-on-error: true\n        with:\n          openai-api-key: ${{ secrets and techniques.OPENAI_API_KEY }}\n          dialog: ${{ steps.dialog.outputs.outcome }}\n\n      # give transient failures a second to clear earlier than retrying\n      - identify: Wait earlier than Codex retry\n        if: steps.codex_try1.consequence == 'failure'\n        run: sleep 20\n\n      - identify: Run Codex (try 2)\n        id: codex_try2\n        if: steps.codex_try1.consequence == 'failure'\n        makes use of: .\/.github\/actions\/codex-pr-review\n        with:\n          openai-api-key: ${{ secrets and techniques.OPENAI_API_KEY }}\n          dialog: ${{ steps.dialog.outputs.outcome }}\n\n      - identify: Put up the assessment and publish a standing\n        makes use of: actions\/github-script@v9\n        env:\n          REVIEW: &gt;-\n            ${}\n        with:\n          script: |\n            const MARKER = '<!-- codex-pr-review -->';\n            const opts = { proprietor: context.repo.proprietor, repo: context.repo.repo };\n            const prNumber = context.payload.pull_request.quantity;\n            const uncooked = course of.env.REVIEW;\n            if (!uncooked) return;\n            const physique = `${MARKER}n${uncooked}`;\n\n            \/\/ one canonical remark, edited in place on each push\n            const feedback = await github.paginate(\n              github.relaxation.points.listComments, { ...opts, issue_number: prNumber });\n            const present = feedback.discover(c =&gt;\n              c.consumer?.login === 'github-actions[bot]' &amp;&amp; c.physique?.consists of(MARKER));\n            const posted = present\n              ? await github.relaxation.points.updateComment(\n                  { ...opts, comment_id: present.id, physique })\n              : await github.relaxation.points.createComment(\n                  { ...opts, issue_number: prNumber, physique });\n\n            \/\/ go\/no-go: a commit standing that stays purple whereas findings are open\n            \/\/ (a lacking trailer reads as \"assessment posted\", by no means as zero findings)\n            const m = uncooked.match(\/<!--s*codex-review:s*unresolved=(d+)\/i);\n            const unresolved = m ? Number(m[1]) : null;\n            await github.rest.repos.createCommitStatus({\n              ...opts,\n              sha: context.payload.pull_request.head.sha,\n              context: &#039;Codex review&#039;,\n              state: unresolved > 0 ? &#039;failure&#039; : &#039;success&#039;,\n              description: unresolved === null\n                ? &#039;Evaluation posted, see Suggestions Abstract&#039;\n                : unresolved > 0\n                  ? `${unresolved} unresolved discovering(s)`\n                  : &#039;All findings resolved&#039;,\n              target_url: posted.knowledge.html_url,\n            });<\/code><\/pre>\n<p class=\"wp-block-paragraph\">A couple of of those strains deserve a better look, since every of them earned its place by a failure I truly hit:<\/p>\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\"><strong>The whole lot runs in a single job.<\/strong> The <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/github.com\/openai\/codex-action\">codex-action README<\/a> reveals a two-job instance: one job runs Codex and exports the assessment by a job output, a second job posts it. I began with precisely that, because the docs counsel, however then evaluations started arriving empty. Apparently, GitHub refuses to cross alongside any job output that comprises a substring of a registered secret (a single warning line buried within the run logs is the one hint), and it fires on false positives on a regular basis: most hash-shaped strings look shut sufficient to a credential, even those Codex would come with in its report as a placeholder. The absurd half is that the blanking protects nothing: the flagged &#8220;secret&#8221; nonetheless sits in plain sight within the first job&#8217;s logs, and the one factor successfully hidden is the assessment itself, which by no means crosses the job boundary. Conserving every little thing in a single job sidesteps the test totally, so the assessment at all times reaches the PR.<\/li>\n<\/ul>\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\"><strong>The dialog fetch sweeps three surfaces.<\/strong> Subject feedback, assessment summaries, and inline feedback are separate APIs, and suggestions lands on all three. Regardless of the sweep collects is handed to Codex verbatim, in chronological order. This ensures that the Codex reviewer has the correct context to do its job. One caveat to pay attention to: this historical past retains rising on long-lived PRs, so excessive threads might ultimately want trimming or summarizing.<\/li>\n<\/ul>\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\"><strong>The important thing test fails quick.<\/strong> Secrets and techniques don't journey with the yml (nor ought to they be written contained in the yml!). Copy this workflow right into a contemporary repo, neglect so as to add <code>OPENAI_API_KEY<\/code>, and also you&#8217;ll get a complicated failure deep contained in the motion as an alternative of the one-line error we added to spotlight the precise downside.<\/li>\n<\/ul>\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\"><strong>Two makes an attempt, manually.<\/strong> A retry wrapper can't wrap <code>makes use of:<\/code> steps, so our retry is a <code>continue-on-error<\/code> first try plus a guarded second one. And the output choice gates try 1 on its consequence: a uncooked fallback (<code>try1 || try2<\/code>) would fortunately choose up a stale partial message from a run that failed midway.<\/li>\n<\/ul>\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\"><strong>The token is least-privilege.<\/strong> It will possibly learn code, put up feedback, and set a standing. It can't push, approve, or merge. Regardless of the reviewer or the creator agent will get confused about, the blast radius ends at a PR remark.<\/li>\n<\/ul>\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\"><strong>Checkout pins <code>refs\/pull\/N\/head<\/code>.<\/strong> GitHub materializes the merge ref asynchronously, and a fast-triggering workflow can race it. The <code>\/head<\/code> ref updates synchronously on each push. The pre-fetch step proper after it makes certain each diff endpoints exist domestically: the immediate fingers Codex the bottom and head SHAs by worth, and Codex runs the git instructions to diff them itself.<\/li>\n<\/ul>\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\"><strong>Perception:<\/strong> the mannequin name is the simple half. Reviewer reliability comes from bizarre CI engineering round it.<\/p>\n<\/blockquote>\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-dotted\"\/>\n<h2 class=\"wp-block-heading\">A assessment that remembers is best than a assessment that repeats<\/h2>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/07\/two_lamps_one_manuscript-1024x576.png\" alt=\"\" class=\"wp-image-673536\"\/><figcaption class=\"wp-element-caption\">The reviewer&#8217;s job is the marked-up web page. The creator&#8217;s job is deciding, in writing, what to do with every discovering. Made with ChatGPT.<\/figcaption><\/figure>\n<p class=\"wp-block-paragraph\">Most AI reviewers produce drive-by feedback: a point-in-time opinion on a point-in-time diff, resulting in an additional wall of textual content on each new push. The immediate contract is the place this setup differs, and it's the core of our assessment agent:<\/p>\n<pre class=\"wp-block-prismatic-blocks\"><code class=\"language-yaml\">identify: Codex PR Evaluation\ndescription: Run openai\/codex-action with the challenge&#039;s PR assessment immediate.\n\ninputs:\n  openai-api-key:\n    description: OpenAI API key.\n    required: true\n  dialog:\n    description: Pre-fetched PR dialog historical past.\n    required: true\n\noutputs:\n  final-message:\n    description: Codex&#039;s assessment textual content.\n    worth: ${{ steps.codex.outputs[&#039;final-message&#039;] }}\n\nruns:\n  utilizing: composite\n  steps:\n    - id: codex\n      makes use of: openai\/codex-action@v1\n      with:\n        openai-api-key: ${{ inputs.openai-api-key }}\n        mannequin: gpt-5.3-codex # pin it: the default floats, and so does your invoice\n        effort: medium # the issues-per-dollar candy spot\n        immediate: |\n          That is PR #${{ github.occasion.pull_request.quantity }} for ${{ github.repository }}.\n          Base SHA: ${{ github.occasion.pull_request.base.sha }}\n          Head SHA: ${{ github.occasion.pull_request.head.sha }}\n\n          Evaluation ONLY the adjustments launched by this PR. Counsel\n          enhancements, potential bugs, and points. Be concise and\n          particular. Write in markdown format, with a brand new title for every\n          challenge\/subject you increase. When producing tables, format them\n          appropriately and always remember the header separator row. Inside a\n          desk cell, a literal `|` character breaks the row: escape\n          each pipe as `|`, together with pipes inside backticks (write\n          `` `curl ... | bash` ``). When doubtful, rewrite the cell to\n          keep away from the pipe totally. Analyze the dialog historical past to\n          perceive the context and the suggestions already given. Report\n          what has been mounted, what has been missed, and any new points\n          launched since.\n\n          If a maintainer has replied to a suggestions merchandise with a reasoned\n          choice to not act on it (a rejection, a &quot;received&#039;t repair&quot;, or an\n          rationalization that it's intentional or out of scope), deal with that\n          merchandise as resolved: mark it resolved within the abstract desk with a\n          one-line notice of their rationale, and do NOT re-raise it. The\n          maintainer&#039;s choice is ultimate: your function is to advise, to not\n          re-litigate. Re-raise such an merchandise provided that later adjustments to the\n          PR have genuinely invalidated their reasoning.\n\n          Format your suggestions as follows:\n            - A `## Suggestions Abstract` part: a desk monitoring ALL suggestions\n              ever raised throughout this PR&#039;s lifetime (columns: `Subject`,\n              `Severity`, `Abstract`, `Resolved`), utilizing &#9989; for resolved and\n              &#10060; for unresolved rows, sorted by severity, unresolved first.\n            - A `## Open Points` part: solely the unresolved points, every\n              beneath its personal `### Subject title [severity]` heading, structured\n              with **Subject:**, **Proposed repair:**, and **Associated information:**.\n            - Finish with a single machine-readable line:\n              `<!-- codex-review: unresolved=N, high=M -->`, the place N counts\n              the unresolved rows and M what number of of these are severity Excessive.\n              At all times embody it, even when N is 0: CI parses it into the\n              standing test.\n\n          Pull request title and physique:\n          ----\n          ${{ github.occasion.pull_request.title }}\n          ${{ github.occasion.pull_request.physique }}\n\n          Dialog historical past:\n          ----\n          ${{ inputs.dialog }}<\/code><\/pre>\n<p class=\"wp-block-paragraph\">Three design selections carry the load right here.<\/p>\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\"><strong>The assessment is stateful.<\/strong> The Suggestions Abstract desk tracks each discovering ever raised on the PR, resolved or not, throughout pushes. Mixed with the edit-in-place remark from the workflow, a PR carries precisely one assessment artifact that displays your complete lifecycle, as an alternative of 5 stale feedback that every mirror an replace within the assessment course of. Anybody opening the PR, human or agent, sees the total historical past at a look.<\/li>\n<\/ul>\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\"><strong>Declines are honored.<\/strong> As a result of Codex reads the entire dialog, the creator can reject a discovering in writing, and the rejection sticks throughout assessment cycles. That is the mechanism that makes an imperfect reviewer workable: false positives value one written reply as an alternative of an infinite nag loop, and the objection stays on file for the following reader.<\/li>\n<\/ul>\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\"><strong>The decision is machine-readable.<\/strong> The trailing <code><!-- codex-review: unresolved=N, high=M --><\/code> line will get parsed by the workflow right into a <code>Codex assessment<\/code> commit standing. The result&#8217;s two unbiased indicators on each PR: the workflow test tells you the assessment ran, and the standing tells you whether or not something continues to be open. Crimson standing means unresolved findings; inexperienced means every little thing is mounted or explicitly declined. I intentionally hold the standing non-blocking, because it solely features to tell. The choice to merge or not stays open, and stays human.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\">Notice how this contract asks for greater than typo searching. Codex evaluations the diff in opposition to the PR\u2019s acknowledged intent, ranks each discovering by severity, and attaches a concrete proposed repair, the form of assessment that repeatedly surfaces architecture-level points no linter would ever produce.<\/p>\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\" wp_automatic_readability=\"9\">\n<p class=\"wp-block-paragraph\"><strong>Perception:<\/strong> construction beats uncooked mannequin high quality in a reviewer. A lifecycle desk, a decline rule, and a machine-readable verdict flip free-form LLM opinions right into a merge gate with a mechanism for written pushback that makes false positives low-cost.<\/p>\n<\/blockquote>\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-dotted\"\/>\n<h2 class=\"wp-block-heading\">Shut the loop: the PR turns inexperienced earlier than you even look<\/h2>\n<p class=\"wp-block-paragraph\">The reviewer is just half the story once you take a look at the larger engineering image. The opposite half is educating the creator agent to answer it, so the assessment round-trip doesn&#8217;t develop into your job (bear in mind: at all times attempt to take away the bottleneck). In my repos I take advantage of three Claude Code abilities to help me, written as plain markdown instruction information beneath <code>.claude\/abilities\/<\/code>:<\/p>\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\"><strong><code>\/pr-open<\/code> creates the PR:<\/strong> analyzes the department\u2019s commits, runs validation domestically, writes an trustworthy title and physique, and targets the best base. The PR physique issues greater than it appears, for the reason that reviewer reads it because the assertion of intent it evaluations in opposition to. I&#8217;ve it spell out the total characteristic request and reference the Linear ticket it implements, so each the reviewer and any human arriving later get the entire image with out leaving the PR.<\/li>\n<li class=\"wp-block-list-item\"><strong><code>\/pr-iterate<\/code> handles one assessment spherical:<\/strong> sweeps all three suggestions surfaces, then triages each merchandise right into a repair, a decline with reasoning, or a defer. Fixes are carried out and validated domestically; declines develop into PR feedback explaining why. The reviewer is advisory, and \u201cok to merge\u201d is a legitimate verdict as soon as the PR meets its acknowledged scope.<\/li>\n<li class=\"wp-block-list-item\"><strong><code>\/pr-babysit<\/code> is the watchdog round all of it:<\/strong> watch the CI, repair purple checks, run an iterate spherical when new suggestions lands, and loop till each test is inexperienced and each discovering is mounted or declined. Its terminal state is \u201cmerge-ready\u201d, and it stops to report slightly than ping-pong commits when the identical test fails twice on the identical root trigger.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\">One ordering rule that you just and your brokers ought to at all times be mindful: <strong>reply first, push after.<\/strong> The assessment kicks off on each push, and it really works with the context that exists at that actual second. Something not but written down (a decline, a repair abstract, a remark of your personal) doesn&#8217;t exist for that assessment spherical, and the reviewer will fortunately re-raise each merchandise it can&#8217;t see a response to.<\/p>\n<p class=\"wp-block-paragraph\">The tip state is a tidy division of labor. Claude writes and defends, Codex challenges and tracks, the loop runs till the PR is inexperienced with a clear suggestions desk. My queue comprises solely PRs which are already merge-ready, every carrying a assessment path I can audit in a single scroll: what was discovered, what was mounted, and what was declined and why.<\/p>\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\" wp_automatic_readability=\"8\">\n<p class=\"wp-block-paragraph\"><strong>Perception:<\/strong> automate each side of the assessment. When the creator agent should repair or contest each discovering in writing earlier than it will probably push, the human queue solely ever comprises merge-ready PRs with an auditable assessment path.<\/p>\n<\/blockquote>\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-dotted\"\/>\n<h2 class=\"wp-block-heading\">What a round-trip appears like in apply<\/h2>\n<p class=\"wp-block-paragraph\">To make the mechanism concrete, right here is one alternate from a backend PR of mine. The PR mounted a capability grid that wrote extra cells than a hardcoded batch cap allowed. The primary assessment spherical got here again with two findings:<\/p>\n<pre class=\"wp-block-prismatic-blocks\"><code class=\"language-markdown\">## Suggestions Abstract\n\n| Subject                           | Severity | Abstract                                  | Resolved |\n| ------------------------------- | -------- | ---------------------------------------- | -------- |\n| Truncation past the batch cap | Excessive     | Cells previous the cap are silently dropped. | \u274c       |\n| Cap duplicates a platform restrict | Medium   | Transfer the cap into app config.            | \u274c       |\n\n<!-- codex-review: unresolved=2, high=1 --><\/code><\/pre>\n<p class=\"wp-block-paragraph\">The <code>Codex assessment<\/code> standing turned purple. Claude carried out the primary discovering for actual, then declined the second in a PR remark (actual remark, anonymized):<\/p>\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\" wp_automatic_readability=\"21\">\n<p class=\"wp-block-paragraph\">Addressed the Excessive discovering (hard-coded row cap within the knowledge layer, inflicting avoidable 502s).<\/p>\n<p class=\"wp-block-paragraph\"><code>_fetch_capacity_grid<\/code> now <strong>adapts<\/strong> as an alternative of failing: on a brief web page (which, as a result of the grid is dense at precisely <strong>days<\/strong> rows\/particular person, unambiguously means the deployed cap truncated the batch) it halves <code>batch_size<\/code> and retries the identical slice, shrinking till pages match. A brief web page solely turns into a loud 502 as soon as <code>batch_size == 1 [...]<\/code><\/p>\n<p class=\"wp-block-paragraph\">On the secondary suggestion (transfer the cap to config): deliberately not accomplished. The true cap lives within the database\u2019s personal hosted API settings, not an app env var. Mirroring it into app config would add a second supply of reality that would silently drift from the database. Adaptive batching removes the necessity to know the precise worth in any respect, which is the stronger repair. Present capacity-grid DB take a look at nonetheless passes; lint and type-check inexperienced.<\/p>\n<\/blockquote>\n<p class=\"wp-block-paragraph\">Reply first, push after, and have Codex re-review:<\/p>\n<pre class=\"wp-block-prismatic-blocks\"><code class=\"language-markdown\">## Suggestions Abstract\n\n| Subject                           | Severity | Abstract                          | Resolved |\n| ------------------------------- | -------- | -------------------------------- | -------- |\n| Truncation past the batch cap | Excessive     | Mounted by way of adaptive batching.     | \u2705       |\n| Cap duplicates a platform restrict | Medium   | Declined: duplicate would drift. | \u2705       |\n\n<!-- codex-review: unresolved=0, high=0 --><\/code><\/pre>\n<p class=\"wp-block-paragraph\">Each rows resolved, one by a repair and one by a written decline that the reviewer accepted and recorded. That second row is your complete mechanism in a single line: no churn was carried out simply to silence a remark, and the reasoning is now a part of the PR\u2019s historical past for whoever reads it subsequent.<\/p>\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-dotted\"\/>\n<h2 class=\"wp-block-heading\">The learnings ledger: what it prices and the place it bites<\/h2>\n<p class=\"wp-block-paragraph\">This setup didn&#8217;t come collectively in a single go. Most of its particulars grew out of surprises and useless ends alongside the way in which, so I&#8217;m writing the learnings down right here, in tough order of cash saved, to spare you from driving into the identical partitions:<\/p>\n<ol class=\"wp-block-list\">\n<li class=\"wp-block-list-item\"><strong>Pin the mannequin.<\/strong> The <code>mannequin<\/code> enter of <code>codex-action<\/code> is optionally available, and leaving it empty means \u201cno matter Codex presently defaults to\u201d. I discovered this on my invoice when the default silently moved to a more recent, pricier mannequin (<code>gpt-5.5<\/code>). Pins resolve this downside, however want upkeep in return; presently I take advantage of <code>gpt-5.3-codex<\/code>, which is already marked deprecated on the Codex fashions web page on the time of writing (whereas remaining out there on the API). I revisit the pin intentionally as an alternative of letting the default resolve for me.<\/li>\n<li class=\"wp-block-list-item\"><strong>Cap the trouble at <code>medium<\/code>.<\/strong> Greater reasoning effort catches marginally extra points for considerably extra time and money. Velocity issues greater than it appears: the assessment sits contained in the agent\u2019s iterate loop, so its latency multiplies throughout each spherical of each PR, and your personal ready time is a part of the worth. Medium has been the issues-per-dollar <em>and<\/em> issues-per-minute candy spot. Sadly, no effort stage catches every little thing; that&#8217;s what the merge-owning human is for.<\/li>\n<li class=\"wp-block-list-item\"><strong>Retry the fast failures.<\/strong> A contemporary runner token often 401s, and the primary connection to the API often hiccups. Two makes an attempt with a gated fallback hold these from turning the workflow purple \u201cfor nothing\u201d, which issues as soon as brokers (and your personal belief) deal with purple as an actual sign.<\/li>\n<li class=\"wp-block-list-item\"><strong>Cancel outmoded evaluations.<\/strong> The <code>concurrency<\/code> block means solely the newest commit of a PR will get reviewed. With out it, an agent pushing three fixes in a row buys you three evaluations, two of them for code that now not exists.<\/li>\n<li class=\"wp-block-list-item\"><strong>Evaluation in opposition to the bottom that can obtain the merge.<\/strong> My PRs goal <code>dev<\/code>, so the assessment diffs in opposition to <code>dev<\/code>, and I hold the characteristic department synced with that base. A stale department will get diffed in opposition to a base that has moved on, and the assessment then studies phantom findings for adjustments the PR by no means made.<\/li>\n<li class=\"wp-block-list-item\"><strong>Characteristic branches are the prerequisite.<\/strong> All of this hangs off <code>pull_request<\/code> occasions. Direct pushes to <code>dev<\/code> or <code>most important<\/code> bypass the reviewer, the standing, and the audit path totally. The one-branch-per-change circulate is what makes each change reviewable within the first place.<\/li>\n<li class=\"wp-block-list-item\"><strong>Assume PR textual content is untrusted enter.<\/strong> The whole lot the reviewer reads (the PR physique, the dialog, the code itself) is a prompt-injection floor, and the agent executes in your runner along with your API key. The least-privilege token caps the blast radius on the GitHub facet, and <code>codex-action<\/code> ships sandboxing and safety-strategy knobs for the runner facet. Additionally know that GitHub doesn&#8217;t expose secrets and techniques to workflows triggered from forked PRs, so this sample assumes same-repo branches from trusted contributors. Evaluation the belief mannequin earlier than copying it into an open-source repo.<\/li>\n<\/ol>\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-dotted\"\/>\n<h2 class=\"wp-block-heading\">You&#8217;re the editor-in-chief now<\/h2>\n<p class=\"wp-block-paragraph\">The purpose of this pipeline was by no means to take away the human from the loop, its intent is to uplevel you. I finished being the primary reader of each diff and have become the one who reads evaluations, adjudicates the occasional standoff between two fashions, and owns the requirements written into the assessment immediate. When Claude and Codex agree, the PR might be high quality. Once they disagree, that disagreement is a precision-guided pointer at precisely the code that deserves my consideration. The place Claude typically acts sloppy, Codex tends to over-engineer, holding that stability in your personal fingers is a superb place to be.<\/p>\n<p class=\"wp-block-paragraph\">Reviewing the evaluations (so meta) is a greater job than proofreading agent output at agent velocity, and it degrades gracefully. On a lazy day I merge inexperienced PRs on the energy of the path, on a cautious day I learn the suggestions desk and spot-check the declines. Both method, nothing reaches the merge button on one mannequin\u2019s assured say-so.<\/p>\n<h3 class=\"wp-block-heading\">Key insights from this put up<\/h3>\n<ol class=\"wp-block-list\">\n<li class=\"wp-block-list-item\">Hallucinations are a relentless, throughput is what modified. Any course of that requires a human to learn each line caps your brokers at human studying velocity, so the very first assessment cross must be automated.<\/li>\n<li class=\"wp-block-list-item\">An creator\u2019s bugs are by definition believable to the creator, and analysis confirms LLMs are positively biased towards their very own output. Evaluation worth comes from decorrelation, by utilizing a mannequin from a unique supplier that fails in another way, and completely different is what catches what you miss.<\/li>\n<li class=\"wp-block-list-item\">The mannequin name is the simple half. Reviewer reliability comes from bizarre CI engineering round it.<\/li>\n<li class=\"wp-block-list-item\">Construction beats uncooked mannequin high quality in a reviewer. A lifecycle desk, a decline rule, and a machine-readable verdict flip free-form LLM opinions right into a merge gate with a mechanism for written pushback that makes false positives low-cost.<\/li>\n<li class=\"wp-block-list-item\">Automate each side of the assessment. When the creator agent should repair or contest each discovering in writing earlier than it will probably push, the human queue solely ever comprises merge-ready PRs with an auditable assessment path.<\/li>\n<\/ol>\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\" wp_automatic_readability=\"13\">\n<p class=\"wp-block-paragraph\"><strong>Ultimate perception:<\/strong> work multimodel. Your brokers will typically hand you assured fiction, and fashions from completely different suppliers fill one another\u2019s gaps, so that you don\u2019t should. A second-provider reviewer with reminiscence and a contract, sitting between your brokers and your most important department is a superb place to begin, however it&#8217;s not the one one. At any time when one mannequin\u2019s output is about to matter, a second opinion from a unique lab is the most cost effective insurance coverage you should purchase.<\/p>\n<\/blockquote>\n<hr class=\"wp-block-separator has-alpha-channel-opacity is-style-dotted\"\/>\n<p class=\"wp-block-paragraph\">Observe me on <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/www.linkedin.com\/in\/rubenbroekx\/\">LinkedIn<\/a> for bite-sized AI insights, <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/towardsdatascience.com\/author\/broekxruben\/\">In direction of Knowledge Science<\/a> for early entry to new posts, or <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/medium.com\/@broekxruben\">Medium<\/a> for the total archive.<\/p>\n<\/div>\n\n","protected":false},"excerpt":{"rendered":"<p>I had a handful of parallel Claude brokers write me up a method folder for a facet challenge. The pull request appeared nice: ten paperwork, all neatly cross-linked, and it learn effectively. Then the automated reviewer posted its findings, and Claude, to its credit score, didn&#8217;t object: Claude conceding, mid-session: \u201cThe automated assessment caught an [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":16777,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55],"tags":[458,2476,9797,8297],"class_list":["post-16775","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","tag-claude","tag-dont","tag-grade","tag-homework"],"_links":{"self":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/16775","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=16775"}],"version-history":[{"count":1,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/16775\/revisions"}],"predecessor-version":[{"id":16776,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/16775\/revisions\/16776"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/media\/16777"}],"wp:attachment":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=16775"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=16775"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=16775"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}