Don’t Let Claude Grade Its Personal Homework

I had a handful of parallel Claude brokers write me up a method folder for a facet challenge. The pull request appeared nice: ten paperwork, all neatly cross-linked, and it learn effectively. Then the automated reviewer posted its findings, and Claude, to its credit score, didn’t object:

Claude conceding, mid-session: “The automated assessment caught an actual bug: the doc writers used guessed filenames for cross-links. Legitimate discovering, I’ll repair it.” The docs it wrote linked to information it invented itself.

The total audit got here again: 30 of the 95 relative hyperlinks pointed at information that didn’t exist. ./market.md as an alternative of market-analysis.md, ./monetization.md as an alternative of business-model.md, some information have been even made up totally! Skimming that PR as a human, I’d have accepted it, since each considered one of them appeared believable. Fortunately, my automated reviewer caught it as a result of it resolved each hyperlink as an alternative of judging whether or not the names “vibed” proper.

To be honest, Claude doesn’t have any unhealthy intent behind this. A language mannequin merely generates probably the most believable continuation, and typically probably the most believable continuation is fiction delivered with full confidence. From the studying facet, although, that distinction brings little consolation. Assured and incorrect reads precisely like assured and proper.

This put up is concerning the system I’ve in place to assist me assessment Claude’s errors and hallucinations: a Codex agent working in GitHub Actions that evaluations each pull request. It walks you thru my setup, why cross-provider assessment beats each self-review and a human-reads-everything assessment, how one can implement reminiscence and accountability within the assessment loop, the associated fee and reliability learnings I gathered alongside the way in which, and the precise code I take advantage of to make this work.

Agent throughput broke the human assessment loop

Hallucinations by themselves are a manageable downside, since a really cautious assessment or purposeful evaluation would catch them. Nevertheless, what broke over the previous yr is the quantity that wants reviewing. A single engineer now runs a number of agent periods in parallel, and every of them can produce a thousand-line diff within the time it takes to refill your espresso. Producing code has develop into the quick and straightforward half; reviewing it truthfully is what eats the day. That makes the human reviewer the bottleneck of the entire supply pipeline, and the bottleneck is at all times the following factor it is best to automate.

That bottleneck invitations two acquainted failure modes. Both you approve regardless of the brokers produce with out actually studying it, and the damaged hyperlinks, phantom parameters, and subtly incorrect edge circumstances ship to manufacturing. Otherwise you nonetheless insist on studying each line your self, and your quick and sensible AI brokers work even lower than you do. The primary is negligent, the second wasteful, and most groups oscillate between each relying on the deadline.

The conclusion I hold touchdown on: our assessment course of has to scale with the velocity we’re producing code. The human aspect stays, however the human reviewer’s focus ought to go to the elements that really matter. The grind of checking whether or not hyperlinks resolve, whether or not the brand new parameter exists, whether or not there’s a bug or vulnerability, and whether or not the code adjustments match what was initially requested, all of that has to run routinely on the similar cadence because the coding brokers, even earlier than an individual ever spends a second reviewing the code.

Perception: hallucinations are a relentless, throughput is what modified. Any course of that requires a human to learn each line caps your brokers at human studying velocity, so the very first assessment cross must be automated.

Decide a reviewer that doesn’t share the creator’s blind spots

The apparent first thought to stop these errors is to have the agent assessment its personal work… however additionally it is the weakest one. A hallucination that made it into the diff is, by definition, believable to the mannequin that produced it. Asking the identical weights to seek out errors within the code, particularly throughout the similar already-biased coding session, normally reproduces the precise blind spots that allow the errors by.

This instinct has analysis behind it. Panickssery et al. confirmed at NeurIPS 2024 that LLM evaluators acknowledge and favor their very own generations: the higher a mannequin is at figuring out its personal output, the extra generously it scores it. Observe-up work ties this self-preference bias to familiarity: AI judges rating textual content increased when it reads as predictable to them, and nothing reads as extra predictable to a mannequin than the textual content it simply wrote itself.

Totally different LLM suppliers prepare on completely different knowledge, with completely different recipes and completely different suggestions loops. No supplier manages to get rid of errors, however the form of errors a mannequin makes traces again to the way it was skilled, and that differs per supplier. That decorrelation is strictly what you need in your assessment agent. Within the yr that I’ve used this setup, OpenAI’s Codex has persistently flagged points in Claude-written PRs that Claude’s personal assessment handed with out blinking, from invented filenames to whole code blocks that had nothing to do with the preliminary request, or that launched outright safety violations.

One conclusion you can’t take from that is that Codex is the higher engineer and it is best to hand it the keyboard as an alternative of Claude. Codex misfires pretty repeatedly, too! Simply in a unique route than Claude does. The facility sits in working multimodel: combining suppliers throughout the similar workflow, the place the fashions fill one another’s gaps.

One other incorrect conclusion is that Codex’s evaluations are good and may be accepted blindly. Hallucinations occur in evaluations too, so deal with the assessment as advisory. Difficult the creator (be it Claude or your self) places the concentrate on potential blind spots, which is the right sign to zoom in and mirror. It stays the creator’s job to personal the ultimate high quality, and typically which means writing again to Codex that its assessment was incorrect. Extra on this mechanism later within the put up.

Perception: an creator’s bugs are by definition believable to the creator, and analysis confirms LLMs are positively biased towards their very own output. Evaluation worth comes from decorrelation, by utilizing a mannequin from a unique supplier that fails in another way, and completely different is what catches what you miss.

The pipeline: one LLM name wrapped in bizarre engineering

Conceptually, an LLM reviewer may be written in a single line: “right here is the code diff, please assessment it”. Each OpenAI and Anthropic ship hosted shortcuts for precisely that: OpenAI has a GitHub assessment integration that auto-reviews PRs, and Anthropic has claude-code-action. They’re high quality beginning factors, however I choose to run my very own motion as an alternative, as a result of I wish to personal the immediate, the assessment lifecycle, the merge gating, and the mannequin pin. All these elements matter when your day-to-day turns into managing and reviewing brokers.

So what am I utilizing then, precisely? Two information that dwell within the repository they assessment:

.github/
├── workflows/
│   └── pr.yml            # orchestration: triggers, context, retries, posting
└── actions/
    └── codex-pr-review/
        └── motion.yml    # the reviewer: pinned Codex name + assessment immediate

The runtime circulate is brief. Each time a PR opens or receives a push, pr.yml checks out the code, collects the PR’s full dialog historical past, and fingers each to the composite motion. The motion runs Codex over the diff and returns its findings as a single message. The workflow then posts that message as a single, constantly up to date remark within the PR dialog, and publishes a commit standing that stays purple whereas findings stay unresolved: the go/no-go sign sitting subsequent to the remainder of CI.

On the heart of all of it sits one deceptively easy step, the one line that truly touches an LLM:

- makes use of: openai/codex-action@v1
  with:
    openai-api-key: ${{ inputs.openai-api-key }}
    immediate: |
      # the assessment contract: what to assessment, how one can observe findings,
      # and how one can report again (unpacked within the subsequent part)

The whole lot else exists to run that single step reliably, and to verify its suggestions truly lands on the PR. Right here is the entire workflow:

identify: PR Reviewer

on:
  pull_request:
    branches: [dev] # all work rides characteristic branches: characteristic -> dev -> most important
    varieties: [opened, synchronize, reopened]

# a brand new push cancels the in-flight (costly) assessment of the earlier commit
concurrency:
  group: pr-reviewer-${{ github.occasion.pull_request.quantity }}
  cancel-in-progress: true

jobs:
  codex:
    if: github.occasion.pull_request.consumer.login != 'dependabot[bot]'
    runs-on: ubuntu-latest
    timeout-minutes: 15 # regular assessment takes a couple of minutes, so 15m is greater than sufficient
    permissions: # the token can learn code and put up suggestions, however can by no means push or merge
      contents: learn
      points: write
      pull-requests: write
      statuses: write
    steps:
      # the /head ref updates synchronously on each push;
      # the auto-generated merge ref can race the checkout
      - makes use of: actions/checkout@v7
        with:
          ref: refs/pull/${{ github.occasion.pull_request.quantity }}/head

      # ensure each diff endpoints exist domestically, so Codex can resolve base...head
      - identify: Pre-fetch base and head refs for the PR
        run: |
          git fetch --no-tags origin 
            ${{ github.occasion.pull_request.base.ref }} 
            +refs/pull/${{ github.occasion.pull_request.quantity }}/head

      # suggestions lives on three surfaces: challenge feedback, evaluations, and inline feedback
      - identify: Fetch PR dialog
        id: dialog
        makes use of: actions/github-script@v9
        with:
          result-encoding: string
          script: |
            const prNumber = context.payload.pull_request.quantity;
            const opts = { proprietor: context.repo.proprietor, repo: context.repo.repo };
            const [comments, reviews, inline] = await Promise.all([
              github.paginate(github.rest.issues.listComments,
                { ...opts, issue_number: prNumber }),
              github.paginate(github.rest.pulls.listReviews,
                { ...opts, pull_number: prNumber }),
              github.paginate(github.rest.pulls.listReviewComments,
                { ...opts, pull_number: prNumber }),
            ]);
            const all = [
              ...comments.map(c => ({
                date: c.created_at,
                text: `@${c.user.login}: ${c.body}`,
              })),
              ...reviews.filter(r => r.body).map(r => ({
                date: r.submitted_at,
                text: `@${r.user.login} (review ${r.state}): ${r.body}`,
              })),
              ...inline.map(c => ({
                date: c.created_at,
                text: `@${c.user.login} (on ${c.path}): ${c.body}`,
              })),
            ];
            all.type((a, b) => new Date(a.date) - new Date(b.date));
            return all.map(e => e.textual content).be a part of('n---n');

      # secrets and techniques don't journey with the yml, talk loudly on lacking keys
      - identify: Fail quick on lacking OPENAI_API_KEY
        env:
          OPENAI_API_KEY: ${{ secrets and techniques.OPENAI_API_KEY }}
        run: |
          if [ -z "${OPENAI_API_KEY}" ]; then
            echo "::error::OPENAI_API_KEY secret will not be set for this repository."
            exit 1
          fi

      # retry wrappers can't wrap `makes use of:` steps, so retry manually: transient
      # failures (a 401 from a contemporary runner token, a hiccup reaching OpenAI)
      # mustn't flip the entire run purple
      - identify: Run Codex (try 1)
        id: codex_try1
        makes use of: ./.github/actions/codex-pr-review
        continue-on-error: true
        with:
          openai-api-key: ${{ secrets and techniques.OPENAI_API_KEY }}
          dialog: ${{ steps.dialog.outputs.outcome }}

      # give transient failures a second to clear earlier than retrying
      - identify: Wait earlier than Codex retry
        if: steps.codex_try1.consequence == 'failure'
        run: sleep 20

      - identify: Run Codex (try 2)
        id: codex_try2
        if: steps.codex_try1.consequence == 'failure'
        makes use of: ./.github/actions/codex-pr-review
        with:
          openai-api-key: ${{ secrets and techniques.OPENAI_API_KEY }}
          dialog: ${{ steps.dialog.outputs.outcome }}

      - identify: Put up the assessment and publish a standing
        makes use of: actions/github-script@v9
        env:
          REVIEW: >-
            ${}
        with:
          script: |
            const MARKER = '';
            const opts = { proprietor: context.repo.proprietor, repo: context.repo.repo };
            const prNumber = context.payload.pull_request.quantity;
            const uncooked = course of.env.REVIEW;
            if (!uncooked) return;
            const physique = `${MARKER}n${uncooked}`;

            // one canonical remark, edited in place on each push
            const feedback = await github.paginate(
              github.relaxation.points.listComments, { ...opts, issue_number: prNumber });
            const present = feedback.discover(c =>
              c.consumer?.login === 'github-actions[bot]' && c.physique?.consists of(MARKER));
            const posted = present
              ? await github.relaxation.points.updateComment(
                  { ...opts, comment_id: present.id, physique })
              : await github.relaxation.points.createComment(
                  { ...opts, issue_number: prNumber, physique });

            // go/no-go: a commit standing that stays purple whereas findings are open
            // (a lacking trailer reads as "assessment posted", by no means as zero findings)
            const m = uncooked.match(/`, the place N counts
              the unresolved rows and M what number of of these are severity Excessive.
              At all times embody it, even when N is 0: CI parses it into the
              standing test.

          Pull request title and physique:
          ----
          ${{ github.occasion.pull_request.title }}
          ${{ github.occasion.pull_request.physique }}

          Dialog historical past:
          ----
          ${{ inputs.dialog }}

Three design selections carry the load right here.

The assessment is stateful. The Suggestions Abstract desk tracks each discovering ever raised on the PR, resolved or not, throughout pushes. Mixed with the edit-in-place remark from the workflow, a PR carries precisely one assessment artifact that displays your complete lifecycle, as an alternative of 5 stale feedback that every mirror an replace within the assessment course of. Anybody opening the PR, human or agent, sees the total historical past at a look.

Declines are honored. As a result of Codex reads the entire dialog, the creator can reject a discovering in writing, and the rejection sticks throughout assessment cycles. That is the mechanism that makes an imperfect reviewer workable: false positives value one written reply as an alternative of an infinite nag loop, and the objection stays on file for the following reader.

The decision is machine-readable. The trailing line will get parsed by the workflow right into a Codex assessment commit standing. The result’s two unbiased indicators on each PR: the workflow test tells you the assessment ran, and the standing tells you whether or not something continues to be open. Crimson standing means unresolved findings; inexperienced means every little thing is mounted or explicitly declined. I intentionally hold the standing non-blocking, because it solely features to tell. The choice to merge or not stays open, and stays human.

Notice how this contract asks for greater than typo searching. Codex evaluations the diff in opposition to the PR’s acknowledged intent, ranks each discovering by severity, and attaches a concrete proposed repair, the form of assessment that repeatedly surfaces architecture-level points no linter would ever produce.

Perception: construction beats uncooked mannequin high quality in a reviewer. A lifecycle desk, a decline rule, and a machine-readable verdict flip free-form LLM opinions right into a merge gate with a mechanism for written pushback that makes false positives low-cost.

Shut the loop: the PR turns inexperienced earlier than you even look

The reviewer is just half the story once you take a look at the larger engineering image. The opposite half is educating the creator agent to answer it, so the assessment round-trip doesn’t develop into your job (bear in mind: at all times attempt to take away the bottleneck). In my repos I take advantage of three Claude Code abilities to help me, written as plain markdown instruction information beneath .claude/abilities/:

/pr-open creates the PR: analyzes the department’s commits, runs validation domestically, writes an trustworthy title and physique, and targets the best base. The PR physique issues greater than it appears, for the reason that reviewer reads it because the assertion of intent it evaluations in opposition to. I’ve it spell out the total characteristic request and reference the Linear ticket it implements, so each the reviewer and any human arriving later get the entire image with out leaving the PR.
/pr-iterate handles one assessment spherical: sweeps all three suggestions surfaces, then triages each merchandise right into a repair, a decline with reasoning, or a defer. Fixes are carried out and validated domestically; declines develop into PR feedback explaining why. The reviewer is advisory, and “ok to merge” is a legitimate verdict as soon as the PR meets its acknowledged scope.
/pr-babysit is the watchdog round all of it: watch the CI, repair purple checks, run an iterate spherical when new suggestions lands, and loop till each test is inexperienced and each discovering is mounted or declined. Its terminal state is “merge-ready”, and it stops to report slightly than ping-pong commits when the identical test fails twice on the identical root trigger.

One ordering rule that you just and your brokers ought to at all times be mindful: reply first, push after. The assessment kicks off on each push, and it really works with the context that exists at that actual second. Something not but written down (a decline, a repair abstract, a remark of your personal) doesn’t exist for that assessment spherical, and the reviewer will fortunately re-raise each merchandise it can’t see a response to.

The tip state is a tidy division of labor. Claude writes and defends, Codex challenges and tracks, the loop runs till the PR is inexperienced with a clear suggestions desk. My queue comprises solely PRs which are already merge-ready, every carrying a assessment path I can audit in a single scroll: what was discovered, what was mounted, and what was declined and why.

Perception: automate each side of the assessment. When the creator agent should repair or contest each discovering in writing earlier than it will probably push, the human queue solely ever comprises merge-ready PRs with an auditable assessment path.

What a round-trip appears like in apply

To make the mechanism concrete, right here is one alternate from a backend PR of mine. The PR mounted a capability grid that wrote extra cells than a hardcoded batch cap allowed. The primary assessment spherical got here again with two findings:

## Suggestions Abstract

| Subject                           | Severity | Abstract                                  | Resolved |
| ------------------------------- | -------- | ---------------------------------------- | -------- |
| Truncation past the batch cap | Excessive     | Cells previous the cap are silently dropped. | ❌       |
| Cap duplicates a platform restrict | Medium   | Transfer the cap into app config.            | ❌       |

The Codex assessment standing turned purple. Claude carried out the primary discovering for actual, then declined the second in a PR remark (actual remark, anonymized):

Addressed the Excessive discovering (hard-coded row cap within the knowledge layer, inflicting avoidable 502s).

_fetch_capacity_grid now adapts as an alternative of failing: on a brief web page (which, as a result of the grid is dense at precisely days rows/particular person, unambiguously means the deployed cap truncated the batch) it halves batch_size and retries the identical slice, shrinking till pages match. A brief web page solely turns into a loud 502 as soon as batch_size == 1 [...]

On the secondary suggestion (transfer the cap to config): deliberately not accomplished. The true cap lives within the database’s personal hosted API settings, not an app env var. Mirroring it into app config would add a second supply of reality that would silently drift from the database. Adaptive batching removes the necessity to know the precise worth in any respect, which is the stronger repair. Present capacity-grid DB take a look at nonetheless passes; lint and type-check inexperienced.

Reply first, push after, and have Codex re-review:

## Suggestions Abstract

| Subject                           | Severity | Abstract                          | Resolved |
| ------------------------------- | -------- | -------------------------------- | -------- |
| Truncation past the batch cap | Excessive     | Mounted by way of adaptive batching.     | ✅       |
| Cap duplicates a platform restrict | Medium   | Declined: duplicate would drift. | ✅       |

Each rows resolved, one by a repair and one by a written decline that the reviewer accepted and recorded. That second row is your complete mechanism in a single line: no churn was carried out simply to silence a remark, and the reasoning is now a part of the PR’s historical past for whoever reads it subsequent.

The learnings ledger: what it prices and the place it bites

This setup didn’t come collectively in a single go. Most of its particulars grew out of surprises and useless ends alongside the way in which, so I’m writing the learnings down right here, in tough order of cash saved, to spare you from driving into the identical partitions:

Pin the mannequin. The mannequin enter of codex-action is optionally available, and leaving it empty means “no matter Codex presently defaults to”. I discovered this on my invoice when the default silently moved to a more recent, pricier mannequin (gpt-5.5). Pins resolve this downside, however want upkeep in return; presently I take advantage of gpt-5.3-codex, which is already marked deprecated on the Codex fashions web page on the time of writing (whereas remaining out there on the API). I revisit the pin intentionally as an alternative of letting the default resolve for me.
Cap the trouble at medium. Greater reasoning effort catches marginally extra points for considerably extra time and money. Velocity issues greater than it appears: the assessment sits contained in the agent’s iterate loop, so its latency multiplies throughout each spherical of each PR, and your personal ready time is a part of the worth. Medium has been the issues-per-dollar and issues-per-minute candy spot. Sadly, no effort stage catches every little thing; that’s what the merge-owning human is for.
Retry the fast failures. A contemporary runner token often 401s, and the primary connection to the API often hiccups. Two makes an attempt with a gated fallback hold these from turning the workflow purple “for nothing”, which issues as soon as brokers (and your personal belief) deal with purple as an actual sign.
Cancel outmoded evaluations. The concurrency block means solely the newest commit of a PR will get reviewed. With out it, an agent pushing three fixes in a row buys you three evaluations, two of them for code that now not exists.
Evaluation in opposition to the bottom that can obtain the merge. My PRs goal dev, so the assessment diffs in opposition to dev, and I hold the characteristic department synced with that base. A stale department will get diffed in opposition to a base that has moved on, and the assessment then studies phantom findings for adjustments the PR by no means made.
Characteristic branches are the prerequisite. All of this hangs off pull_request occasions. Direct pushes to dev or most important bypass the reviewer, the standing, and the audit path totally. The one-branch-per-change circulate is what makes each change reviewable within the first place.
Assume PR textual content is untrusted enter. The whole lot the reviewer reads (the PR physique, the dialog, the code itself) is a prompt-injection floor, and the agent executes in your runner along with your API key. The least-privilege token caps the blast radius on the GitHub facet, and codex-action ships sandboxing and safety-strategy knobs for the runner facet. Additionally know that GitHub doesn’t expose secrets and techniques to workflows triggered from forked PRs, so this sample assumes same-repo branches from trusted contributors. Evaluation the belief mannequin earlier than copying it into an open-source repo.

You’re the editor-in-chief now

The purpose of this pipeline was by no means to take away the human from the loop, its intent is to uplevel you. I finished being the primary reader of each diff and have become the one who reads evaluations, adjudicates the occasional standoff between two fashions, and owns the requirements written into the assessment immediate. When Claude and Codex agree, the PR might be high quality. Once they disagree, that disagreement is a precision-guided pointer at precisely the code that deserves my consideration. The place Claude typically acts sloppy, Codex tends to over-engineer, holding that stability in your personal fingers is a superb place to be.

Reviewing the evaluations (so meta) is a greater job than proofreading agent output at agent velocity, and it degrades gracefully. On a lazy day I merge inexperienced PRs on the energy of the path, on a cautious day I learn the suggestions desk and spot-check the declines. Both method, nothing reaches the merge button on one mannequin’s assured say-so.

Key insights from this put up

Hallucinations are a relentless, throughput is what modified. Any course of that requires a human to learn each line caps your brokers at human studying velocity, so the very first assessment cross must be automated.
An creator’s bugs are by definition believable to the creator, and analysis confirms LLMs are positively biased towards their very own output. Evaluation worth comes from decorrelation, by utilizing a mannequin from a unique supplier that fails in another way, and completely different is what catches what you miss.
The mannequin name is the simple half. Reviewer reliability comes from bizarre CI engineering round it.
Construction beats uncooked mannequin high quality in a reviewer. A lifecycle desk, a decline rule, and a machine-readable verdict flip free-form LLM opinions right into a merge gate with a mechanism for written pushback that makes false positives low-cost.
Automate each side of the assessment. When the creator agent should repair or contest each discovering in writing earlier than it will probably push, the human queue solely ever comprises merge-ready PRs with an auditable assessment path.

Ultimate perception: work multimodel. Your brokers will typically hand you assured fiction, and fashions from completely different suppliers fill one another’s gaps, so that you don’t should. A second-provider reviewer with reminiscence and a contract, sitting between your brokers and your most important department is a superb place to begin, however it’s not the one one. At any time when one mannequin’s output is about to matter, a second opinion from a unique lab is the most cost effective insurance coverage you should purchase.

Observe me on LinkedIn for bite-sized AI insights, In direction of Knowledge Science for early entry to new posts, or Medium for the total archive.