The Financial Good thing about Refactoring

As a part of attending to grips with the brand new world of agentic engineering,
I constructed an utility to help my work. It’s a classy app:
high-quality internet UI with dynamic refresh and look-up, modals and
auto-save, integrations to exterior methods, machine studying and textual content
evaluation, background jobs, and a correct atmosphere setup with totally
automated deployment. It’s roughly 150,000 strains of code,
primarily in Rust (~120 kLoC) with the rest in TypeScript and
Terraform.

This was totally written by brokers. Largely Claude Code, and a few use
of Cursor. I didn’t learn or evaluate any of the code, besides
often, out of curiosity.

Whereas constructing the applying, I may see some issues going
awry. After watching an edit to line 4,000 of a file scroll by within the
terminal, I had a more in-depth look. The info entry layer had grown to over
6,000 strains. As extra options landed, this continued to
develop. Each question, learn or write, repeated the identical HTTP request
setup, the identical JSON encoding and decoding. Finally, it reached 17,155
strains. In a single Rust file.

An experiment in refactoring

The 17,155 line file was your entire knowledge entry layer. A single,
self-contained module. Reviewing the code, there was no
de-duplication, no inner language, restricted extraction of capabilities,
and little or no extraction of lessons. It did have a transparent boundary
with an interface to protect. It was an important goal for refactoring.

The aim of refactoring an agentic code base is to spend tokens now in
refactoring to make token consumption for future work decrease. An
experiment ought to be capable to present that as this file was refactored the
token value of creating separate characteristic implementations on this code
base would lower.

Exactly as a result of brokers by no means be taught this was now doable to run as
an experiment. I may immediate a recent agent to make precisely the identical
change after each refactoring stage. In contrast to a human engineer, the
experiment wouldn’t be tainted by studying from earlier steps.

Create an general refactoring plan, following strict refactoring
self-discipline.
Craft a consultant change, described in a single immediate.
Set up a baseline value of change: in a sub-agent, execute that
immediate, together with asking the sub-agent to report token consumption.
Throw away the change.
In a loop:
1. Apply a single step of the general refactoring.
2. In a sub-agent, execute precisely the identical change receiving the
  token value of the change.
3. Throw away the change.
Report all token prices, time to execute the change, and features of
code after every step of the refactoring, together with the baseline.

The immediate used for the consultant change and the refactoring
steps utilized are proven within the appendices, under.

One caveat: Claude doesn’t present dependable strategies for counting
tokens reside regardless of exhibiting token counts, reporting tokens consumed
per session, and billing for tokens. I’m assuming it is a
non permanent problem that can enhance over time. As an alternative, the sub-agent
reported the variety of characters obtained and despatched and used
tiktoken to approximate tokens, by dividing character rely by
4.

Outcomes

Step	Information Entry Layer LoC	Largest file LoC	Complete Rust LoC	Enter tokens per change	Output tokens per change	Time per change (s)
Baseline	17,155	17,155	50,359	159,564	1,705	342
Step 1 (FirestoreClient)	16,706	16,706	49,910	155,205	1,723	530
Step 2 (extract_doc_id, new_link)	16,562	16,562	49,766	159,227	2,105	574
Step 3 (link-query helpers)	16,567	16,567	49,771	154,054	2,105	524
Step 4 (FakeStore predicates)	16,577	16,577	49,781	154,146	2,060	654
Step 5 (worth ctors)	16,469	16,469	49,673	171,251	2,036	1,353
Step 6 (FieldsBuilder)	16,469	16,469	49,673	171,251	2,036	1,353
Step 7 (queries.rs)	16,474	15,670	49,678	151,850	1,800	587
Step 8 (traits.rs)	16,508	13,845	49,712	132,558	1,723	446
Step 9 (traits/ break up)	16,508	13,845	49,712	132,558	1,723	446
Step 10 (codec.rs)	16,521	12,846	49,725	131,871	1,750	540
Step 11 (fake_store.rs)	16,535	11,122	49,739	133,016	2,460	600
Step 12 (retailer/ break up)	16,550	9,269	49,754	104,080	2,050	490
Step 13 (co-locate assessments)	16,550	9,269	49,754	104,080	2,050	490
Step 14 (full fake_store.rs)	16,553	7,225	49,757	107,205	2,453	523
Step 15 (retailer/ break up)	16,608	3,695	49,812	27,360	2,113	454

The attention-grabbing metrics listed here are the full strains of code within the knowledge
entry layer, the full strains of code within the largest single file in
the information entry layer and the enter tokens consumed whereas producing
the change.

This chart exhibits 4 issues. The primary level is the baseline, step 0,
after which the identical metrics are repeated after every refactoring step
has been utilized.

The entire strains of code within the knowledge entry layer as a
complete. Initially, that is simply the only file I began with. This
turns into many recordsdata as refactorings are utilized. By the tip there
are 19 Rust recordsdata.
The strains of code within the single largest file within the knowledge entry
layer. This began because the entirety of the information layer within the single
preliminary file. By the tip, the only largest file is a take a look at
library. Additional refactoring passes may apply the identical strategy
to this.
The entire enter tokens consumed by the sub-agent whereas making use of the
consultant change.
The entire output tokens produced by the sub-agent whereas making use of
the consultant change.

Refactoring reduces token consumption

The outcomes are clear. Enter tokens keep pretty flat till the biggest
file begins to fall, after which they drop earlier than, within the phrases of
Claude, falling off a cliff.

Between the bottom line and the ultimate refactoring, enter tokens for the
identical activity diminished from 159,564 to 27,360. A saving of
132,204 tokens, or 83%. And that saving is just not a one-off. Each single
change that touches the information entry layer from this level ahead now
prices considerably much less.

How a lot of a saving? Assuming Sonnet 5 pricing on the time of writing
of $3/MTok, 39.7 cents. Not quite a bit. Does it multiply? How will this
play out throughout debugging? Extra sophisticated options? That is
refactoring just one portion of the code base, can the entire code base
be aggressively refactored to seek out financial savings in every single place? How a lot would
these refactorings value?

This saving is as a result of the agent has to learn much less code. However it isn’t
as a result of there may be much less code to learn. The general code within the knowledge
entry layer as an entire has stayed pretty fixed. Due to this fact to be
in a position to financial institution this saving, the agent should be capable to efficiently
determine the smallest subset of recordsdata essential to learn. The outcomes
make it seem this was taking place. Studying the Claude Code considering
output and file learn summaries because the change was being utilized additionally
signifies the sub-agent was efficiently studying smaller and smaller
sections of code every time.

In different phrases, randomly chopping the file into smaller recordsdata is
unlikely to assist as a lot: even when every file have been smaller, the agent
can be compelled to learn via many recordsdata in search of the related
code. Whereas the step with the most important impact occurs on the finish, the
earlier steps have been refactorings to arrange this saving. This was not
deliberate. It was merely a results of how refactoring sometimes proceeds:
native file adjustments to extract duplication, earlier than breaking down into
smaller recordsdata as soon as a repeating core emerges.

The refactoring didn’t make the consultant change smaller. The
variety of tokens produced when writing code was largely unaffected:
the output tokens don’t transfer very a lot. These tokens are 5 instances
the value of the enter tokens. However, there are quite a bit much less of them. Are
there refactorings that may very well be utilized to scale back output token
manufacturing? I would like a extra advanced pattern change to discover these
questions. The noise of the non-deterministic code technology course of
is hiding any variance brought on by adjustments within the factoring of the code.

Notes on the method

Claude was not good at refactoring. If you happen to learn the immediate and the
refactoring steps under, it’s clear that the refactorings produced
have been instantly in response to the immediate. Claude is unable to have a look at
code, take a look at refactorings typically and work out that are appropriate
to use: a human must actively information it. This marries with wider
expertise on this app. The event harness consists of an express
refactoring step. That refactoring step didn’t immediate Claude into
enhancing this file. Extra anecdotally, Claude.ai was higher than
Claude Code. I used each interfaces to create the refactoring
plan. Claude Code noticed extract operate as the primary
step. Claude.ai went additional and noticed a whole consumer class to be
extracted.

It was additionally unhealthy at making use of them. The mechanical act of refactoring
was carried out by writing Python scripts utilizing grep and sed. These
scripts steadily acquired confused by indentation. Oh, the irony. In
addition, the only most beneficial refactoring was missed within the first
go, and needed to be re-applied as a follow-up step. This is the reason the
variety of steps within the determine don’t match the refactoring steps within the
appendix.

It took about eight hours to finish your entire experiment. This was
largely unattended. The one intervention was after six hours 40
minutes when it appeared to have completed, however had skipped that step
and wanted to be redirected. This experiment was operating on gradual
resort WiFi. I questioned if that contributed to time taken. However on
deeper evaluation of the code base, the cargo non permanent construct cache had
turn out to be very massive. Check execution was struggling, considerably.

Additional work and broader implications

Sadly, it didn’t happen to me to carry out a rely of the tokens
required to create and execute the refactoring plan till it was
already full. I’ve checked out my combination consumption throughout the
time window the place I used to be doing this work, together with designing and
operating the experiment. I can’t say what number of tokens have been required to
carry out the refactoring. The higher sure is 5 million,
nevertheless. This consists of creating the refactoring plan twice, the work
to design the experiment together with the consultant change, and
varied different duties. Future work ought to embrace a extra correct rely
of tokens consumed to refactor.

This is only one experiment, on a big utility that’s
nonetheless greenfield and constructed and maintained by a single developer. However,
I consider it is a probably attention-grabbing first step. This effort
exhibits the worth, in money and time of refactoring. In addition to
measuring how costly refactoring is. It could be attention-grabbing to
take a look at extra advanced adjustments, at wider refactoring, refactoring
repeatedly, and even the relative worth of various refactoring
approaches.

That is only the start.

Appendices

Be aware: These appendices embrace the prompts that I used, and the
output that was returned. The one modifying utilized has been to take away
the precise code adjustments to be made. These are included with out
modifying to point out how the brokers have been directed. There aren’t any hidden
tips. As such, there may be some language in right here that is likely to be
complicated. The error is within the authentic.

The consultant change

That is the recorded immediate that was fed to every sub-agent, there was
no additional context equipped apart from the code base and accompanying
structure documentation. Each sub-agent was beginning with precisely
the identical info.

You might be working within the Rust mission at ~/dev/your-project-name.

Add a brand new ItemWatchStore public async trait to the Firestore layer, following present patterns precisely. The trait will need to have three strategies:

async fn watch_item(&self, item_id: &str, user_id: &str) -> Outcome<()>

async fn unwatch_item(&self, item_id: &str, user_id: &str) -> Outcome<()>

async fn watched_items_for_user(&self, user_id: &str) -> Outcome>

Watches are saved in a item_watches Firestore assortment. Every doc has fields: itemId (string), userId (string), createdAt (timestamp). There isn’t any Rust struct for a watch report — the strategies return Vec (merchandise ids).

Implement the trait for each FakeStore (utilizing an in-memory Vec<(String, String)> subject added to FakeStoreInner) and FirestoreStore (utilizing the identical HTTP patterns used for different retailer impls on this file).

On the very finish of your response, output precisely this JSON block (fill in actual values):
{
  "files_read": [
    {"path": "src/firestore.rs", "chars": 123456},
    ...
  ],
  "response_chars": 7890
}
Do NOT commit the change. Cease after writing the code.

Refactoring steps

That is the immediate that was used to create the refactoring plan.

Following the strict definition {that a} refactoring is a provably
correctness preserving sequence of code edits, and utilizing Martin
Fowler’s 2nd version of Refactoring because the supply, study
@src/firestore.rs. This can be a 17K LoC Rust file. No file needs to be
that lengthy. It’s nearly actually not utilizing an inner language to
construct and handle queries. Produce and describe, however don’t execute, a
sequence of refactorings that will massively cut back the road rely
of that file, with out altering the interface in any respect.

Following is the outline of the refactorings utilized, extracted
from the plan constructed and adopted by Claude. The precise plan consists of
predicted code adjustments. For every refactoring, the person steps to
observe have been listed. Every of these steps was individually testable, and
was individually examined. This can be a stricter refactoring than most
human engineers would observe.

The steps listed right here don’t line up instantly with the measured adjustments
above as Claude skipped essentially the most useful single refactoring
(splitting out the shop into sub-files) on the primary go and needed to
full that afterwards as two extra steps.

Fowler ref: Extract Class (7.5); Extract Operate (6.1) for
every primitive

FirestoreStore at the moment conflates two tasks:

Area question orchestration — which question to run, which paperwork
to put in writing, parse outcomes into area sorts
Firestore HTTP transport — auth headers, URL development, JSON
encoding/decoding of Firestore wire sorts,
retry-on-PRECONDITION_FAILED

Fowler §7.5 requires extracting a brand new class when you’ll be able to determine a
coherent subset of a category’s knowledge and behavior. The transport
accountability owns: consumer: reqwest::Consumer, project_id: String,
MetadataAuth, and documents_url() / auth_header(). Extract these
into a brand new FirestoreClient struct.

Estimated financial savings: ~1,200 strains in FirestoreStore impls;
FirestoreClient provides ~120 strains web.

Step 2 — Extract Operate: `extract_doc_id` and `new_link` (Fowler §6.1)

Fowler ref: Extract Operate (6.1)

extract_doc_id — The expression
doc.title.rsplit('/').subsequent()?.to_string() seems verbatim on the
begin of all 20 parse_*_document capabilities. Extract it.
new_link — Constructing a Hyperlink struct with metadata: HashMap::new() and provenance: None and a recent UUID seems 62
instances. Extract a manufacturing unit operate.

Estimated financial savings: ~500 strains (62 × ~10-line structs → 62 ×
~2-line calls; 20 parse capabilities every lose 1 line of boilerplate).

Fowler ref: Extract Operate (6.1)

Two sub-patterns recur contained in the FirestoreStore trait impls after operating a hyperlink question:

Sample A — gather all hyperlink paperwork from question rows (~15
websites).
Sample B — question hyperlinks and return precisely one goal ID, error if
lacking (~8 websites):

Estimated financial savings: ~200 strains.

Fowler ref: Extract Operate (6.1)

Contained in the FakeStore impls, ~15 strategies repeat variations of
inside.hyperlinks.iter()....

Extract two strategies on FakeStoreInner. The 15 callsites then turn out to be
single-line. Strategies that moreover filter by a second predicate
(e.g. additionally checking to_kind) chain .into_iter().filter(…) on the
results of the helper.

Estimated financial savings: ~120 strains.

Step 5 — Change Inline Code with Operate Name × 4: Firestore worth constructors

Fowler ref: Change Inline Code with Operate Name (8.5)

Add 4 non-public free capabilities (file-level, not strategies) earlier than the
codec block. Change all 128+ json!({"stringValue": …}) /
json!({"timestampValue": …}) and so forth. inline expressions with calls to
these capabilities. Every multi-word json macro name turns into a single
brief name.

Estimated financial savings: ~80 strains (largely from multi-line json macros collapsing to one-liners).

Fowler ref: Extract Class (7.3)

The ~20 encoder capabilities all observe this form:

let mut fields = serde_json::Map::new();
fields.insert("foo".to_string(), str_val(&x.foo));
fields.insert("bar".to_string(), ts_val(x.bar));
json!({"title": path, "fields": fields})

Extract a small builder. Rewrite every encoder operate to make use of the
builder. A ~40-line encoder shrinks to ~12 strains.

Estimated financial savings: ~500–600 strains throughout the 20 encoder capabilities.