Experiences with native fashions for coding

That is the second memo the place I describe my current experiences on operating
small fashions domestically on my developer machine for agentic coding. In
the primary memo, I lined the various components that may affect the viability of that setup — {hardware}, mannequin selection, runtime, harness. Right here I deal with the concrete experiences, the duties I gave the fashions, what occurred, and my remaining conclusions.

Scope

To recap: my focus is on agentic coding particularly, not simply auto-complete. The machines I used have been an M3 Max with 48GB RAM and an M5 Professional with 64GB RAM.

Viability funnel

Evaluating small fashions is kind of tedious to be trustworthy: downloading takes some time (on my non-fibre web connection in the midst of Berlin), then configuring the brand new mannequin in a harness, utilizing it for duties, decoding the outcomes, …

I considered it as working by means of a funnel of accelerating viability:

Does it match into the RAM? If I haven’t got sufficient RAM, I will not even be capable to run
the mannequin, interval. For the overall evaluation of broader viability, let’s assume as a
baseline of 48GB out there RAM.
Does it run at cheap velocity? As soon as a mannequin is operating, the primary smoke take a look at is
to see how rapidly it may reply to a easy request.
Can it deal with software calling? As soon as I noticed that the request velocity was bearable, I
gave it a easy job from a coding harness that concerned studying and altering information, to
see if it may deal with the software calling.
Does it construct functionally right code?
Can it deal with a continued dialog / extra context? As soon as a easy job labored,
I continued the dialog for longer, to see how a lot forwards and backwards it might soak up
phrases of size of the context.
Can it deal with a bigger or extra complicated job? If a setup survives 1-5, the following stage could be to provide it a extra complicated job to resolve and see what occurs.
Is the code high quality acceptable? What’s the stability like between velocity of coding and overview effort?

The journey

It has been fairly a curler coaster journey, which is an remark in itself!

Section 1 – guide evals: I began with choosing a number of duties, after which doing them time and again with completely different fashions and configurations, “manually”. I wished to see what it feels like by way of the person expertise, which is difficult to do with an automatic setup.
Section 2 – automated evals: Nevertheless, after I shared these outcomes with different colleagues in Thoughtworks, I instantly realized new issues from their enter and experiences. So to retry a few of my duties with completely different configurations, I vibe coded a small automated eval setup in any case, which gave me extra knowledge.
Section 3 – day-to-day use: In any case of this, I labored with probably the most promising mannequin (Qwen3.6 35B MoE) for some time on the duties that saved popping up in my each day, to see if I might combine it into my workflows.

Duties

Which job to provide to the fashions makes a big distinction, and is without doubt one of the key challenges of analysis. The principle duties I utilized in my extra systematical comparability have been each JavaScript/Typescript, so not quite a lot of tech stack variety, maintain that in thoughts.

In a much less systematic manner, I additionally wrote a number of shell and Python scripts with it, which labored high quality.

Because the duties are so vital, I am providing you with some particulars right here for 2 of them, to assist inform your interpretation of my outcomes. The selection of job is finally one of many greatest components that determines viability of small, domestically run fashions – it is all about expectations. It is about complexity of the duty (How good is the mannequin at reasoning?), concerning the variety of information we estimate the agent must learn and write (How good is the mannequin at software calling? How massive is the context window?)

Job 1: Kind and cumulate an current bar chart

I wish to change the diagram within the frontend titled, “Messages per
nameless poster”
– It must be “per poster”, not “nameless”
– I would like the bars within the bar chart to be sorted by variety of messages, with highest bar
first on the left
– I would like the x-axis to not present numbers (“#75”), as an alternative I would like it to indicate what
share of the general messages a sure bar is at. E.g., if the primary 10 bars add
as much as 240 messages, and the general messages in that point interval are 1000, then I would like
to see 24% on the x-axis on the tenth bar. I would like this share to be proven each tenth
bar.

Wants some code search, albeit trivial (I give away the lede within the immediate, it must search for a selected chart title)
Wants adjustments to 1-2 information
The cumulation of values on the x-axis was a factor that fashions most steadily struggled with
No pre-existing assessments for these information, no express expectation of assessments so as to add

Section 1: Handbook analysis

I attempted this with Qwen3.6 35B, Gemma 4 31B, Gemma 4 26B and Qwen Coder Subsequent 80B MoE, with each OpenCode and Pi because the harnesses.

Qwen Coder Subsequent did reach a functionally right implementation of this inside these 2.5 minutes on my 64GB M5 Professional. Nevertheless, once I added one other message to the dialog after that, the runtime crashed – so it is succesful, however not realistically runnable.
Qwen3.6 35B and Gemma 4 31B adjusted the sorting with out subject, however then it took me quarter-hour to travel with them concerning the cumulative percentages till that performance labored. Within the spirit of my “funnel” described at first, at this stage I simply wished to deal with performance, and did not even take a look at the standard of the code – if the fashions can not even give me performance proper, the standard would not matter a lot but.
Gemma 4 26B was probably the most profitable right here. It did implement the total set of issues I requested for. However then once I continued the dialog to ask for a refactoring, I received the “textual content wall of doom”… My understanding is that this may be mitigated with activating “presence penalty”, however I’ve not tried that but.

Section 2: Automated analysis

Frustratingly, the automated setup didn’t affirm the guide expertise. Gemma 4 26B didn’t ship a functionally right answer 3/3 occasions, wheras Qwen3 35B MoE succeeded 2/2. The failure was at all times that it did not correctly implement the x-axis labels I requested for. They both weren’t displayed in any respect, or incorrect.

Notice that for the automated analysis, I anticipate the agent to “one-shot” the issue, which isn’t absolutely lifelike. I gave it entry to the browser as the one sensor to self-correct, nevertheless it by no means known as it. Increasing sensors and sensor use might result in self-correction that will make this extra viable in actual utilization.

Job 2: Create a bar chart of nations based mostly on access_log knowledge

I wish to add a horizontal bar chart to our visualisation that reveals which international locations the requests have come from. The title of the chart must be “Nations”. It ought to present the highest 10 international locations, sorted by variety of requests, with the nation with probably the most requests on the prime. All different international locations must be grouped into an “Different” class on the backside of the chart. For entry entries and not using a nation worth, they need to not present up as their very own class, however be lumped into “Different” as nicely.

As a reminder, the entries now appear like this:

{”ts”: “2026-05-24T01:10:41+02:00”, “ip”: “x.x.x.x”, “standing”: 200, “ref”: “https://the-referer.com”, “nation”: “Malta”}

We are going to solely should learn and alter this one file, don’t be concerned about the remainder of the applying. We’re studying a bunch of *.ndjson information with entry log entries such because the one given for instance above, and visualising them

@scripts/visualise_access_logs.mjs

Adjustments to precisely one file, a script that generates a HTML web page
Pre-existing use of D3 for diagrams already within the file
No take a look at setup for this script, no expectation of assessments being written

Section 1: Handbook analysis

I attempted this with Gemma 4 31B and Qwen 35B, and it was surprisingly tough! I noticed my reminiscence utilization improve fairly a bit, very lengthy reasoning chains adopted by excruciatingly sluggish makes an attempt to lastly edit the file. I attempted some completely different variations, like refactoring into smaller information with an enormous mannequin first. I attempted with each OpenCode and Pi, nevertheless it didn’t make a lot of a distinction. My notes say issues like, “Gave up after 11 minutes”, “Gave up after 12 minutes”, “Stopped after 8 minutes”. No thought what was completely different right here in comparison with the opposite duties, it would not appear that sophisticated.

As this job required change to precisely one standalone file, I additionally tried to go old-fashioned right here: I began a plain chat with a mannequin in LM Studio, with none harness use, and pasted the total file in there – that gave me a working answer! Nevertheless, the entire thing took about 6 minutes, and most of that point was spent by the mannequin regurgitating my 450 strains of earlier code, plus the added strains for the chart.

Section 2: Automated analysis

Within the automated setup, I ran this job with Qwen 35B MoE 7 occasions, and it didn’t correctly remedy it 5 out of these 7 occasions… The difficulty within the failures was at all times that there have been no labels displayed on the bars to depict the international locations.

Right here is the place it will get attention-grabbing although: I then ran the evals once more on the M5 machine with 64GB, and it failed solely as soon as! Which was actually shocking to me, as a result of I used to be anticipating the 2 machines to ship completely different speeds, however I didn’t anticipate this huge distinction in high quality of output with the identical mannequin settings. This stays a thriller to me.

Each day use

Job traits

Along with these structured comparisons, I additionally used Qwen3.6 35B MoE recurrently for each day duties, each work and private.

Just a few bash and Python scripts – typically okay
Including new content material entries to my private web site – good
Making small, very nicely outlined adjustments to current codebases – typically okay
Constructing a sport from scratch (planning with Claude Sonnet, then delegating coding execution to the native mannequin) – began nicely, however fell aside for extra complicated logic

These are a few of my reflections on the selection of job, as of now:

Will it require code analysis, or can the immediate level on the particular information to alter? The extra discovery the agent has to do, the extra it requires software calling, and the extra it fills up the context window, and subsequently treasured RAM.
What number of information would possibly it should edit? Even when we are able to level the agent at particular information and skip a number of the code search, the larger these information are, the extra strain on the context window.
How particular are the directions? We have all grow to be used to being much more imprecise with our coding agent directions, as the massive fashions have gotten so succesful. Prompting brokers with native fashions feels a bit like a mixture of the early auto full days, and my first makes an attempt at utilizing Copilot’s “multi-file edit” mode in November 2024…
What’s the tech stack? It is exhausting to inform from this very small knowledge set, however my impression was that I used to be normally rather more profitable with the Bash and Python duties than the JavaScript ones. However there might have been different components at play within the job sort.

Again to fundamentals

In my each day utilization, I’ve really fairly loved the journey again in time, to weaker capabilities. It felt virtually like a detox. A small mannequin has larger sensitivity to adjustments within the setup, so I received extra alerts about what helps and what would not. And the temptation to only try and never perceive the outcomes anymore is quite a bit decrease, I went again to reviewing outcomes rather more – and I imply that in a great way. My expertise with giving sturdy fashions plenty of autonomy, and surrendering to them in a manner, has typically led to fixed rework and surprises at later levels. So I’ve loved that the weaker fashions make me go slower, and take extra care within the second, as an alternative of deferring that comprehension and care to the longer term.

A second perspective

My colleague Jigar Jani works with Qwen 35B MoE (4BIT) recurrently on a 48GB Macbook, and does virtually all of his coding with it, on a “actual world” Python and React codebase. He’s constantly enhancing his harness with abilities, and has discovered Graphify and Perceive Something to be significantly helpful to assist the fashions with code search and understanding. Jigar finds it fairly helpful and sees enhancements as he improves the harness, however he additionally stresses that code overview is tremendous vital.

To sum up…

It has been a irritating expertise with typically complicated outcomes, I subsequently discover that it is positively not a plug-and-play sort of expertise but. However, I really feel like I perceive a bit higher which varieties of duties have the potential to work with these small fashions at this level, and I’m incorporating Qwen 3.6 in my workflows to additional enhance my instinct when to achieve for it.

Total although, the agentic coding capabilities are positively very far-off from what I’ve now grow to be used to with greater fashions.

On the time of publishing, my default setup is:

Mannequin: qwen3.6-35b-a3b, 4BIT quantization (48GB of obtainable RAM are stretching it with that mannequin, I’ve to shut different excessive RAM functions throughout utilization)
Reasoning off (LM Studio: Inference > Settings Customized Fields > Allow Pondering > disable)
Max context window (LM Studio: Load > Context and Offload > Context Size > drag slider to max)
OpenCode or Pi as coding harnesses
I attain for it when I’ve small, simple duties, typically pre-planned by an even bigger mannequin