Till lately I hadn’t tried operating fashions regionally for fairly some time, the frustration
had simply all the time been too excessive once I did it. A few month in the past although, I dove again in – there
had been simply too many claims on the market to disregard, about how far they’ve come, the way it’s now a lot
extra possible to run them, and the way a few of them have change into actually good at coding. So that is
my private expertise with utilizing them, on and off, over the previous 4 weeks or so.
On this memo, I’ll begin with a extra normal introduction and undergo the elements that affect the
viability of those fashions for coding. In a follow-up memo, I’ll describe my precise experiences in additional element.
Scope
My most important curiosity is how helpful they’re for coding, and never simply auto full, however
agentic coding. Secondly, I am concerned about how prepared and usable they’re extra
broadly, for builders who do not need to dive right into a bunch of specs and additional tooling and
tweaks to make it work.
By way of {hardware}, I’ve been operating fashions on these two machines:
- Apple M3 Max, 48GB RAM
- Apple M5 Professional, 64GB RAM
Components that affect viability
There are a myriad of things at play that may affect the outcomes, which makes it fairly
a tedious enterprise to judge which setup works greatest below the useful resource constraints we
have. It additionally makes it very laborious to filter the alerts from the noise when folks share their
successes with these fashions on-line.
I discovered it significantly baffling that within the automated eval setup, one mannequin clearly delivered higher outcomes on the stronger machine (not simply velocity, however higher code!), regardless of all different settings being the identical.
I will begin with a abstract, earlier than going into particulars of every of those elements.
prefills
(system immediate, instruments, …)
various instrument schemas affect…
offers templates to enhance
higher {hardware} match results in higher…
differ in optimisation for…
Determine 1: The various elements that may
influence the outcomes – faucet or hover over containers to focus on arrows.
- Runnability: RAM is the core constraint. I used fashions between 15-25GB, context
home windows of most 64K, harnesses OpenCode and Pi with zero Expertise and MCP servers lively) - Response velocity: Is impacted by many elements, however was fairly good for some fashions, and leaps and bounds from the place it was a yr in the past. Setups: LM Studio + 4BIT quantization + M3 Max/M5 Professional + GGUF/MLX each
- Viability for agentic coding: Device calling was tough nonetheless, the fashions usually failed, however can often self-recover from their failures. This can be a key part of agentic coding particularly. With out it, you’ll be able to nonetheless go the old style manner of copying and pasting from a chat window in fact. And small fashions are undoubtedly much more viable for auto full than agentic use.
- High quality of outcomes: Is dependent upon the duty (extra on that within the subsequent memo), however clearly nowhere close to the kind of functionality we are able to get from the massive fashions. Total it was very hit or miss. I solely checked out correctness of performance, did not go into code high quality.
RAM
The mannequin weights should match into the accessible RAM, or extra particularly, VRAM. In the event that they
do not, the runtime both crashes (occurred to me as soon as!) or drops to unusably gradual
speeds.
On Apple Silicon, nearly all RAM is accessible to the GPU with no separate VRAM
restrict,
this may be totally different in different machine configurations.
Impacts: Mannequin runnability; Pace of responses
My expertise: On the machine with 48GB, I ran fashions between 8 and nearly 30GB.
The 30GB stretches it a variety of course, particularly when the context window will get added,
the 15-25GB dimension is extra comfy and I haven’t got to shut fairly as many different
purposes. On the machine with 64GB, I as soon as ran a mannequin that was 48GB huge – which
went nice at first, however then it rapidly crashed…
Processing energy
Extra cores typically means quicker token era, however structure issues too, and
newer chip generations can shut the hole even with fewer cores. This can be a robust one to
examine between machines with out diving a lot deeper into the main points of every
configuration
and structure.
Impacts: Pace of responses
My expertise: On each the M3 Max and the M5 Professional, all fashions I ran had fairly an
spectacular velocity, in comparison with the place they had been a yr in the past. Pace degrades although the longer a dialog will get. I’m okay utilizing them on the present velocity for some duties – if the standard of the output had been acceptable.
Reminiscence bandwidth
Reminiscence bandwidth is a bottleneck for token era, figuring out how briskly knowledge strikes
between RAM and the compute items.
Impacts: Pace of responses
My expertise: Each the M3 Max and the M5 Professional I used have a virtually equivalent
bandwidth of ~300 GB/s, so I do not actually have a comparability to one thing else. However as
talked about earlier than, the velocity of all fashions I attempted was fairly acceptable.
Variety of parameters
The parameter rely principally represents the scale of a mannequin’s discovered information and
capabilities. Extra parameters typically imply higher output high quality, but additionally a bigger
file
dimension.
Impacts: Quantity of RAM wanted; High quality of outcomes
My expertise: With 48GB, I used fashions round 30B parameters, +/- 5B. The
greatest mannequin I loaded on the 64GB machine was Qwen3 Coder Subsequent 80B (MoE), which solved
the duty I gave it so much higher than the smaller fashions – however then crashed after I
continued the dialog.
Reasoning capabilities
Reasoning fashions undergo a “chain of thought” course of earlier than responding, which helps
with complicated multi-step duties, however may also generate considerably extra tokens and gradual
responses down.
Impacts: Complexity of the duties; Pace of response; Context window dimension (and
subsequently want for RAM)
My expertise: The entire fashions I attempted had reasoning capabilities, and so they had been switched on by default all through most of my experiments. Nonetheless, I
usually seen them entering into infinite circles within the reasoning chain, particularly
the smaller ones. (“Wait, …”, “Really, …”, “However wait, …”) So I additionally did a number of runs of my automated setup with reasoning off – and lo and behold, it isn’t solely quicker (which was to be anticipated), but additionally carried out the identical to barely higher! A superb reminder that reasoning will not be all the time needed, and might generally even be counterproductive.
Device calling capabilities
For agentic use, a mannequin should be capable of reliably emit structured instrument name syntax that
matches the schema the harness expects. Fashions that weren’t particularly skilled or
fine-tuned for instrument calling usually produce malformed calls.
Impacts: Capability to make use of agentic harnesses
My expertise: This was a typical difficulty with the fashions I attempted, although they
may usually self-correct and get well from a failed instrument name (e.g. utilizing improper parameter
names like file.path as a substitute of filePath).
Format
GGUF is the usual format for llama.cpp-based runtimes like LM Studio and Ollama, and
has by far the most important mannequin library. MLX is Apple’s personal framework constructed particularly
for
Apple Silicon and might be quicker, however fewer MLX-formatted fashions can be found on the
second.
Impacts: Pace of responses
My expertise: I attempted each codecs for one or two of the fashions, however I
personally did not really feel a lot of a distinction. That may very well be because of the
unstructured sort of analysis I did – alternatively, as any consumer expertise
researcher would inform us, the
perceived velocity is in the end what issues, not what the clock says…
Quantization
Quantization compresses mannequin weights to cut back the file dimension, buying and selling some high quality for
a
a lot smaller RAM footprint. The extent of quantization is often marked in mannequin names
and
descriptions as This fall / Q6 / Q8, or 4BIT / 6BIT / 8BIT, with decrease numbers that means larger
compression. The most recent buzz lately have been QAT (Quantization-Conscious Coaching)
variants
of fashions, that are skilled with quantization simulated in the course of the coaching. They’re
alleged to protect high quality higher than customary quantization.
Impacts: Quantity of RAM wanted; Pace of responses; High quality of responses
My expertise: The entire fashions I downloaded had been at This fall / 4BIT, I did not get
round to making an attempt totally different variations but. I additionally have not gotten round to making an attempt a QAT
one.
Structure
MoE (Combination of Specialists) fashions have a big whole parameter rely however solely
activate a subsection of their weights at inference time, so a 35B MoE mannequin wants
considerably much less RAM and might run quicker than a 35B dense mannequin.
Impacts: Quantity of RAM wanted; Pace of responses
My expertise: The Qwen3.6 35B MoE mannequin was by far giving me the most effective stability between variety of parameters and RAM utilization, and subsequently runnability and high quality of outcomes. This may very well be because of the MoE structure, I am undecided. The structure may also clarify my expertise of getting higher coding skills out of the mannequin on the 64GB machine than the 48GB – it is perhaps loading extra specialists there? I am undecided if that is true, however it’s the one smart speculation I’ve up to now.
Context window dimension setting
Context window dimension consumes RAM on prime of mannequin weights by way of the KV cache, which
grows with context size. The default dimension configured within the runtimes is much too small
for agentic coding, it needs to be set to no less than 32K, if not 64K.
Impacts: Dimension and complexity of duties; Quantity of RAM wanted; Pace of responses;
Capability to make use of reasoning
My expertise: I attempted to see how little I may get away with. For small duties
I may generally work with 32K, however usually I needed to enhance to 64K, in order that appears to be
default minimal. Because the fashions themselves had been already pushing the boundaries of
my accessible RAM, I am undecided how how far more I may nonetheless enhance it, even on the
64GB machine… So whereas many of those fashions in principle help a bigger window,
really utilizing it’s restricted by the reminiscence constraints.
Checklist of fashions used
The excitement for coding has been throughout Qwen3 and Gemma 4, so these had been those I
went
for.
Qwen3
- Qwen3.6 35B-A3B MoE This fall GGUF (22 GB)
- Qwen3.6 Coder Subsequent 80B MoE GGUF (45 GB)
Gemma 4
- Gemma 4 12B This fall GGUF (7.5 GB)
- Gemma 4 26B 4BIT MLX (15.6 GB)
- Gemma 4 31B 4BIT MLX (29 GB)
Runtime
The runtime handles mannequin discovery, configuration, and loading. It additionally determines the
sensible query of how we combine our harness with the mannequin. Often this occurs by
beginning an internet server that gives a spread of typical APIs that the harnesses help,
and the localhost URL of that internet server is then configured as a mannequin
supplier within the harness. Most generally supported by harnesses is the OpenAI API, however Claude
Code e.g. expects Anthropic’s
Claude API .
Impacts: Ease of configuration and discoverability; ease of integration with
harnesses; velocity of responses
Determine 2: The “Developer” view in LM Studio, displaying a
operating server and lots of the elements described up to now (supplier url, mannequin dimension, APIs,
context window dimension configuration)
My expertise: Whereas I’ve used different runtimes up to now, I’m at present again
utilizing LM Studio, primarily for its consumer expertise. There are many ins and outs of which runtime is probably the most optimised for which {hardware} and what sort of fashions, to get much more velocity out of it. However pondering again to the broader viability of operating native fashions, consumer expertise performs an enormous function
for that. For what it is price, probably the most incessantly talked about various from my colleagues was oMLX.
Harness (Claude Code, OpenCode, Pi, …)
Coding harnesses can fluctuate considerably in how a lot overhead they inject into the context
window (system immediate, variety of instruments), which turns into extra of an issue regionally the place
we’re
so useful resource constrained. Our personal expanded harness round that additionally makes a distinction,
e.g.
what number of expertise or MCP servers are lively. An outline of every of them will probably be despatched to
the
mannequin, and once more, take up area within the context window.
I discussed above that small fashions nonetheless wrestle with instrument calling – and it most likely
does not assist that every harness has barely totally different schemas for the essential instruments. Let’s
take modifying a file for example:
- Pi:
old_textandnew_text(see
right here) - OpenCode:
oldStringandnewString(see
right here) - Claude Code:
old_stringandnew_string(no less than that is what
it says once I requested it)
Lastly, not all harnesses simply help the combination of native fashions. Open supply
instruments are often the go-to, however Claude Code will also be pointed at native suppliers. GitHub
Copilot appears to help it for his or her CLI, and I feel it is doable in Cursor as effectively
to override the OpenAI base URL and level it to localhost.
Impacts: Dimension of context window wanted; Device calling success; Integratability
My expertise: In my makes an attempt, I used OpenCode and Pi. I averted Claude Code because it
apparently would burden the context
window fairly a bit.
The main points
In second memo, I’m diving extra deeply into the kinds of duties I gave the fashions, and what I skilled.
A preview of my total conclusions: Utilizing small fashions like that is nonetheless fairly messy and laborious to judge. It was a irritating expertise to return to conclusions, because the outcomes rely upon so many issues. I might subsequently say that it is nonetheless not prepared for a easy “plug and play” expertise for builders who do not need to spend an excessive amount of time on it.
Nonetheless, primarily based on this expertise I do have a go-to mannequin that I am utilizing regionally now, which is Qwen3.6 35B MoE. It provided the most effective stability of functionality, velocity and RAM footprint amongst what I attempted.
