5 Small Language Fashions for Agentic Device Calling

# Introduction

 
Agentic AI techniques depend upon a mannequin’s capacity to reliably name instruments, choosing the precise operate, formatting arguments appropriately, and integrating outcomes into multi-step workflows. Giant frontier fashions equivalent to ChatGPT, Claude, and Gemini deal with this nicely, however they arrive with tradeoffs in price, latency, and {hardware} necessities that make them impractical for a lot of real-world deployments. Small language fashions have finished nicely to shut that hole, and several other compact, open-weight choices now supply first-class tool-calling help with out the necessity for an information heart to run them.

And now, in no explicit order, listed here are 5 small language fashions for agentic instrument calling. Observe that, for comfort and consistency, all mannequin hyperlinks level to Hugging Face-hosted fashions.

# 1. SmolLM3-3B

SmolLM3 is a 3B parameter language mannequin designed to push the boundaries of small fashions, supporting dual-mode reasoning, 6 languages, and lengthy context. It’s a decoder-only transformer utilizing Grouped Question Consideration (GQA) and No Positional Embeddings (NoPE) (with a 3:1 ratio), pretrained on 11.2T tokens with a staged curriculum of internet, code, math, and reasoning knowledge. Put up-training included a mid-training section on 140 billion reasoning tokens, adopted by supervised fine-tuning and alignment by way of Anchored Choice Optimization (APO), HuggingFace’s off-policy strategy to desire alignment. The mannequin helps two distinct tool-calling interfaces, JSON/XML blobs by way of xml_tools and Python-style operate calls by way of python_tools, making it extremely versatile for agentic pipelines and RAG techniques. As a completely open launch, together with weights, datasets, and coaching code, SmolLM3 is good for chatbots, RAG techniques, and code assistants on constrained {hardware} equivalent to edge gadgets or low-VRAM machines.

# 2. Qwen3-4B-Instruct-2507

Qwen3-4B-Instruct-2507 is an up to date model of the Qwen3-4B non-thinking mode, that includes vital enhancements basically capabilities together with: instruction following, logical reasoning, textual content comprehension, arithmetic, science, coding, and gear utilization. It additionally possesses substantial positive factors in long-tail data protection throughout a number of languages. Each the Instruct and Pondering variants share 4 billion complete parameters (3.6B excluding embeddings) constructed throughout 36 transformer layers, utilizing GQA with 32 question heads and eight key/worth heads, enabling environment friendly reminiscence administration for very lengthy contexts. This particular non-thinking variant is optimized for direct, fast-response use instances, equivalent to delivering concise solutions with out express chain-of-thought traces, making it well-suited for chatbots, buyer help, and tool-calling brokers the place low latency issues. Qwen3 excels in tool-calling capabilities, and Alibaba recommends utilizing the Qwen-Agent framework, which encapsulates tool-calling templates and parsers internally, lowering coding complexity, with help for MCP server configuration recordsdata.

# 3. Phi-3-mini-4k-instruct

Phi-3-Mini-4K-Instruct is a 3.8B parameter, light-weight, state-of-the-art open mannequin educated with the Phi-3 datasets that embrace each artificial knowledge and filtered publicly accessible internet knowledge, with a concentrate on high-quality and reasoning-dense properties. The mannequin underwent a post-training course of incorporating each Supervised Wonderful-Tuning (SFT) and Direct Choice Optimization (DPO) for instruction following and security. Microsoft’s flagship “small however sensible” mannequin, Phi-3-mini was notable at launch for its capacity to run on-device, together with smartphones, whereas rivaling GPT-3.5 in functionality benchmarks. The mannequin is primarily supposed for memory- and compute-constrained environments, latency-bound situations, and duties requiring robust reasoning, particularly math and logic. Whereas older than the opposite fashions on this record and restricted to a 4K context window, the MIT license makes it one of the permissively licensed choices accessible, and its robust basic reasoning has made it a preferred base for fine-tuning in industrial functions.

# 4. Gemma-4-E2B-it

Gemma-4-E2B is a part of Google DeepMind’s Gemma 4 household, which contains a hybrid consideration mechanism, native sliding window consideration with full international consideration. This design delivers the processing pace and low reminiscence footprint of a light-weight mannequin with out sacrificing the deep consciousness required for complicated, long-context duties. The “E” in E2B stands for “efficient” parameters, enabled by a key architectural innovation known as Per-Layer Embeddings (PLE), which provides a devoted conditioning vector at each decoder layer. That is the mechanism which permits the E2B to run in below 1.5 GB of reminiscence with quantization and nonetheless produce priceless outputs. The mannequin helps native operate calling, enabling agentic workflows, and is optimized for on-device deployment on cell and IoT gadgets, able to dealing with textual content, picture, audio, and video inputs. Launched below Apache 2.0 (a change from earlier Gemma generations’ extra restrictive customized license), Gemma 4 E2B is a pretty possibility for builders constructing multimodal agentic functions working fully on the edge.

# 5. Mistral-7B-Instruct-v0.3

Mistral-7B-Instruct-v0.3 is an instruct fine-tuned model of Mistral-7B-v0.3, which launched three key adjustments over v0.2: an prolonged vocabulary to 32,768 tokens, help for the v3 tokenizer, and help for operate calling. The mannequin employs grouped-query consideration for sooner inference and Sliding Window Consideration (SWA) to deal with lengthy sequences effectively, and performance calling help is made doable via the prolonged vocabulary together with devoted tokens for TOOL_CALLS, AVAILABLE_TOOLS, and TOOL_RESULTS. As the most important mannequin on this roundup at 7B parameters, Mistral-7B-Instruct-v0.3 provides the perfect basic instruction-following efficiency of the group and has turn out to be an industry-standard workhorse, broadly accessible via Ollama, vLLM, and most inference platforms.

# Wrapping Up

 
The 5 fashions lined right here — SmolLM3-3B, Qwen3-4B-Instruct-2507, Phi-3-mini-4k-instruct, Gemma-4-E2B-it, and Mistral-7B-Instruct-v0.3 — span a variety of architectures, parameter counts, context home windows, and launch dates, however share one necessary trait: all of them help structured instrument calling in a compact, open-weight package deal.

From Hugging Face’s totally clear SmolLM3 to Google DeepMind’s multimodal edge-optimized Gemma 4 E2B, the choice demonstrates that succesful agentic fashions now not require large infrastructure and frontier fashions to deploy. Whether or not your precedence is on-device inference, long-context dealing with, multilingual protection, or essentially the most permissive license doable, there’s a mannequin on this record price exploring.

Needless to say these aren’t the one small language fashions with tool-calling capabilities. They do, nevertheless, do a superb job representing these with which I’ve direct expertise, and which I really feel snug together with based mostly on my outcomes.
 
 

Matthew Mayo (@mattmayo13) holds a grasp’s diploma in laptop science and a graduate diploma in knowledge mining. As managing editor of KDnuggets & Statology, and contributing editor at Machine Studying Mastery, Matthew goals to make complicated knowledge science ideas accessible. His skilled pursuits embrace pure language processing, language fashions, machine studying algorithms, and exploring rising AI. He’s pushed by a mission to democratize data within the knowledge science group. Matthew has been coding since he was 6 years previous.