Think about constructing a totally autonomous robotic that may see, hear, and react to its surroundings in actual time, fully offline on a single compact gadget like Raspberry Pi. Edge AI unlocks this precise autonomy. It permits builders to construct extremely safe and self-contained programs like clever robotics and native AI brokers with zero cloud dependencies, ultra-low latency, and complete knowledge privateness.
We’ve made operating edge AI on the Raspberry Pi a breeze with Google AI Edge’s LiteRT, high-performance production-proven on-device inference runtime. LiteRT lets you seamlessly deploy the whole lot from classical ML fashions to state-of-the-art LLMs proper out of the field throughout platforms. By delivering optimized execution and hyper-efficient reminiscence utilization on each CPU and GPU, LiteRT maximizes your Raspberry Pi’s full computing potential.
Reachy Mini reacts with responses and motion, powered by Gemma and LiteRT on Raspberry Pi 5.
This hardware-software synergy shines brightest when paired with Gemma, Google’s household of light-weight open fashions. To indicate you what’s doable, we’ll reveal how Gemma and LiteRT on a Raspberry Pi 5 can energy the Reachy Mini robotic to understand and react to its surroundings solely domestically in actual time. Learn on to get began with your individual deployment.
Uncover the Agentic Functionality of Gemma
Gemma fashions are nicely fitted to constructing autonomous brokers, good cameras, and social robotics that may cause and execute advanced, multi-step workflows immediately in your Raspberry Pi. To accommodate completely different {hardware} constraints, the Gemma household of fashions gives a number of extremely environment friendly choices:
- Gemma 3 270M: A hyper-efficient and compact base mannequin designed for task-specific submit fine-tuning, enabling high-speed, low-latency options like sentiment evaluation or entity extraction in resource-constrained environments.
- EmbeddingGemma 300M: A textual content embedding mannequin that produces high-quality embeddings on-device, nice for Retrieval Augmented Technology (RAG), semantic search, and classification.
- Gemma 3 1B: A light-weight and multilingual text-only mannequin that balances compact measurement with robust generative capabilities, making it excellent for a variety of on-device duties, equivalent to summarization and content material creation.
- Gemma 4 E2B: Tailor-made particularly for cellular and tight edge environments, it options memory-mapped per-layer embeddings, and is good for steady monitoring, quick textual content/picture/audio inference, and edge-based speech processing the place saving RAM is completely essential.
- Gemma 4 E4B: The candy spot for efficiency and measurement. This mannequin delivers noticeably stronger reasoning capabilities and frontier-level edge efficiency whereas remaining compact. It’s the excellent alternative for advanced multi-step planning with out overwhelming the Pi’s assets.
Gemma Efficiency on Raspberry Pi CPU
By means of LiteRT-LM, a specialised orchestration layer on prime of LiteRT, builders can seamlessly deploy Gemma proper out of the field. Beneath the hood, subtle CPU acceleration by way of LiteRT and XNNPACK ensures the Gemma household of fashions is very optimized for useful resource effectivity and low-latency execution immediately on the Raspberry Pi.
On a Raspberry Pi 5, LiteRT-LM delivers a strong efficiency for Gemma 4 E2B, attaining 99 tokens/sec for prefill and 9 tokens/sec for decode, all whereas sustaining a remarkably low peak reminiscence footprint of simply 1432 MB. This brings Gemma’s extremely responsive, general-purpose intelligence to Raspberry Pi.
Due to Gemma 4 E2B’s extremely environment friendly tokenizer, which packs extra textual content into fewer tokens (averaging ~4.2 characters per token), LiteRT-LM achieves a powerful end-to-end technology velocity of ~27.3 characters per sec, roughly 300 phrases per minute (wpm), within the Reachy Mini voice demo. This throughput makes Gemma 4 E2B glorious for real-time speech and translation duties, delivering textual content at twice the velocity of regular human speech (~150 wpm).
Discover extra ready-to-use open fashions to run on Raspberry Pi from the LiteRT Hugging Face Group.
Execute on Raspberry Pi GPU with LiteRT
On the Raspberry Pi 5, the quad-core ARM Cortex-A76 CPU is a uncooked computing powerhouse, delivering ~153.6 GFLOPS (FP32) and as much as ~2.0 TOPS (INT8). Compared, the built-in Broadcom VideoCore VII GPU is clocked at 800 MHz and affords a peak of ~76.8 GFLOPS (FP32) and ~0.24 TOPS (INT8).
Whereas the CPU possesses an enormous capability benefit, the GPU introduces heterogeneous parallel execution, a paradigm essential for real-time edge purposes. Somewhat than saturating the CPU, builders can delegate duties throughout each processors to optimize general system and thermal effectivity. For instance, by offloading steady imaginative and prescient or audio fashions to the VideoCore VII GPU, it preserves high-priority CPU cycles for general system monitoring, pipeline orchestration, or computationally demanding LLM inference.
As such, we’ve got enabled GPU inference on the Raspberry Pi with LiteRT’s WebGPU (Vulkan) backend by way of ML Drift. This integration lets you run a variety of pc imaginative and prescient, audio, and embedding fashions immediately from the LiteRT Hugging Face Group, together with seamless help for in style MediaPipe fashions, Ultralytics YOLO fashions, Moonshine, and far more.
Actual-time object detection with the Ultralytics YOLO26n mannequin operating on Raspberry Pi 5 with LiteRT. Get began with the pattern code from YOLO information.
The desk beneath demonstrates the CPU and GPU latency of operating basic pc imaginative and prescient and audio fashions by way of LiteRT:
Deep Dive: Reachy Mini Pipeline Powered by LiteRT
The Reachy Mini pipeline is a strong showcase of low-latency, real-time edge AI inference operating solely on the Raspberry Pi 5. By leveraging LiteRT, the system splits intensive imaginative and prescient and language workloads right into a concurrent, dual-processing structure throughout CPU and GPU.
Right here is how the parallel structure works underneath the hood to make sure seamless interactions:
- Object Detection (Ultralytics YOLO on GPU): Digital camera frames are streamed to the Pi, the place a Ultralytics YOLO detection runs constantly on the GPU, avoiding useful resource rivalry and frees up the CPU.
- Speech Recognition (Moonshine on CPU): When the person speaks, the ASR part transcribes the audio into textual content immediately on the CPU.
- Reasoning & Motion (Gemma 4 E2B on CPU): The Gemma 4 E2B mannequin processes the ensuing transcript alongside the newest visible metadata to generate low-latency, streaming responses, equivalent to speech replies and bodily robotic gestures.
- Textual content-to-Speech (TTS on CPU): The TTS part synthesizes the generated textual content into audio in streaming. The system streams the synthesized voice again to the Reachy Mini robotic.
See the complete supply code of the Reachy demo within the LiteRT Samples Github repo.
Agentic Coding with LiteRT on Raspberry Pi
LiteRT gives a complete suite of instruments that covers the complete improvement cycle: conversion, quantization, benchmark, and inference. For a quick, frictionless setup, probably the most easy method is utilizing the LiteRT CLI device. Somewhat than requiring builders or coding brokers to manually handle a number of impartial libraries, the LiteRT CLI aggregates core edge workflows right into a single, unified command set.
Streamline your improvement cycle with LiteRT CLI: convert, quantize, benchmark, and inference.
Now you can supercharge your improvement cycle by including the LiteRT CLI ability and different superior LiteRT expertise into your AI coding agent, equivalent to Google Antigravity. This empowers brokers to autonomously orchestrate and execute advanced, multi-stage machine studying workflows in your behalf. For instance, you possibly can simply construct your individual voice translator fully offline on a Raspberry Pi, just like the Gemma Translator proven beneath.
Discover the entire implementation particulars within the Gemma Translator GitHub repo.
For resource-constrained IoT gadgets, minimizing storage and reminiscence overhead is essential. With out particular optimization, generic AI runtimes typically bundle heavy desktop or server dependencies. In distinction, LiteRT is engineered particularly for on-device deployment, sustaining an exceptionally lean and modular distribution.
The desk beneath compares the obtain footprint required to run LLM inference on a Raspberry Pi (ARM64 Linux).
Operating Your First Mannequin
You possibly can set up the LiteRT CLI and run your first mannequin on a Raspberry Pi 5 with just some easy instructions.
1.Set up LiteRT CLI
To get began, set up the LiteRT CLI by way of pip (ideally inside a digital surroundings):
pip set up litert-cliShell
2. Run the mannequin
Obtain and run any appropriate mannequin immediately from the LiteRT Hugging Face Group. The code snippet beneath demonstrates learn how to execute Gemma 4 E2B (e.g. gemma-4-E2B-it-litert-lm) on the Raspberry Pi 5.
Run the mannequin by offering your Hugging Face authentication token:
export HUGGING_FACE_HUB_TOKEN=
litert lm run
--from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm
gemma-4-E2B-it.litertlm
--attachment=picture.jpg
--prompt="You're Reachy Mini. Establish the primary object in entrance of you, "
"state its location (Left/Proper/Middle), and counsel head motion in "
"10 phrases or much less." Shell
What’s Subsequent
We’re excited to share that LiteRT integration and Gemma fashions are coming quickly to Hailo AI accelerators! This replace will will let you seamlessly offload mannequin inference to the Raspberry Pi AI HAT+ and AI HAT+ 2, delivering huge {hardware} acceleration advantages by the very same, acquainted LiteRT workflows you employ at the moment.
Discover our assets and begin your journey with LiteRT:
We worth your enter. Please share your ideas, suggestions, or characteristic requests by opening a difficulty on our GitHub Challenge Tracker. Share your cool Raspberry Pi + LiteRT + Gemma tasks with @googlegemma. We won’t wait to see what you construct!
Acknowledgements
Google: Changming Solar, Chintan Parikh, Cormac Brick, Dillon Sharlet, Daisuke Majima, Erin Walsh, Frank Barchard, Glenn Cameron, Ian Ballantyne, Jingjiang Li, Jun Jiang, Kimish Patel, Lu Wang, Matthias Grundmann, Rodney Witcher, Sachin Kotwani, Sasha Denisov, Scott Loftin, Shuangfeng Li, Somdatta Banerjee, Terry (Woncheol) Heo, Volodymyr Kysenko, Weiyi Wang, Yi-Chun Kuo, Yu-hui Chen, and gtech staff
Raspberry Pi & Hailo: Ashley Whittaker, Eldad Rubinstein, José MarÃa Casanova (Igalia), Naushir Patuck, and Sarah Cunningham
Ultralytics: Francesco Mattioli, Onuralp Sezer, Lakshantha Dissanayake
Moonshine AI: Pete Warden
