We’re excited to announce LiteRT.js, a JavaScript binding of LiteRT for operating AI immediately inside the net browser. By bringing the trusted on-device inference library LiteRT to the net, internet builders can now run ML and AI fashions with most efficiency fully domestically. This implies enhanced person privateness, zero server prices, and ultra-low latency for real-time experiences. For builders with current .tflite fashions, LiteRT.js makes deployment to cellular and desktop internet browsers smoother than ever, serving as a robust evolution from TensorFlow.js for executing .tflite fashions.
Whereas prior internet AI options like TensorFlow.js relied on much less performant JavaScript-based kernels, we at the moment are making our native, cross-platform runtime with all its optimizations immediately obtainable to internet builders via WebAssembly. LiteRT.js unlocks spectacular efficiency by operating your .tflite fashions immediately within the browser leveraging the state-of-the-art {hardware} acceleration of LiteRT, together with XNNPACK for CPU, ML Drift for GPU, and the upcoming WebNN for NPUs.
Our preliminary launch offers all of the instruments wanted to get began, together with the brand new LiteRT.js npm package deal and a assortment of demos showcasing real-world implementation.
How LiteRT.js advantages internet builders
With LiteRT.js, internet builders can combine fashions into their apps written in JavaScript or TypeScript to deal with advanced duties like textual content era, object detection, and audio processing fully client-side. As LiteRT.js shares a unified cross-platform stack with LiteRT, your internet purposes routinely profit from the newest efficiency upgrades, quantization enhancements, and {hardware} optimizations developed for Android, iOS, and desktop.
By leveraging LiteRT’s decreasing circulation and runtime, you get easy conversion of fashions from a wide range of Python ML frameworks and native {hardware} acceleration throughout all main accelerators (CPU / GPU / NPU). That can assist you unlock these AI capabilities simply, listed below are the principle highlights of LiteRT.js:
1.PyTorch conversion & tailor-made quantization
With LiteRT Torch, PyTorch fashions might be transformed in a single step, making them immediately able to leverage superior browser-based {hardware} acceleration. Get began at the moment by following the LiteRT Torch information.
For additional optimization, AI Edge Quantizer permits you to configure tailor-made quantization schemes throughout totally different mannequin layers. This achieves substantial dimension reductions and efficiency positive aspects whereas preserving total mannequin high quality. Discover the quantization colab to see this in motion.
2.Native {hardware} acceleration throughout CPU, GPU, and NPU
LiteRT.js permits high-performance AI inference for a various number of {hardware} backends.
- CPU: makes use of XNNPACK, Google’s extremely optimized library for on-device CPU acceleration, offering sturdy multi-thread assist and a relaxed SIMD construct for enhanced efficiency.
- GPU: powered by ML Drift, Google’s main answer for on-device GPU acceleration. LiteRT.js leverages WebGPU to allow state-of-the-art GPU acceleration on the internet.
- NPU: harnesses the rising WebNN API (at the moment experimental in Chrome and Edge) to focus on devoted NPUs for power-efficient, extremely low-latency inference.
Able to speed up your internet purposes? Dive into the LiteRT.js documentation to get began.
LiteRT.js Structure Overview
Efficiency and real-world influence
To reveal the real-world influence of the unified runtime and hardware-accelerated backends, we evaluated LiteRT.js in opposition to current internet options. Throughout classical pc imaginative and prescient and audio processing fashions, LiteRT.js delivers important speedups—outperforming different internet runtimes by as much as 3x throughout each CPU and GPU inference.
Notice: Efficiency benchmarks carried out on a 2024 Apple MacBook Professional with M4 Apple Silicon in a managed browser surroundings. Particular person person efficiency could fluctuate primarily based on native GPU capabilities, thermal throttling, and browser driver optimization.
To floor these claims in real-world effectivity, we benchmarked well-liked AI fashions utilizing LiteRT.js throughout three distinct internet execution backends: CPU (through XNNPACK), WebGPU, and WebNN (through Apple CoreML). For demanding real-time purposes like object monitoring, audio transcription, or picture manipulation, leveraging the GPU or NPU through WebGPU or WebNN delivers 5-60x speedup in comparison with normal CPU execution, making certain decrease latency with out compromising efficiency.
Notice: Efficiency benchmarks carried out on a 2024 Apple MacBook Professional with M4 Apple Silicon in a managed browser surroundings. Particular person person efficiency could fluctuate primarily based on native GPU capabilities, thermal throttling, and browser driver optimization.
See it in motion
To see LiteRT.js in motion, discover our stay implementations. LiteRT.js demo supply code is accessible on the LiteRT GitHub repository and through Ultralytics.
LiteRT Ultralytics YOLO integration
Ultralytics is a man-made intelligence firm that focuses on constructing pc imaginative and prescient instruments and fashions. It’s best often called the creator of the YOLO (You Solely Look As soon as) framework, household of real-time object detection and picture segmentation fashions.
We’re excited to share official LiteRT export assist constructed immediately into the Ultralytics Python package deal. Simply deploy Ultralytics YOLO fashions throughout cellular, edge, and browsers—and go from compilation to runtime in just some strains of code.
Demo: YOLO26, household of real-time imaginative and prescient fashions
Depth Estimation
Depth Something – monocular depth estimation showcases the right way to rework an ordinary webcam feed into an interactive 3D level cloud in real-time. Powered by LiteRT.js through WebGPU, it makes use of the Depth-Something-V2 mannequin to immediately calculate depth knowledge and map video pixels right into a responsive 3D area.
Demo: Monocular depth estimation utilizing DepthAnything and WebGPU.
Picture Upscaling
Upscale photographs by 4x within the browser utilizing the Actual-ESRGAN mannequin with LiteRT.js, which works by upscaling 128×128 pixel patches to 512×512 that are then reassembled into the ultimate picture.
Get began with LiteRT.js
Integrating LiteRT.js into your improvement workflow is simple, whether or not you’re launching a recent implementation or migrating an current software to our high-performance runtime. LiteRT.js abstracts the complexities of hardware-level optimization, enabling you to ship responsive, privacy-focused experiences with out the overhead of handbook platform tuning.
The next snippet highlights the streamlined course of for initializing, compiling, and operating a .tflite mannequin with GPU acceleration. Utilizing clear, trendy JavaScript, you’ll be able to load your mannequin, feed enter tensors, and seize high-speed inference ends in real-time. For extra detailed directions, demos, and steering, please confer with our documentation right here.
import { loadLiteRt, loadAndCompile, Tensor } from '@litertjs/core';
await loadLiteRt('path/to/wasm/listing/');
const mannequin = await loadAndCompile('path/to/your/mannequin.tflite',{ accelerator: webgpu });
const inputTypedArray = new Float32Array(1 * 3 * 244 * 244);
const inputTensor = new Tensor(inputTypedArray, [1, 3, 244, 244]);
const outcomes = await mannequin.run(inputTensor);
// outcomes is a Tensor saved on GPU. To maneuver it to CPU & convert to a typedArray we use
const resultArray = (await outcomes[0].moveTo('wasm')).toTypedArray();JavaScript
What’s subsequent
We’re dedicated to repeatedly increasing LiteRT.js efficiency, mannequin protection, and developer tooling. Trying forward, our improvement roadmap facilities on advancing WebNN integration for native NPU efficiency and delivering extremely optimized assist for on-device generative AI.
Acknowledgements
Ultralytics, for offering YOLO26 media and efficiency knowledge. Jason Mayes for LiteRT.js demos.
