Google Introduces LiteRT Tensor API for On-Device Model Development

A C++ Path to Portable On-Device ML Graphs

Google has introduced the LiteRT Tensor API, a lightweight C++ interface aimed at developers building and refining machine-learning workloads for local execution. The API is positioned for cases where developers need direct control over a model’s computation graph, its optimizations and its use of hardware accelerators, while retaining the ability to run the resulting graph through LiteRT on CPUs, GPUs and NPUs.

The release addresses a familiar trade-off in native ML development. Writing computations directly for a particular graphics or accelerator backend can offer detailed control, but the implementation may be tied to technologies such as Metal, OpenCL or Vulkan. Supporting another hardware target can then require another implementation. Google also points to the engineering involved in coordinating buffers and synchronization across processors, graphics hardware and neural-processing hardware, as well as joining pre-processing, inference and post-processing steps.

With the Tensor API, developers construct graphs one operation at a time in C++. This gives the graph definition a central role: the same C++ graph can be used across supported LiteRT targets rather than being rewritten for every device backend. The API also supports loading model weights directly from checkpoints, including Hugging Face .safetensors files. Its zero-copy buffer interoperability is intended to avoid unnecessary movement of accelerator buffers between stages of an application pipeline.

Google says the API can also work with AI coding agents that generate C++ to create, alter, check and benchmark graphs. As an example, the company used a coding agent to build an application for multi-object segmentation and tracking based on SAM 2. The project was then run with the same C++ code on GPUs in a macOS laptop, an iOS phone and an Android phone, using Metal on the Apple platforms and OpenCL on Android.

In Google’s MacBook Pro comparison, the SAM 2 workload, covering the image encoder and mask decoder in fp16, was reported at roughly 1.5 times the speed of Apple MLX. Google described graph-level changes made by the agent, including moving per-frame visibility decisions into the graph and replacing WebGPU operations that fell back to the CPU with GPU-compatible arithmetic.

The announcement nevertheless does not frame the C++ route as the default starting point for every deployment. Arm AI director Laurence Moroney advised beginning with model conversion, reserving the C++ API for needs such as graph-level control or keeping pre- and post-processing on an accelerator. He highlighted a WebGPU example that moved from below 10 frames per second to above 30 frames per second at 512 by 512, with an agent carrying out iterations.

Google lists further work in progress, including ML Drift backend support, more NPU-oriented operations, automatic full-graph quantization and broader dynamic-shape support for image dimensions, batch sizes and sequence lengths.