Google has announced a major upgrade to its LiteRT framework, a modern on-device inference engine that evolved from TensorFlow Lite. The new version introduces sophisticated hardware acceleration capabilities, making it faster and more versatile for a wide range of devices. This move aims to improve performance for apps that rely on AI models running locally on smartphones, tablets, and other edge devices.

Enhanced GPU Performance with ML Drift

One of the key updates is the integration of a next-generation GPU engine called ML Drift. Google claims that LiteRT now delivers 1.4 times faster GPU performance compared to its previous version. This means AI models can run more quickly, reducing latency and improving user experience. The ML Drift engine supports multiple graphics APIs, including OpenCL, OpenGL, Metal, and WebGPU. This broad compatibility allows developers to deploy models seamlessly across mobile, desktop, and web platforms.

Google highlighted that the new acceleration capabilities were previewed last May and are now ready for wider use. LiteRT is available on GitHub and is powering many popular applications that require low latency and high privacy. The improvements are especially important for generative AI models, which demand significant processing power but need to run efficiently on edge devices.

Unified Workflow for GPU and NPU Acceleration

Beyond GPU speedups, LiteRT now offers a simplified and unified workflow for deploying models across different hardware types, including NPUs (Neural Processing Units). This means developers no longer need to juggle multiple vendor-specific SDKs or worry about device fragmentation caused by various SoC (system on chip) designs. LiteRT handles these complexities internally, making it easier to get models running optimally on diverse hardware.

For Android devices, LiteRT intelligently prioritizes GPU acceleration when available for maximum performance. If a device lacks the necessary GPU support, the framework automatically falls back to OpenGL, ensuring broader compatibility without sacrificing functionality. This automatic adaptation helps developers reach a wider audience without needing to optimize for specific hardware configurations.

Google also emphasized that LiteRT supports seamless conversion for popular AI frameworks like PyTorch and JAX. This cross-platform support enables developers to deploy their models easily, whether they are working on mobile apps, desktop applications, or web-based projects. The goal is to make on-device AI deployment faster, easier, and more reliable across all types of devices.

Overall, the new features in LiteRT mark a significant step forward in on-device AI. By combining advanced GPU acceleration with a unified, simplified deployment process, Google aims to make AI models more accessible and efficient on billions of devices worldwide.

Inspired by

• https://www.infoworld.com/article/4123828/googles-litert-adds-advanced-hardware-acceleration.html

Sources

• github.com
• arxiv.org
• googleblog.com
• googleblog.com
• github.com