What? Even large language models are finally getting "weight-loss shots"?

This isn’t just my speculation. According to CNBC, Apple is in talks with startup PrismML, a company renowned for its newly launched model compression technology. Apple hopes to leverage this tech to assess the feasibility of running much larger AI models natively on iPhones.

(Image source: CNBC)

For years now, whenever smartphone makers get to the AI segment of their launch events, I usually reach for my water cup without thinking.

It’s not that I’ve got a grudge against device manufacturers. It’s just that this routine has become far too familiar to everyone. First, the AI summarizes all kinds of content displayed on screen. Next, various image editing tools are trotted out for custom color grading or erasing random strangers from photos. A new feature widely rolled out this year is summoning a voice assistant to order a coffee for you.

Still, we can’t place all the blame on phone brands. State-of-the-art large language models nowadays are simply too bulky to fit inside mobile phones. The downsized edge AI models, meanwhile, leave much to be desired in terms of performance. At the end of the day, manufacturers can only showcase cloud-based features in their pitches. Take Doubao’s newly launched AI podcast function for example—major smartphone brands all rushed to roll out identical features within roughly three months.

This begs a crucial question: if full-scale large language models can be slimmed down enough to fit on smartphones, will on-device AI assistants finally become fully functional, reliable tools?

From 54GB down to 4GB: Is model compression technology poised for widespread adoption?

First off, let’s break down these two questions together with Leitech (ID: leitech):

Who exactly is PrismML?

According to its official website, PrismML is a startup specializing exclusively in model compression. Spun off from a research team at the California Institute of Technology, the firm counts Khosla Ventures, Cerberus and Google among its backers. Its core research focus lies in drastically cutting down model file sizes and operational overhead—all while avoiding crippling the model’s reasoning capabilities in the process.

(Image source: PrismML)

What have they accomplished?

PrismML’s approach shares similarities with low-bit model solutions such as BitNet. It shrinks model size by drastically simplifying how information is stored inside AI models. The technology restricts each weight within the model to binary or ternary representations, greatly cutting down the memory required for storage and model execution.

To elaborate, a single parameter in conventional large language models typically requires 16-bit or even 32-bit storage space.

(图片来源: HuggingFace)

Under these conditions, a 27-billion-parameter model using FP16 precision would take up approximately 54GB of storage, calculated as 27 billion × 2 Bytes. This matches the rough file size of Qwen3.6-27B under FP16 precision.

Not to mention smartphones, even many consumer-grade personal computers struggle to run the full model smoothly.

By contrast, PrismML’s approach simplifies all parameters in its 1-bit variant to only two values: {-1, +1}. To draw an analogy: old photos once stored 16 grayscale levels per pixel, while this method only keeps two shades—black and white. Although this leads to substantial information loss, the model size can be compressed to one-fourteenth of the original. Moreover, training techniques are applied to restore its inference performance.

(Image source: PrismML)

Building on this technology, the company officially released the Bonsai-27B model on July 15. Fine-tuned from Qwen3.6-27B, it shrinks the model from roughly 54GB to under 4GB while retaining full context windows, enabling native execution on iPhones equipped with 12GB of RAM.

For reference, Google’s Gemma 4 E4B built for mobile and edge devices occupies around 3.65GB of storage. In other words, PrismML has managed to fit a dense model boasting a nominal 27 billion parameters within roughly the same storage footprint.

Setting aside real-world user experience for now, hardware manufacturers are undoubtedly thrilled by this breakthrough.

(Image source: PrismML)

It comes as no surprise that Apple is interested in this technology.

Apple’s proprietary on-device model only has around 3 billion parameters. Although it leverages techniques such as 2-bit quantization and cache sharing, it can only handle lightweight tasks on iPhones including real-time translation, photo album search, and email summarization. It barely possesses any execution capabilities for AI Agents.

In contrast, the Bonsai-27B model retains a portion of the Agent capabilities native to Qwen3.6-27B.

Naturally, some performance degradation is inevitable. According to PrismML’s internal benchmarks, the ternary version retains roughly 95% of the full-precision model’s overall performance, while the 1-bit variant retains approximately 90%. Performance drops are far more pronounced for Agent-critical tasks like tool calling.

Feedback from community testing also indicates that PrismML’s ternary model still suffers from issues such as hallucinations and Agent loops when compared to Q4_K_XL. Its standout advantage, however, lies in its ultra-small size: it delivers comparable performance to the 17.9GB model at just 5.9GB.

(Image source: Reddit)

Still, a usable model beats one that can’t run at all.

We’ve moved past the physical limitation of models being too large to fit onto devices; now the focus shifts to whether real-world performance is acceptable. If this technological advancement keeps moving forward, the industry is in for fierce competition ahead.

The AI smartphone boom is imminent, and on-device AI capabilities are in urgent need of improvement

Interestingly, on July 15, seven on-device generative AI model services for mobile phones—including Apple Intelligence, Huawei Xiaoyi, OPPO, Xiaomi and vivo—all completed filing with the cyberspace administration authorities.

The lineup is quite extensive, making it clear that smartphone brands have fully committed to developing on-device AI this year.

(Image source:  Cyberspace Administration)

The reasoning is easy to follow. Tasks such as notification summarization, call log organization, album search, and image recognition do not require repeated queuing on remote cloud servers.

This is especially true for private data including chat histories, photos and documents. Processing such data locally on one’s own phone is undoubtedly the ideal solution. Given Grok’s recent massive privacy scandal, I fully understand why users are reluctant to have their personal data transmitted back and forth across the internet.

(Image source: LeiTech)

The issue is that from my personal experience, the AI features offered by all smartphone brands still rely primarily on cloud computing. The vast majority of these functions stop working entirely once the device loses internet access.

Why is this the case? Where exactly does on-device AI for smartphones stand in its current stage of development?

Coincidentally, I recently tested Gemma 4 E4B on Google AI Edge Gallery, and I’d like to share my hands-on experience with everyone.

(Image source:  Google)

First of all, it’s worth noting that Gemma 4 E4B is already one of the most capable on-device models for smartphones, supporting text, image and audio processing. Once downloaded, it works perfectly offline for conversations without an internet connection.

Take Ask Image as an example. It delivers multimodal input capabilities that most previous on-device mobile AI systems struggled to achieve.

My hands-on testing shows Gemma 4 boasts solid image recognition performance. While it still lacks familiarity with anime characters, it excels at capturing visual features within frames and can accurately identify common items such as food, electronic hardware and flowers.

(Image source: LeiTech)

There is also Ask Audio, which supports uploading audio clips of up to 30 seconds for transcription, summarization and other operations.

This feature, however, leaves much to be desired. Possibly due to the poor clarity of my recorded audio, the transcribed text bore almost no resemblance to the original audio content. Its practical usability is rather limited for now. It is far more reliable to rely on Doubao or Qwen for audio summarization tasks.

(Image source: LeiTech)

As for text processing...

I fed an article of roughly 2,500 words into several models deployable on mobile phones, asking each to generate a corresponding summary.

In the end, only Gemma 3n E4B and Gemma 4 E4B managed to complete the task. That said, the former took nearly two minutes to run and produced summaries that failed to capture core points. In contrast, outputs from Gemma 4 E4B are far more concise and to the point, covering almost all key information. It works perfectly for quickly skimming through reference materials.

Even for certain logic puzzles that previous models failed to solve, Gemma 4 E4B can crack them with extended thinking time—though its reasoning latency far outpaces that of cloud-based large language models.

From Leitech’s perspective, Gemma 4 E4B has proven that locally deployed mobile AI models are genuinely capable of handling practical tasks.

That said, I’m only willing to rely on it for straightforward jobs like text summarization, rewriting, and basic image recognition. For slightly more complex assignments—especially those involving lengthy Chinese text, nuanced judgment, and creative content generation—the performance gap between it and cloud LLMs remains glaringly obvious, not to mention tasks requiring Agent function calls.

Bear in mind that Gemma 4 is already the most powerful on-device mobile AI model available today, both in terms of compression ratio and functional breadth.

To push past this performance ceiling, Apple will have to abandon its current compression strategies. Only by packing higher-parameter models into the same storage footprint can iPhones be equipped with an on-device AI "brain" less prone to flawed reasoning.

Parameter Scale Is Irrelevant; Efficiency Is the Core of Model Application

In earlier discussions of large language models, the general consensus was that more parameters equate to greater prestige.

Billions of parameters were merely a starting point, and trillions were seen as desirable. Announcing parameter counts at product launches became akin to weighing produce at a wet market, with brands vying to boast that their "radishes are bigger and sturdier" than competitors’.

This logic may sound reasonable on the surface, yet real-world runtime performance tells a completely different story.

According to research on the Chinchilla Scaling Law conducted by Jordan Hoffmann’s team, under a fixed training compute budget, a fully trained 70-billion-parameter model comprehensively outperforms undertrained models with 280 billion or 530 billion parameters. Even trillion-parameter Mixture-of-Experts (MoE) architectures face substantial bottlenecks in terms of VRAM footprint and memory bandwidth consumption.

(Image source: arXiv)

More importantly, models with such massive parameter counts can never run on consumer-grade end devices.

If on-device models are to operate permanently on smartphones, efficiency improvements are a must to tackle constraints of memory, power consumption and heat dissipation. PrismML’s compression algorithm has paved a viable path for all hardware manufacturers. Furthermore, Apple’s local AI collaborations with Qwen and Baidu have brought Apple Intelligence, which Apple promised three years ago, one big step closer to full-fledged implementation.

Future advances in large language models won’t necessarily rely on eye-watering parameter figures.

The next frontier of competition for on-device AI lies in models that run stably within limited storage space, produce fewer erroneous outputs, generate less heat, and deliver genuine practical assistance when needed.

The WAIC2026, themed “Intelligent Partners, Co-Creating the Future,” opens soon.

The AI narrative is shifting—from stacking model parameters toward tangible agent-driven productivity; heterogeneous computing and photonic computing continue pushing computational ceilings upward; embodied intelligence accelerates real-world deployment, bringing robots into homes and factories—and turning physical AI into reality.

LeiTech’s WAIC exhibition team has arrived in Shanghai—capturing the annual peak moment of AI industrialization firsthand!