Show notes
Not every AI model needs a data centre. Machine learning engineer Ravi Menon explains how smaller models run directly on phones and laptops, what they're good at, and why privacy is only part of the story.
On this page
Most of the conversation about artificial intelligence focuses on very large models, trained and run in vast data centres. But a parallel story has been unfolding at a much smaller scale: AI models compact enough to run directly on a laptop, a phone or even smaller devices, without sending anything to the cloud.
Ravi Menon is a machine learning engineer who specialises in making models smaller and faster. In this episode, he explains how on-device AI works, what it can and can't do, and why it might matter more than the headlines suggest.
In this episode
- 00:00 — The first time Ravi ran a model on a phone
- 05:50 — Big models versus small models
- 13:05 — How you shrink a model: quantisation and distillation, explained
- 22:20 — What small models are surprisingly good at
- 29:40 — What they still can't do
- 36:15 — Privacy, latency and cost
- 42:30 — Where this is heading
How you shrink a model
Ravi walks through two of the main techniques used to make models smaller, in plain language.
Quantisation reduces the precision of the numbers inside a model. Many models store their internal values with considerable precision; quantisation rounds them to fewer bits. Done carefully, this can shrink a model dramatically while keeping much of its usefulness.
Distillation trains a smaller "student" model to imitate a larger "teacher" model. The student doesn't learn everything the teacher knows, but it can learn to mimic its behaviour on particular tasks surprisingly well.
"A small model doesn't need to know everything. It needs to be very good at the few things you actually ask it."
What small models are good at
According to Ravi, small on-device models tend to shine at focused, well-defined tasks: summarising a document, transcribing speech, suggesting replies, sorting photos, correcting grammar, translating short passages. They are less suited to tasks that require broad knowledge or long chains of reasoning, where larger models still tend to have a clear advantage.
Many real products, he notes, combine both approaches: handling simple tasks on the device and sending only harder requests to a larger model in the cloud.
Why it matters
Privacy is the most obvious benefit. If a model runs on your device, your data doesn't need to leave it. But Ravi argues that two other benefits are just as important.
Speed. There's no round trip to a server, so responses can be almost instant — essential for things like live transcription.
Cost and availability. On-device models work offline, on trains and planes and in places with poor connections. They also don't incur ongoing server costs for every request, which can make some features viable that otherwise wouldn't be.
The trade-offs
Daniel asks about the downsides. Ravi is candid: smaller models make more mistakes, can be harder to update and drain batteries if used heavily. Running them well requires capable hardware, which isn't evenly distributed. And the fact that a model runs locally doesn't automatically mean an app is private — it depends on what else the app does with your information.
Key takeaways
- Not all AI needs a data centre. Smaller models can run directly on laptops, phones and other devices, often without an internet connection.
- Quantisation shrinks models by storing their internal numbers with less precision, trading a little accuracy for a much smaller footprint.
- Distillation teaches a small model to imitate a larger one on specific tasks, keeping much of the useful behaviour.
- Small models suit focused tasks — transcription, summarising, translation, sorting photos and suggesting replies.
- Big models still lead on breadth. Tasks needing wide knowledge or long reasoning often go to larger cloud models, and many products combine both.
- Privacy, speed and cost all improve when work happens locally: no data leaves the device, responses are faster and features work offline.
- Local doesn't automatically mean private. What matters is what the whole app does with your information, not just where the model runs.
Transcript excerpt
Maya: Will most AI eventually run on our own devices?
Ravi: I think most everyday AI will. The quick, personal, repetitive stuff. The big, hard questions will probably still go to big models for a long while. It'll feel less like choosing one or the other and more like having a fast assistant in your pocket who knows when to phone a specialist.
Mentioned in this episode
- Quantisation and knowledge distillation
- On-device speech recognition and translation
- Hybrid approaches combining local and cloud models
About our guest
Ravi Menon is a machine learning engineer who focuses on model compression and on-device inference. He has worked on speech recognition, translation and text features designed to run on phones and laptops, and he contributes to several open-source tools for running models locally. Before moving into machine learning, he worked on embedded systems for consumer electronics, which he says taught him to respect every byte of memory. He runs a small model on his own phone to transcribe voice notes, mostly shopping lists.
Credits
Signal is hosted by Maya Chen and Daniel Osei.