How local Whisper transcription works on Apple Silicon
Real-time speech-to-text on a laptop is less about the model and more about what you refuse to send it.
People assume the hard part of local transcription is the model. It isn't — whisper.cpp solved that. The hard part is everything either side of it: getting clean audio out of macOS, and being ruthless about how little of it you hand to the model.
This is how Oats does it. The code is open source if you'd rather read that than prose.
1. Getting both sides of the conversation
A meeting has two audio sources and macOS treats them completely differently:
- Your voice arrives through Core Audio from the microphone. Straightforward.
- Everyone else arrives as system output — the audio your speakers are playing. Since macOS 13 this is available through ScreenCaptureKit.
This is why an app that only wants sound asks for screen recording permission: Apple put system-audio capture behind that gate. It surprises people, and it's the single most common setup confusion.
2. Mixing two streams that don't agree
The two streams arrive asynchronously, at different rates, in different buffer sizes. Naively summing them produces clipping and a recording where whoever is louder wins.
So the samples land in a ring buffer and are only mixed once both sides have a full aligned window (50ms). Then RMS-based ducking attenuates system audio when you're speaking, so your voice isn't buried under a shared screen's video. The output is a single stream where both parties are intelligible.
Worth noting: the recording path and the transcription path diverge here. The saved recording gets the nicely mixed, listenable version. The transcriber gets something rawer and much shorter.
3. Voice activity detection, where the real win is
A one-hour meeting is not one hour of speech. It's pauses, breathing, keyboard noise, someone's dog, and the ninety seconds at the start where two people wait for a third.
Voice activity detection runs before inference and discards everything that isn't speech. In a typical meeting this removes roughly 70% of the audio — which is 70% of the inference you don't pay for in heat, battery, and latency. It's a much bigger lever than swapping model sizes.
The tuning problem VAD creates: if you only cut on silence, someone talking continuously for four minutes produces one enormous segment, and nothing appears on screen until they stop. Long utterances have to be force-cut every few seconds so text keeps streaming. Getting this wrong is why some local tools feel laggy in exactly the meetings where you most want notes.
4. The model, and the GPU
Transcription runs through whisper-rs, with acceleration chosen at build time:
| Platform | Acceleration |
|---|---|
| Apple Silicon | Metal + CoreML |
| NVIDIA | CUDA |
| AMD / Intel | Vulkan |
| Fallback | CPU |
On Apple Silicon, Metal plus CoreML is the difference between comfortably real-time and visibly behind. It's roughly a 5–10× swing versus CPU-only, and it's why “local transcription is too slow” stopped being true around the M1.
Parakeet is available as an alternative: faster than Whisper, English-only. If your meetings are English and latency is what you care about, it's the better pick.
5. Summarising without a server
Transcript to summary runs through a local model via Ollama by default. If you want a sharper write-up you can supply your own Claude, Gemini, or OpenAI key — and that's the one path where text leaves the machine. It's opt-in, it's obvious in the UI, and the key is stored on device.
What this buys
No per-minute billing, because inference is your hardware's problem. No connection required. And a claim about privacy that can be checked by reading the source rather than trusting a policy page.