On-Device AI vs Cloud AI on a Mac: What Actually Runs Where

"Runs on device" is the most overloaded phrase in AI marketing right now. It can mean the whole model runs on your Mac. It can mean a small classifier runs on your Mac and everything else is a network call. It can mean the app caches your settings locally.

All three are technically true statements. Only one of them means your text stays on the machine.

The quickest check takes one step: turn off Wi-Fi and use the feature. If it still works, the text is being generated on your Mac. Three more checks, and the arithmetic for what a Mac can actually run, are below.

The three things "on-device" can refer to

Before any comparison, separate what is being claimed. A writing tool has at least three stages, and each one can independently be local or remote.

• Preprocessing. Detecting the language, finding the text around your cursor, detecting and masking personal information. Small, fast, and almost always feasible locally.
• Generation. The actual language model producing the text. Large, and the expensive one.
• Telemetry and account. Sign in, usage counting, crash reports. Always remote, and rarely what anyone is asking about.

When a product says "on-device", ask which stage. A tool that masks locally and generates remotely is being honest if it says so, and is a meaningfully different product from one that generates locally. Neither is automatically better. They are different tradeoffs.

What a Mac can actually run

The binding constraint on a Mac is unified memory, because the CPU and GPU share one pool. Model weights have to fit in it alongside macOS, your browser, and everything else.

A rough sizing rule that holds up well in practice: at 4-bit quantisation, a model needs a little over half a gigabyte of memory per billion parameters, plus room for the context. Real 4-bit formats store about 4.5 to 5 bits per weight once their scaling data is counted. So:

• A 3B model at 4-bit: roughly 1.7 GB of weights. Comfortable on any Apple Silicon Mac.
• An 8B model at 4-bit: roughly 4.5 GB. Workable on 16 GB, tight on 8 GB.
• A 32B model at 4-bit: roughly 18 GB. Needs 32 GB to be pleasant, 24 GB at a squeeze.
• A 70B model at 4-bit: roughly 40 GB. Needs 64 GB or more.

Quantisation is what makes any of this possible. A model trained in 16-bit precision stores each weight in two bytes; 4-bit storage cuts that by four. The cost is real but small at 4 bits and grows sharply below it. At 8-bit the quality difference from full precision is usually not noticeable. At 4-bit it is measurable but acceptable for most tasks. At 2-bit, models tend to degrade in ways that look like sudden stupidity rather than gentle decline.

For reference, Apple's own on-device foundation model in Apple Intelligence is around 3 billion parameters, and Apple routes harder requests to its Private Cloud Compute servers. That is the same architecture decision every serious product faces, made by the company with the most control over the hardware.

How fast it will be, without benchmarking anything

Token generation on a local model is limited by memory bandwidth, not by compute. Each token generated requires reading every weight in the model once, or for a mixture-of-experts model, every weight that is active for that token. So:

tokens per second is at most memory bandwidth divided by model size in memory.

Apple publishes the bandwidth of each chip, and it rises with every generation. On the M4 generation it is 120 GB/s for the base chip, 273 GB/s for the Pro, and up to 546 GB/s for the Max. Ultra chips reach 800 GB/s and more.

Put those together. An 8B model at 4-bit is about 4.5 GB, so a base M4 gives at most roughly 120 divided by 4.5, or about 27 tokens per second. An M4 Max gives at most about 120. A 32B model at 4-bit, about 18 GB, gives at most about 7 tokens per second on a base M4 with enough memory, which is roughly reading speed.

Treat these as ceilings. No runtime reaches the chip's full memory bandwidth, so real speeds come in below them.

Two things this rule explains that otherwise look strange:

• A bigger model is not just slower to start, it is proportionally slower forever. Double the model, halve the speed, on identical hardware.
• Prompt processing behaves differently. Reading your existing text is compute-bound and parallel, so it is fast and scales with GPU cores. Generating is bandwidth-bound and sequential. A long document costs you at the start, once; a long answer costs you throughout.

One more detail that surprises people: most local LLM runtimes on macOS use the GPU through Metal, not the Neural Engine. The Neural Engine is efficient and excellent at the fixed-shape models Apple's own frameworks target, but it is not generally programmable in the way a large language model runtime needs. So "it uses the Neural Engine" is a specific claim about a specific implementation, not a general property of on-device AI.

What it costs you while you are working

This is the part comparisons skip, and it is what actually decides whether people keep a local model turned on.

• Memory pressure. A 4.5 GB model resident on a 16 GB Mac is more than a quarter of your memory, permanently, while it is loaded. Your browser notices.
• Battery. Sustained GPU work is one of the most expensive things a laptop does. A local model answering continuously is not a background task; it is a workload.
• Cold start. Loading several gigabytes of weights from disk takes seconds. Keeping the model resident avoids it and costs the memory above. Every local tool picks one of these problems.
• Thermals. On a fanless Mac, sustained generation throttles. Speed measured on the first answer is not speed on the tenth.
• Update cost. A better model is a multi-gigabyte download, not a server-side change you never notice.

Against that, the things a local model gives you are genuine and not small: it works with no network, there is no per-request cost, latency has no round trip, and no text leaves the machine at all.

What actually belongs on the device

The useful question is not "local or cloud" but "which stage where". Three properties decide it.

Put it locally when the work is small, constant, and touches raw text. Language detection, personal information detection and masking, finding the paragraph around the cursor, deciding whether to offer a suggestion at all. These run in milliseconds, need no large model, and are exactly the steps that see your unredacted text. A masking step that runs on a server is not a masking step; it is one more place your data arrives.

Put it on a server when quality is the product and the model is large. A 3B model and a 30B model are not the same product with different latency. On real writing tasks the gap shows up as tone that misses, instructions ignored, and confident nonsense.

Fail closed, wherever it runs. If the local stage cannot run, the correct behaviour is to send nothing rather than to send the raw text. A privacy step that quietly disables itself on failure is worse than none, because you stop checking.

This is why most serious products are hybrid, and why the honest ones say so.

How to test a claim in two minutes

You do not have to take anyone's word for this.

Turn off Wi-Fi and use the feature. If it still works, generation is local. If it fails or hangs, it is not. This single test answers the question that matters most, and no marketing page can argue with it.

Look for the weights. A local model is a large file. Check ~/Library/Application Support, the app's container, and ~/Library/Caches for the app's folder, and sort by size. A few hundred megabytes suggests a small specialised model, such as a masking or classification model. Several gigabytes suggests a general language model.

Watch memory and GPU. Open Activity Monitor on the Memory tab, open Window, then GPU History, and use the feature. Local generation is visible: memory jumps by the model size and GPU usage spikes for the duration of the answer. Remote generation shows almost nothing.

Read the size of the download. An app that ships a general language model is a multi-gigabyte download or has a multi-gigabyte first run. A 40 MB app is not shipping a 7B model of its own, though it can still generate text locally with Apple's built-in model, which is why the Wi-Fi test comes first.

Those four checks agree with each other. When they do not, the disagreement is itself the answer: something small runs locally and something large does not.

What BlinkWrite does, specifically

BlinkWrite is a hybrid, and it is worth being exact about the split rather than claiming a category.

On your Mac: your raw text, language detection, and the detection and masking of personal information. The masking model is a small named entity model, a download of about 140 MB that unpacks to about 200 MB, exported to ONNX and quantised to INT8 so it runs with no Python runtime and no GPU. It replaces your own name, addresses, phone numbers, card numbers, and credentials with labelled placeholders before anything is sent, and the real values are restored locally in the answer. Other people's names are not masked. The implementation is open source under MIT at github.com/BlinkWrite/pii-masker.

Through BlinkWrite and an external LLM provider: generation. BlinkWrite sends masked text for suggestions with production API response storage disabled. Provider abuse-monitoring retention applies. BlinkWrite does not use production writing for training. Models & languages covers the model side and the 36 supported languages; How it works walks through the sequence from reading context to showing the popup.

The reason for the split is the sizing arithmetic above. A model small enough to sit resident on every supported Mac, alongside whatever else you are running, is not a model that writes a reply worth accepting. Rather than ship a weaker product and call it private, the sensitive values are removed before they leave and the generation happens where the quality is.

For the separate question of who receives text and what they keep, what your AI writing tool actually sends to the cloud covers sub-processor lists, retention, and how masking works in detail.

Common questions

Is on-device AI always more private?

More private about the network, yes: nothing leaves. It says nothing about what the app does locally, what it logs, or what it uploads later. "Local" and "private" overlap; they are not the same claim.

How much RAM do I need to run a local LLM on a Mac?

16 GB is the practical floor for an 8B model at 4-bit with room to work. 8 GB runs 3B models fine and struggles above that. 32 GB opens up 30B-class models, and 64 GB or more is what 70B-class models need.

Does the Neural Engine make local models faster?

For the models Apple's own frameworks target, it is very efficient. Most third-party LLM runtimes use the GPU through Metal instead, because that is what the runtime needs. If a product specifically claims Neural Engine use, that is a real implementation detail worth asking about rather than a general property.

Why not just run everything locally and avoid the question?

Quality and battery. The models that fit comfortably on a typical Mac are meaningfully weaker at real writing tasks than server-class models, and running one continuously is a sustained GPU workload on a laptop. The realistic choice is where to put each stage, not whether to have a server at all.

Does quantisation make the model worse?

At 8-bit, barely. At 4-bit, measurably but usually acceptably. Below 4-bit, quality falls off quickly and unpredictably. Most local tooling defaults to 4-bit because that is where the tradeoff still works.