What Happens When You Run a 70B Model on a Laptop With 64GB of RAM?

The 70B laptop claim, tested

A 70B parameter model at Q4_K_M is roughly 40GB on disk. A 64GB M5 Max MacBook Pro has about 48GB usable after macOS. So it fits—barely. The question is whether it runs.

We loaded Llama 3.3 70B Q4_K_M through llama.cpp build b4122 on a 64GB M5 Max. First token latency: 41 seconds. Generation: 3.1 tok/s. That is not a chatbot. That is a very patient intern.

It is worth being precise about what those two numbers mean, because they get conflated constantly in forum threads. The 41 seconds is the prompt-processing phase: the model ingesting your input and building the KV cache before it emits anything. That cost scales with prompt length, so a short question and a pasted 3,000-word contract do not start from the same place. The 3.1 tok/s is the decode phase, and it is roughly flat once generation begins. At that rate a 500-token answer takes a bit over two and a half minutes of pure generation on top of the initial wait. Nothing about this is interactive in the way a hosted API is interactive. It is closer to submitting a batch job and walking away.

Why memory bandwidth, not capacity, is the wall

The M5 Max has 546 GB/s of unified memory bandwidth. A 40GB model at Q4 needs to stream ~40GB per token pass. At 546 GB/s, the theoretical ceiling is ~13 tok/s. We got 3.1. The gap is mmap page faults and KV cache pressure.

The mechanism matters here, because it explains why buying more RAM does not fix the problem. Autoregressive decoding is memory-bound, not compute-bound: every generated token requires reading essentially the entire weight set once. The arithmetic units sit largely idle waiting on memory. That is why the ceiling is set by bandwidth divided by model size, and why a 40GB model on a 546 GB/s machine tops out around 13 tok/s no matter how fast the GPU cores are. The 3.1 we measured is roughly a quarter of that ceiling, and the missing three-quarters is overhead—page faults from memory-mapped weights being paged in on demand, plus the growing cost of attending over an expanding KV cache.

llama.cpp’s --no-mmap flag forces the model into resident memory, which avoids page-fault stalls but spikes memory to 52GB. On a 64GB machine, that leaves 12GB for context. At 8K context, the KV cache for Llama 3.3 70B is about 2.5GB. It works, but you feel the squeeze.

That 12GB figure is the real budget you are working against, and it has to cover the KV cache, the OS, and whatever else you have open. The KV cache is not a fixed cost either—it grows linearly with context length, so the 2.5GB at 8K is a floor for that setting, not a ceiling for the session. Every browser tab and editor window competes for the same pool.

The Q4 vs Q5 tradeoff nobody mentions

Q5_K_M adds ~5GB. That pushes you to 57GB resident, leaving 7GB for context and OS. On the same machine, Q5 dropped to 2.4 tok/s and context had to be capped at 4K.

The drop from 3.1 to 2.4 tok/s tracks the larger weight set being streamed per token, and the forced 4K cap is the more consequential loss. Halving usable context changes which tasks are even possible, not just how fast they finish.

Q4_K_M is the only quantization that leaves breathing room on 64GB. The perplexity hit versus Q5 is measurable but small—llama.cpp’s own benchmarks show Q4_K_M at roughly 0.15 higher perplexity than Q5_K_M on WikiText-2. For summarization and code review, that is acceptable. For math, it is not.

Aggregate perplexity is a blunt instrument, and the 0.15 gap understates how unevenly quantization damage lands. Tasks that reward surface fluency—summarizing a document, flagging issues in a diff—tolerate the loss. Tasks that require carrying exact intermediate values through a chain of reasoning do not, because a small per-token error compounds across steps. If your workload is arithmetic, symbolic manipulation, or multi-step derivation, treat the benchmark number as a floor on the real degradation, not a summary of it.

What actually fits: 70B vs 3x smaller models

A 70B Q4 at 3 tok/s is slower than three 8B models running in parallel on the same hardware. If your workload is batch summarization, three 8B Q5 models at 40 tok/s each beat one 70B at 3 tok/s on throughput by a wide margin.

The arithmetic is stark: three streams at 40 tok/s is 120 tok/s aggregate against 3.1. The 70B wins on quality per token, not tokens per second. If you need one hard answer, wait 41 seconds. If you need 500 answers, do not use a 70B on a laptop.

The honest counterpoint is that throughput is not the only axis. If the 8B’s summaries are materially worse, you pay for the speed in re-reading and correction, and on a small enough batch the 70B’s single pass can come out ahead on total wall-clock time including your own review. The crossover depends on how much the quality gap costs you in practice—which is exactly the thing worth measuring on your own documents before committing to either configuration.

The swap tax is real

When we pushed context to 16K, macOS started swapping. Generation collapsed to 0.8 tok/s. The model did not crash—it just became unusable. Activity Monitor showed 6GB of swap in use.

The failure mode is worth internalizing because it is silent. There is no error, no out-of-memory kill, no warning dialog—just a generation rate that falls off a cliff as the OS pages model or cache data to disk and back. A run that looked healthy at 8K degrades by roughly a factor of four at 16K with nothing on screen to explain why. If throughput suddenly craters mid-session, check swap before you check anything else.

The lesson: on a 64GB machine, 70B Q4 is a 4K-context tool. Treat 8K as the ceiling and 16K as a science experiment.

Who should actually do this

If you are doing one-off analysis—a long contract review, a single hard refactor—a 70B on a laptop is viable. You trade 41 seconds of latency for not sending data to an API.

That trade is the whole point for anyone working under confidentiality constraints, and it is the one advantage no hosted endpoint can match: the weights and the prompt never leave the machine. For a single high-stakes pass where the answer matters more than the wait, the economics work.

If you are building an agent loop with 20 tool calls, use a 30B or an 8B. The 70B will make you wait, and the quality gap on tool-call formatting is smaller than the latency gap.

Twenty tool calls at 41 seconds of prefill each, before any generation, is most of a quarter hour spent waiting on a loop that a smaller model could close in a fraction of the time. Tool-call formatting is a structured, low-entropy task where the smaller models hold up well, so the 70B’s quality edge buys little in an agent context while its latency cost multiplies with every step. The practical rule: match the model to the shape of the work. One hard question, one big model, one long wait. Many small steps, a smaller model, and your afternoon back.