How Fast Do Local LLMs Run on an 8 GB Laptop GPU?

We ran five models from Ollama's library on an RTX 5070 Laptop GPU with 8 GB of VRAM and compared their real generation speed with the speed our model pages estimate for an 8 GB card. The two models that fit entirely in VRAM ran at 45% and 60% of our estimate. A 12B model that our calculator says fits ran at 23%, because only 67.1% of it was placed in VRAM. The two 32B models, mostly running from system RAM, generated 1.8 and 2.2 tok/s against the 2.7 we estimate for a model that spills.

What we measured

The laptop has an RTX 5070 Laptop GPU with 8 GB of VRAM, an Intel Core i7-14650HX and 32 GB of DDR5-5600 RAM. It runs Windows 11 with NVIDIA driver 591.44, plugged in, on the Balanced power plan. We used Ollama 0.40.5 with default settings and only models that were already installed: nothing was downloaded for this test. Each model got one loading pass, then three timed passes with the same text prompt (908 to 1,059 tokens, depending on each model's tokenizer), a 4,096-token context, temperature 0 and 128 generated tokens. Speed is the number of generated tokens divided by the generation time Ollama reports; the share of the model in VRAM comes from ollama ps and VRAM use from nvidia-smi. Sizes follow the site's convention, where 1 GB is 1,024³ bytes.

Measured vs estimated

All five are Q4_K_M builds. Our estimate is the figure our model page shows for a public GGUF of the same model (8 GB of VRAM, 32 GB of RAM, 4,096 tokens; the pages are linked at the end). The files Ollama installed are within 0.5 GB of those, except qwen3.5:9b (6.14 GB installed against 5.29 GB).

Why the estimate runs high when a model fits

For a dense model, decoding reads every weight once per generated token, so memory bandwidth divided by the size of the weights is a ceiling on speed. Our calculator uses a fixed 400 GB/s for every GPU. This card's peak is lower: NVIDIA lists a 128-bit GDDR7 interface for the RTX 5070 Laptop, and Tom's Hardware's spec table gives 24 Gbps, which is 384 GB/s. For mistral:7b that makes the physical ceiling about 88 tok/s, and the measured 58.8 tok/s is 67% of it. Decoding also spends time on things that are not weight reads, such as attention over the KV cache, and we did not isolate their share.

Heat mattered too. In a second session, started with the GPU already at 84 °C instead of 76 °C, the same mistral:7b ran at about 40 tok/s instead of 58.8, which is 41% of our estimate instead of 60%.

When a model "fits" but spills

Our calculator gives gemma4:12b a total of 7.68 GB at 4,096 tokens, under the 8 GB tier, so it shows "fits in VRAM" and an estimate of 60.3 tok/s. In practice Ollama needed 8.34 GB in total, 0.66 GB more than our figure, and placed only 67.1% of the model on the GPU. Part of the gap is the laptop itself: the desktop and other apps were already using about 0.45 GB of the card's VRAM before we loaded anything, so 7.51 GB were free. The result was 13.6 tok/s, 23% of the estimate. We did not isolate why Ollama's total is above ours.

qwen3:32b and deepseek-r1:32b need about 20 GB, so only 28% and 29.5% of each went to the GPU. They generated 1.8 and 2.2 tok/s against our 2.7 estimate for a model that spills (67% and 81%): lower, but of the same order.

Memory: how close were the totals?

For the two models that fit, our totals were close to the VRAM actually added: 0.60 GB above for mistral:7b and 0.06 GB below for qwen3.5:9b. The estimate was optimistic where it mattered most, for the model that barely fit.

How to use these numbers

• Read the tokens per second on our pages as a ceiling. On this laptop, a model that fit entirely in VRAM ran at 45% to 60% of it when the GPU started at 76 to 78 °C, and at 41% when it started at 84 °C.
• Keep a margin. A laptop's desktop already uses some VRAM (about 0.45 GB here), and gemma4:12b spilled with a total 0.32 GB under the tier.
• Expect a steep price for partial offload: with 67.1% of the model in VRAM, gemma4:12b ran at 23% of our estimate.
• Measure your own machine: ollama run <model> --verbose prints timings after each response, and ollama ps shows how much of the model is on the GPU.

Limits of this test

This is one laptop on one day, with Ollama 0.40.5 at default settings and 128 generated tokens per pass. Speeds moved with temperature: the three timed passes of mistral:7b ranged from 42.3 to 65.4 tok/s in one session, and a later session ran at about 40. Other cards, cooling, drivers, runtimes such as llama.cpp or LM Studio, and settings such as flash attention or a quantized KV cache will give different numbers, and this test says nothing about output quality.

Check any model against your own card with the calculator, see what fits in 8 GB of VRAM, and read how we compute memory in our methodology. The model pages we used for the estimates: Mistral 7B Instruct v0.3, Qwen3.5 9B, Gemma 4 12B, Qwen3 32B and DeepSeek-R1 Distill Qwen 32B. Card specifications: NVIDIA's laptop GPU comparison and Tom's Hardware's spec table.

How we made this: the measurements were run on the editor's laptop on October 26, 2026 with Ollama 0.40.5 (three timed passes per model, logs kept); our estimates are the figures our calculator shows for the matching public GGUF repositories (see our methodology); the draft was written with AI assistance and reviewed by the editor before publication.