Learn / CPU-only LLM inference

Can you run an LLM without a GPU?

Yes — language models can run on an ordinary CPU with no GPU at all. The keys are quantization (storing weights in ~4 bits instead of 16, shrinking memory several-fold) and CPU-native code paths such as hand-written AVX2 kernels. The result is private, local inference on everyday laptops — at smaller model sizes and lower speeds than datacenter GPUs.

Also called:run LLM without GPULLM on CPUlocal LLM on laptopCPU inference engine4-bit quantizationquantized language modelsrun AI locally without graphics cardon-device LLMprivate local AIno-GPU AI inference

How CPUs manage it

Two ingredients. Quantization compresses model weights from 16-bit floats to about 4 bits with modest quality loss, cutting both memory footprint and memory bandwidth — the true bottleneck of CPU inference. Then vectorized kernels (AVX2 and similar instruction sets) keep every core's SIMD lanes full while the weights stream through.

An engine written for this from scratch — rather than a GPU engine ported down — decides everything about real speed: kernel quality, cache behavior, and thread scheduling are the whole game.

What it's good for

CPU inference suits assistants, drafting, extraction, summarization, and agent tooling at small-to-mid model sizes, on hardware you already own, with total privacy. It does not compete with GPU clusters for large models or high throughput — the point is different: no cloud, no new hardware, nothing leaves the machine.

From ClassEve

RAI is ClassEve's CPU-only LLM inference engine: pure Rust, 4-bit quantized models, hand-written AVX2 kernels, a built-in local chat UI — open source.

RAI — CPU-only inference engine in Rust

CPU-only LLM inference · FAQ

Common questions.

How fast is LLM inference on a normal laptop CPU?

It depends on model size, quantization, and the machine — small quantized models reach conversational speeds on recent laptops. Treat any specific number as hardware-specific; measure on your own machine before deciding.

Why run an LLM on CPU instead of using a cloud API?

Privacy (prompts never leave the machine), cost (no per-token billing), and availability (works offline). The trade is capability: local CPU models are smaller than frontier cloud models.

What does ClassEve build for CPU inference?

RAI — a CPU-only LLM inference engine written in pure Rust, running 4-bit quantized models through hand-written AVX2 kernels, with a local chat UI and HTTP serving built in. Open source on GitHub.

Compared head to head

RAI vs. llama.cpp