CLASSEVE
RoutePublic
Public / RAI

RAI.

A CPU-only LLM inference engine written in Rust — 4-bit quantized kernels, model conversion, speculative decoding, and local serving. No CUDA, ROCm, Metal, PyTorch, or GGML at runtime.

Open source under Apache-2.0 — clone it or browse the repo:

git clone https://github.com/classeve-public/rai.git
Open repository on GitHub
What it is

Local inference, on the CPU you already have.

RAI is a Rust workspace for CPU-only LLM inference: quantized model execution, model conversion, and a small local server. It is built for local inference research and systems development, and it runs without a GPU stack — no CUDA, ROCm, Metal, PyTorch, or GGML required at runtime.

How you use it
  1. Build it. Stable Rust and one cargo build --release — no GPU stack, no Python environment, no driver hunt.
  2. Convert a model. The bundled converter turns supported weights into the native .raimodel format. Model files stay on your machine.
  3. Serve it locally. Start the built-in HTTP server and send prompts to it, or run MCP stdio mode and plug it straight into agent tooling.
  4. Tune per request. Temperature, top-k, top-p, repetition penalty, and speculative decoding are all request-level switches.

Fair warning: RAI is a systems tool for people who like reading docs — not a double-click chat app.

Capabilities

Quantized kernels

W4A32 and W4A8 matrix and vector paths with AVX2-oriented implementations.

Model format

A native .raimodel loader and binary-format parser.

Transformer layers

RMSNorm, RoPE, grouped-query attention, SwiGLU MLP, and a KV cache.

Decoding

Temperature, top-k, top-p, and repetition penalty, plus speculative decoding.

Compression

A GPTQ-oriented calibration flow with residual and adaptive compression experiments.

Serving

A local HTTP server and an MCP stdio mode for tool integration.

Requirements

Stable Rust (edition 2021) and an x86_64 CPU with AVX2, FMA, and F16C for the optimized inference paths. Windows, Linux, or macOS on supported hardware. No runtime GPU. Model assets and calibration data stay on your machine — they are intentionally not committed to the repository.

More public buildsFrontier researchAbout ClassEve
FAQ

Frequently asked questions

What is RAI?

RAI is a free, open-source LLM inference engine written in pure Rust that runs entirely on the CPU. It includes 4-bit quantized kernels, the .raimodel format, speculative decoding, and a local HTTP and MCP server.

Can I run a language model without a GPU?

Yes — that is the point of RAI. It performs inference on a standard x86-64 CPU with no graphics card, using AVX2-oriented quantized kernels.

Does RAI need CUDA, PyTorch, or Python?

No. There is no CUDA, ROCm, Metal, PyTorch, or GGML at runtime. RAI is a self-contained Rust workspace with no Python dependency.

What hardware does RAI need?

An x86-64 CPU with AVX2, FMA, and F16C for the optimized paths. It builds and runs on Windows, Linux, and macOS.

Is RAI available on crates.io?

Yes. The crates are published under the classeve-rai-* namespace. You can install the local server with cargo install classeve-rai-server.

How does RAI compare to llama.cpp?

RAI is a research-oriented Rust engine focused on CPU-only quantized inference and a clean local-serving path. It is not a drop-in replacement for mature projects like llama.cpp; it is an independent implementation you can read, audit, and build on. The source is Apache-2.0 on GitHub.