8.2 KiB
How Inference-X Works
A model's intelligence is in its weights. Everything between the weights and your screen is overhead. We removed the overhead.
The problem nobody talks about
When you download a 7B model from Hugging Face, you're getting an artifact that was trained for months on thousands of GPUs in FP32 precision. That model — the original, the teacher's intent — produces a certain quality of output.
But you never see that output.
What you see is the model's intelligence after it's been pushed through an inference engine. And every engine adds noise:
Original model (FP32)
→ Quantized to Q4_K (75% of data removed — intentional, necessary)
→ Loaded into framework (PyTorch, llama.cpp, vLLM: 10-500 MB of overhead)
→ Dequantized to intermediate buffer (rounding errors introduced)
→ Matrix multiply (separate pass — more rounding)
→ All experts loaded in memory (97% unused, competing for cache)
→ Uniform precision across all layers (simple queries processed like complex ones)
→ Output
What you get is the model's signal + accumulated noise from every step.
This is how every inference engine works. The model is the same. The output varies depending on how much noise the engine adds.
What Inference-X does differently
We don't add features. We remove steps.
1. Fused computation — zero intermediate buffers
Standard: dequantize block → store in FP32 buffer → matrix multiply against buffer. Two passes. One temporary allocation. Rounding errors at each boundary.
Inference-X: dequantize and multiply in the same instruction loop. One pass. No buffer. The quantized value goes directly from the block structure to the accumulator in a single fused operation.
// Standard: two passes, one buffer
float buffer[K];
dequant_q4k(buffer, weights, K); // pass 1: dequant → buffer (rounding)
float result = dot(buffer, input, K); // pass 2: buffer × input (rounding)
// Inference-X: one pass, no buffer
float result = fused_dot_q4k(weights, input, K); // dequant + dot in one loop
Fewer floating-point operations = fewer rounding errors = output closer to the theoretical FP32 result.
This is implemented for 10 quantization formats with hand-tuned AVX2/AVX-512 SIMD kernels.
2. Adaptive precision — the model breathes
Not every question is hard. "What's 2+2?" doesn't need the same computational depth as "Explain quantum entanglement in terms a physicist would appreciate."
Inference-X analyzes each query before inference begins using Shannon entropy and vocabulary diversity. The result is a complexity score that determines how much precision each layer gets:
| Query complexity | Early layers | Middle layers | Late layers | RAM savings |
|---|---|---|---|---|
| Simple (H < 0.3) | Q2_K | Q4_K | Base | ~26% |
| Moderate (0.3–0.6) | Q4_K | Base | Base | ~10% |
| Complex (H > 0.6) | Base | Base | Base | 0% |
Simple queries get faster answers with no quality loss — because the extra precision wasn't contributing signal, only noise. Complex queries get full precision where it matters.
The model file doesn't change. The binary doesn't change. The depth adapts to the question.
3. Surgical expert loading — silence the irrelevant
Mixture-of-Experts models (DeepSeek, Kimi K2.5) have 256–384 experts per layer but activate only 8 per token. Standard engines load all experts into RAM and let the OS manage caching.
Inference-X does something different: it tells the OS exactly which experts are needed (predictive prefetch via madvise(WILLNEED)) and which are not (madvise(DONTNEED)). Inactive experts are surgically evicted from memory.
Result: 48× I/O reduction. But more importantly — the inactive experts don't compete with active ones for CPU cache. The signal path is clean.
This is how a 226 GB model (Kimi K2.5, 1 trillion parameters) runs on a machine with 17 GB of RAM. Not by being clever about loading. By being precise about unloading.
4. Direct quantization support — 23 formats, native
Every quantization format has a different way of packing information into fewer bits. Most engines support a handful and convert the rest.
Inference-X has native dequantization and fused dot products for 23 formats — from Q2_K (2 bits) to FP32 (32 bits). No conversion step. No intermediate format. The engine speaks the model's native dialect.
| Format family | Variants | Block size | Bits/weight |
|---|---|---|---|
| K-quant | Q2_K, Q3_K, Q4_K, Q5_K, Q6_K | 256 | 2.6–6.6 |
| Standard | Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q8_1 | 32 | 4.5–9.0 |
| IQ (importance) | IQ1, IQ2, IQ3, IQ4 | varies | 1.0–4.5 |
| Float | F16, BF16, F32 | 1 | 16–32 |
5. 305 KB — the engine is invisible
A framework has opinions. It has abstractions. It has object hierarchies. Every layer of abstraction is a layer of interpretation between the model and the hardware.
Inference-X is 305 KB compiled. Header-only C++. No runtime. No garbage collector. No memory allocator. The binary is so small that the entire engine fits in L2 cache of a modern CPU.
The engine should be invisible. You should hear the model, not the engine. 305 KB is the engineering of disappearance.
What this means in practice
Better output quality at the same model size
Because the computation path is cleaner, the same Q4_K model produces output that is closer to its FP32 theoretical maximum through Inference-X than through engines with intermediate buffers and uniform precision.
This isn't a benchmark number. It's a property of the mathematics: fewer rounding operations = less accumulated error = higher fidelity to the original training.
Same output quality at a smaller model size
If a 7B model through Inference-X produces output quality comparable to a 7B through a standard engine, then a smaller model through Inference-X may match a larger model through a standard engine.
Less RAM. Less storage. Faster inference. Same answers.
Runs on hardware that "shouldn't" work
Kimi K2.5 (1T parameters, 226 GB) runs on 17 GB of RAM. Not in theory — in production. Because surgical expert management reduces the active memory footprint to what actually contributes to each token.
7B models run on Raspberry Pi. 3B models run on ESP32. The engine's minimal footprint leaves almost all system resources for the model.
The business case
Standard deployment:
Model: Llama-3.1-70B-Q4_K
Hardware: 128 GB server, 2× A100
Cost: ~$40,000/year (cloud)
Output quality: baseline
Inference-X deployment:
Model: Llama-3.1-70B-Q4_K (same model)
Hardware: 128 GB server, CPU-only
Cost: ~$2,400/year (Hetzner EPYC)
Output quality: equal or better (cleaner computation path)
Savings: 94%
Quality: maintained or improved
The savings come from two places:
- No GPU required (the engine runs efficiently on CPU with SIMD optimization)
- Adaptive precision reduces effective memory bandwidth by 10-26% for typical workloads
For MoE models, the advantage is larger:
Kimi K2.5 on GPU cluster:
8× H100, NVLink, ~$200,000/year
All 384 experts loaded, 376 idle per token
Kimi K2.5 on Inference-X:
256 GB EPYC server, ~$4,800/year
8 active experts loaded, 376 surgically evicted
Same model. Same answers. 97.6% cost reduction.
Who this is for
Edge deployments — Run models on devices without cloud connectivity. The 305 KB binary deploys to ARM, RISC-V, FPGA, microcontrollers.
Cost-sensitive inference — Replace GPU clusters with CPU servers for the same or better quality. Pay for RAM, not for CUDA cores.
Hardware manufacturers — Integrate Inference-X as the inference layer for custom silicon. One integration covers every model format.
Sovereign AI — Run national-language models on national infrastructure. No data leaves the country. No dependency on foreign API providers.
Research — Test models across 19 hardware targets from a single binary. Compare performance across architectures without rewriting code.
Try it
git clone https://git.inference-x.com/salka/inference-x
cd inference-x
make
./inference-x model.gguf -p "Hello"
One binary. One command. The model speaks directly.
The best inference engine is the one you don't notice. You should hear the model, not the framework.
◆