● wip MIT Rust Linux · macOS early WIP

LLM inference.
From the metal.

Pure-Rust inference engine, Vulkan-first — built to run on any mainstream GPU without a CUDA toolchain.

pull / run / serve / bench over one engine. OpenAI-compatible /v1 API with SSE streaming and tool calls — a local backend for hrdr, opencode, and Claude Code CLI. The only non-Rust parts are the Vulkan driver calls (ash) and the SPIR-V compute shaders.

view on github → why infr ↓

infr — serve

$ infr serve unsloth/Qwen3-14B-GGUF:Q4_K_M
INFO infr-server listening addr=127.0.0.1:8080

$ curl -N http://127.0.0.1:8080/v1/chat/completions \
    -H 'content-type: application/json' \
    -d '{"model":"qwen3-14b","stream":true,
         "messages":[{"role":"user","content":"hello"}]}'
data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":"Hey"}}]}
data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":" there"}}]}
data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":"!"}}]}
data: {"choices":[{"delta":{},"finish_reason":"stop"}]}
data: [DONE]

why

Any mainstream GPU. One unified runner. No C++ toolchain.

01 — PURE

Pure Rust

The only non-Rust parts are the GPU driver calls (Vulkan via ash) and the compute shaders (SPIR-V). No CUDA toolchain, no C++ build system.

02 — VULKAN

Vulkan-first, Metal too

AMD / NVIDIA / Intel via Vulkan cooperative-matrix matmul. Apple gets a native Metal backend (INFR_METAL=1) — simdgroup-matrix GEMM + flash attention, within ~1.3–1.5× of llama.cpp Metal on M3 Pro.

03 — API

OpenAI-compatible server

infr serve — axum + SSE. /v1/chat/completions and /v1/models, streaming deltas, tool calls with finish_reason: "tool_calls". Works with hrdr, opencode, and Claude Code CLI.

04 — MODELS

Dense, MoE, hybrid, diffusion

Llama, Qwen2/2.5, Qwen3 (dense + MoE), Gemma 3, Gemma 4 (dense + E2B), Qwen3.5/3.6 hybrid gated-DeltaNet, and DiffusionGemma block text-diffusion — all on one unified runner.

05 — PULL

llama.cpp-compatible refs

Model refs match llama.cpp's -hf: org/repo[:quant], quant default Q4_K_M. Models land in the standard HuggingFace Hub cache — one download, shared with llama.cpp. Chat template comes from the GGUF's own tokenizer.chat_template.

06 — BENCH

Benchmarks you can compare

infr bench matches llama-bench's -p/-n/-d/-r flags; infr compare runs both tools on coding-agent-shaped workloads. Competitive with llama.cpp at long context.

07 — PROFILE

Per-op GPU profiling

INFR_PROF2=1 prints GPU timestamp-query blocks per submit, tagged by op label — attn_flash, expert_gateup, lm_head, … Sum a label for its total.

08 — VRAM

MoE expert CPU offload

INFR_NCMOE=N offloads MoE experts to the CPU for tight VRAM. INFR_MAX_CTX, INFR_MAX_NEW, and INFR_TEMP / TOP_K / TOP_P (greedy at TEMP=0) tune the rest.

09 — SEAM

Backend + loader seams

Compute sits behind a Backend trait — Vulkan via ash + SPIR-V, native Metal via MSL, CUDA addable later. Weights behind WeightSource — GGUF now, safetensors later.

models

Nine architectures. GGUF in, tokens out.

All run on the Vulkan GPU backend unless noted. The chat template — turn markers, system prompt — is read from the GGUF's own tokenizer.chat_template.

gguf · llama

Llama

dense transformer

gguf · qwen2

Qwen2 / Qwen2.5

dense · QKV bias · NEOX rope

gguf · qwen3

Qwen3

dense · QK-norm

gguf · qwen3moe

Qwen3 MoE

softmax router · top-k experts · expert CPU offload

gguf · gemma3

Gemma 3

SWA + QK-norm + GeGLU · dual-RoPE

gguf · gemma4

Gemma 4 (dense)

per-layer head dims · proportional RoPE · V-norm

gguf · gemma4

Gemma 4 E2B

per-layer input embeddings / FFN widths · KV sharing

gguf · qwen35

Qwen3.5 / Qwen3.6

hybrid gated-DeltaNet + attention · CPU + Vulkan

gguf · diffusion-gemma

DiffusionGemma

block text-diffusion MoE · entropy-bound denoise · CPU + Vulkan

install

Early WIP — build from source.

No packaged releases yet. You need a Rust toolchain and a Vulkan driver (AMD / NVIDIA / Intel); on Apple, the native Metal backend is enabled with INFR_METAL=1. infr run and infr serve auto-pull the model if it isn't cached.

view on github → README ↗

infr — from source

$ git clone https://github.com/kryptic-sh/infr
$ cd infr
$ cargo build --release

# binary in target/release/infr

infr — first run

# terminal chat (auto-pulls the GGUF from HuggingFace)
$ infr run unsloth/Qwen3-1.7B-GGUF:Q4_K_M "What is bash?"

# MoE (expert CPU offload with INFR_NCMOE=N for tight VRAM)
$ infr run unsloth/Qwen3-30B-A3B-GGUF:Q4_K_M "Explain MoE routing."

# serve the OpenAI-compatible API
$ infr serve unsloth/Qwen3-14B-GGUF:Q4_K_M