LLM inference.
From the metal.
Pure-Rust inference engine, Vulkan-first — built to run on any mainstream GPU without a CUDA toolchain.
pull / run / serve /
multi / bench over one engine. OpenAI-compatible
/v1 API with SSE streaming and tool calls — a local backend for
hrdr, opencode, and Claude Code CLI. The only
non-Rust parts are the Vulkan driver calls (ash) and the SPIR-V
compute shaders.
$ infr serve unsloth/Qwen3-14B-GGUF:Q4_K_M INFO infr-server listening addr=127.0.0.1:8080 $ curl -N http://127.0.0.1:8080/v1/chat/completions \ -H 'content-type: application/json' \ -d '{"model":"qwen3-14b","stream":true, "messages":[{"role":"user","content":"hello"}]}' data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":"Hey"}}]} data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":" there"}}]} data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":"!"}}]} data: {"choices":[{"delta":{},"finish_reason":"stop"}]} data: [DONE]
why
Any mainstream GPU. One unified runner. No C++ toolchain.
Pure Rust
The only non-Rust parts are the GPU driver calls (Vulkan via
ash) and the compute shaders (SPIR-V). No CUDA toolchain,
no C++ build system.
Vulkan-first, Metal too
AMD / NVIDIA / Intel via Vulkan cooperative-matrix matmul. Apple gets a
native Metal backend (--dev metal) — simdgroup-matrix GEMM
+ flash attention, within ~1.3–1.5× of llama.cpp Metal on M3 Pro. A CPU
reference backend (--dev cpu) runs every architecture and
is what the GPU output is checked against.
OpenAI-compatible server
infr serve — axum + SSE.
/v1/chat/completions and /v1/models, streaming
deltas, tool calls with finish_reason: "tool_calls" (a
forced tool_choice is grammar-constrained through
llguidance). Optional bearer auth (INFR_API_KEY) gates both
/v1 routes but never /health. Works with
hrdr, opencode, and Claude Code CLI.
Dense, MoE, hybrid, diffusion
Llama, Llama 4 Scout, Qwen2/2.5, Qwen3 (dense + MoE), Gemma 3, Gemma 4 (dense + E2B + 26B-A4B MoE), Qwen3.5/3.6 hybrid gated-DeltaNet (dense + MoE), and DiffusionGemma block text-diffusion — all on one unified runner.
llama.cpp-compatible refs
Model refs match llama.cpp's -hf:
org/repo[:quant], quant default Q4_K_M. Models
land in the standard HuggingFace Hub cache — one download, shared with
llama.cpp. Chat template comes from the GGUF's own
tokenizer.chat_template.
Benchmarks you can compare
infr bench matches llama-bench's
-p/-n/-d/-r/-pg/-ngl flags and its
-o json shape; infr compare --sweep runs both
tools across your model cache. On an RX 7900 XTX, decode — the
reproducible half — wins 29 of 35 rows at tg128 and 24 of
35 at tg64@d4096; the full table names the losses too.
Per-op device profiling
INFR_PROF_OPS=1 timestamps every dispatch — on Vulkan,
Metal and CPU alike — labelled with its kernel name (plus role overrides
like expert_gateup): one block per submit, then one
aggregated report at exit. INFR_PROF_OP_SHAPES=1
itemizes the GEMV/GEMM buckets by shape.
Paged experts, spillable KV
MoE experts that don't fit stream through a paged VRAM cache —
INFR_CACHE=19g (or 80%) forces and sizes it,
which is how a 37 GB Llama 4 Scout runs on a 24 GB card. The KV cache
can be stored as q8_0 and can overflow to system RAM when
VRAM runs out.
Backend + loader seams
Compute sits behind a Backend trait — Vulkan via ash +
SPIR-V, native Metal via MSL, CUDA addable later. Weights behind
WeightSource — GGUF now, safetensors later.
One resolved config
Device, context, sampling, KV format, paging budgets, every kernel-tier
switch — one typed value resolved once at startup from four layers,
later wins:
defaults < infr.toml < INFR_* env < flags. Knobs
without a dedicated flag are reachable with
--set kernels.vulkan.flash_splits=2, and a flag that
collides with a --set wins and says so.
Concurrent slots, many models
infr serve --parallel N runs N generation slots, each with
its own KV cache, divided out of the same VRAM budget one slot fits —
raising it can't OOM a box that -np 1 fit.
infr multi qwen@Vulkan0 gemma@Vulkan1 hosts several models
on one server, each pinned to its own GPU, routed by model name.
One model, several GPUs
multi.pipeline and multi.tensor_parallel (both
want ≥ 2 devices) plus multi.expert_parallel split a single
model across cards, moving tensors GPU-to-GPU directly or staging them
through host RAM.
models
Twelve architectures. GGUF in, tokens out.
All run on the Vulkan GPU backend unless noted. The chat template — turn
markers, system prompt — is read from the GGUF's own
tokenizer.chat_template. Fine-tunes on any of these backbones
run unchanged — Ornith-1.0 and Ternary-Bonsai both ride existing skeletons
with zero code changes. infr is also the only engine that runs llama.cpp's
new Q2_0 ternary weight dtype (2.25 bpw) on a GPU, with
in-shader dequant and dp4a mmq — upstream merged it CPU-only.
install
Early WIP — build from source.
No packaged releases yet. You need a Rust toolchain and a Vulkan driver (AMD
/ NVIDIA / Intel); on Apple, pick the native Metal backend with
--dev metal. infr run and
infr serve auto-pull the model if it isn't cached.
$ git clone https://github.com/kryptic-sh/infr $ cd infr $ cargo build --release # binary in target/release/infr
# terminal chat (auto-pulls the GGUF from HuggingFace) $ infr run unsloth/Qwen3-1.7B-GGUF:Q4_K_M "What is bash?" # MoE (experts page through the VRAM cache when they don't fit) $ infr run unsloth/Qwen3-30B-A3B-GGUF:Q4_K_M "Explain MoE routing." # serve the OpenAI-compatible API $ infr serve unsloth/Qwen3-14B-GGUF:Q4_K_M