llama.cpp Overview and Router Mode

llama.cpp is an open-source inference engine written in plain C/C++ that runs large language models locally - no Python runtime, no cloud, no telemetry. This article gives a practical overview of the tool suite, the two operating modes of llama-server (inference and router), and the terminal commands to control a running server.

Table of Contents

What llama.cpp is

llama.cpp performs LLM inference on commodity hardware. Models are loaded from the GGUF format - a single-file container holding weights, tokenizer and metadata - which makes distribution as simple as copying one file (or a few shards).

Key properties:

Aspect Details
Language C/C++, compiled to standalone binaries, zero dependencies
Model format GGUF (*.gguf)
Quantization 1.5-bit to 16-bit (Q2_K … Q8_0, F16), trading quality for memory/speed
CPU x86 (AVX2/AVX-512), ARM (NEON)
GPU Apple Metal, CUDA, Vulkan, ROCm/HIP, SYCL
APIs Native HTTP API plus OpenAI-compatible endpoints

Why it matters: quantization is the core trick. A 27B model at Q4_K_M fits in ~15 GiB instead of ~54 GiB at F16, which is what makes local inference on workstations feasible at all.

The tool suite

A build produces several binaries under build/bin/. The most relevant ones:

Binary Purpose
llama-cli Interactive chat / one-shot completion from the terminal
llama-server OpenAI-compatible HTTP server with built-in web UI
llama-quantize Convert a model to lower precision
llama-bench Benchmark tokens/s across configurations
llama-perplexity Evaluate model quality on a dataset
llama-mtmd-cli Multimodal CLI (images/audio with vision models)
llama-embedding / llama-rerank Embeddings and reranking

Building from source with CUDA support:

1
2
3
4

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=on
cmake --build build --config Release -j$(nproc)

Prebuilt releases for Linux/macOS/Windows are available on the GitHub releases page if you do not want to compile.

Getting models

Models come from HuggingFace as GGUF files. The fastest way is the built-in -hf flag, which downloads straight into the llama.cpp cache (~/.cache/llama.cpp, override via the LLAMA_CACHE environment variable):

1
2

# Download and run in one step; :Q4_K_M picks the quantization
llama-server -hf ggml-org/Qwen3-4B-GGUF:Q4_K_M

Anything downloaded this way is automatically discovered by router mode later. For authenticated bulk downloads of large GGUFs, see the dedicated Linux article linked at the bottom.

llama-cli: quick interactive testing

Before setting up a server, sanity-check a model directly:

1
2
3
4
5

# Interactive chat (conversation mode is default for instruct models)
./build/bin/llama-cli -m models/Qwen3-4B-Q4_K_M.gguf -ngl 99

# One-shot completion from a prompt
./build/bin/llama-cli -m models/Qwen3-4B-Q4_K_M.gguf -p "Explain KV caching in three sentences:" -n 128

Common flags: -ngl N offloads N layers to GPU (999 = everything), -n limits generated tokens, -c sets the context size.

llama-server: the HTTP API

llama-server exposes the model over HTTP and ships with a web UI on the same port:

1
2
3
4
5

./build/bin/llama-server \
  -m models/Qwen3-4B-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8080 \
  -c 8192 -ngl 99 \
  --jinja            # proper chat templates + tool calling

Important flags:

Flag Meaning
-m FILE Model to load (presence or absence decides the mode, see below)
--host / --port Bind address, default 127.0.0.1:8080
-c, --ctx-size Context window per slot
-ngl, --n-gpu-layers Layers offloaded to GPU
--api-key KEY Require a bearer token
--jinja Use the model’s Jinja chat template (needed for tool calling)
--flash-attn on Flash attention, saves KV-cache memory
-np, --parallel Number of concurrent slots
--alias NAME Name reported via /v1/models

Key endpoints

Endpoint Purpose
GET /health Liveness: 200 when ready, 503 while loading
POST /v1/chat/completions OpenAI-compatible chat endpoint
POST /v1/completions OpenAI-compatible raw completion
GET /v1/models List available models
POST /completion Native endpoint with full llama.cpp option set
GET /props Model metadata and server configuration
GET /slots State of each processing slot
GET /metrics Prometheus metrics (needs --metrics)
POST /tokenize / POST /detokenize Text <-> token conversion

Any OpenAI client works by changing only the base URL:

1
2
3
4
5
6
7
8

import openai

client = openai.OpenAI(base_url="http://localhost:8080/v1", api_key="none")
resp = client.chat.completions.create(
    model="local-model",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)

Router mode: one server, many models

Since December 2025, llama-server supports a second operating mode. The rule is simple:

With a model flag (-m / -hf) it runs in inference mode serving exactly one model. Without a model flag it starts in router mode and manages multiple models dynamically.

Router mode turns the server into a dispatcher that spawns one child llama-server process per requested model:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

                     clients (curl, IDE, Web UI, Open WebUI ...)
                                  |
                                  v
                  +-------------------------------+
                  |  llama-server  :8080  ROUTER  |
                  |  (started without -m/-hf)     |
                  +---------------+---------------+
              loads on demand     |      routes by the
              LRU eviction        v      request's "model" field
              +-------------+  +-------------+  +-------------+
              | child :rand |  | child :rand |  | child :rand |
              | Model A     |  | Model B     |  | Model C     |
              | (unloaded)  |  | (loaded)    |  | (loaded)    |
              +-------------+  +-------------+  +-------------+

How it behaves

  1. Discovery - the router finds models in the llama.cpp HF cache by default, in a directory (--models-dir) or from a preset file (--models-preset). Restarting the router refreshes the list after you add files manually.
  2. On-demand loading - a request naming an unloaded model triggers a load; first request pays the load time, subsequent ones are instant.
  3. Routing by name - the "model" field of the OpenAI request selects the backend. Client code never changes.
  4. LRU eviction - when --models-max (default 4) loaded models is exceeded, the least-recently-used one is unloaded automatically.
  5. Crash isolation - every model runs in its own process; a crashing model does not take down the router or other models.

Model lifecycle states: downloading -> downloaded -> unloaded -> loading -> loaded (plus sleeping for idle-unloaded).

Router flags

Flag Default Description
--models-dir PATH disabled Directory scanned for GGUF files
--models-preset PATH disabled INI file with per-model settings
--models-max N 4 Max simultaneously loaded models
--no-models-autoload off Load only via explicit POST /models/load
--sleep-idle-seconds N disabled Auto-unload models idle for N seconds
--fit on/off on Auto-adjust context to fit device memory

Starting the router:

1
2
3
4
5
6
7
8
9
10
11

# Simplest form: discover everything in the HF cache
llama-server --port 8080

# Production-style: explicit directory with limits
llama-server \
  --models-dir /var/lib/llama.cpp/models \
  --models-max 2 \
  --sleep-idle-seconds 900 \
  --host 127.0.0.1 --port 8080 \
  --api-key secret123 \
  -c 8192 -ngl 99 --flash-attn on --jinja

Global flags like -c or -ngl are inherited by every child; per-model overrides go into a preset INI:

1
2
3
4
5
6
7
8
9
10
11
12
13

[*]
n-gpu-layers = 999
flash-attn = on
jinja = true

[coder-model]
c = 131072
parallel = 2
temp = 0.7

[chat-model]
c = 32768
temp = 0.9

Each section name becomes the model identifier clients use in the "model" field.

Terminal control commands

Everything below assumes a running server on localhost:8080; add -H "Authorization: Bearer $KEY" when --api-key is set.

Check readiness (public, works while loading - 503 means not ready yet):

1
2

curl http://localhost:8080/health
# {"status":"ok"}

List all models known to the router:

1

curl -s http://localhost:8080/v1/models | jq .

Detailed status including load state per model:

1

curl -s http://localhost:8080/models | jq '.models[] | {name, state}'

Load / unload explicitly (useful with --no-models-autoload):

1
2
3
4
5
6
7

curl -X POST http://localhost:8080/models/load \
  -H "Content-Type: application/json" \
  -d '{"model": "coder-model"}'

curl -X POST http://localhost:8080/models/unload \
  -H "Content-Type: application/json" \
  -d '{"model": "coder-model"}'

Chat completion - switching models is just the "model" field:

1
2
3
4
5
6
7
8
9

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "chat-model",
    "messages": [
      {"role": "system", "content": "You are concise."},
      {"role": "user",   "content": "Summarize what a KV cache does."}
    ]
  }' | jq -r '.choices[0].message.content'

Native completion with full parameter control:

1
2
3

curl http://localhost:8080/completion \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Building a website can be done in 10 simple steps:", "n_predict": 128}'

Inspect internals:

1
2
3

curl -s http://localhost:8080/props | jq .          # template, modalities, defaults
curl -s http://localhost:8080/slots | jq .           # slot occupancy, cached tokens
curl -s http://localhost:8080/metrics                # Prometheus format (needs --metrics)

Stop the server: it runs in the foreground unless daemonized, so Ctrl+C or kill <pid>; under systemd use sudo systemctl stop llama-router.

Running as a systemd service

For a persistent homelab deployment:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

[Unit]
Description=llama.cpp Router - Multi-Model Inference Server
After=network.target

[Service]
Type=simple
ExecStart=/opt/llama.cpp/build/bin/llama-server \
  --models-dir /var/lib/llama.cpp/models \
  --models-preset /etc/llama.cpp/models.ini \
  --models-max 2 \
  --host 127.0.0.1 --port 8080 \
  -ngl 999 --flash-attn on --jinja --metrics
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Prefer systemctl restart over systemctl reload after editing the preset file - a HUP signal can kill the router outright if the INI fails to parse.

Router mode vs Ollama vs llama-swap

Feature llama.cpp router Ollama llama-swap
Built-in yes separate product external proxy binary
Dynamic loading yes yes yes
LRU eviction yes (--models-max) TTL-based configurable timeouts
Per-model config INI preset Modelfile YAML
Process isolation per model yes (child processes) yes yes
Runtime control maximum, all native flags opinionated high
Maturity newer mature stable

Rule of thumb: pick the built-in router for minimal stack complexity and full control over every llama.cpp flag; reach for llama-swap when you need battle-tested production behavior today.

See also

Deep dive into authenticated GGUF downloads and KV-cache sizing for models.ini: Linux - llama.cpp router: GGUF dw and tuning.

Connecting VS Code Copilot to local model servers: Github Copilot - Overview.