Pi coding agent: system prompt, AGENTS.md and KV cache

This article explains how a coding agent such as Pi builds its system prompt, how your own instruction files (SYSTEM.md, AGENTS.md) fit in, and what happens under the hood with the KV cache of a local model server such as llama.cpp. Understanding this has a direct effect on two things you can feel immediately: response latency and context consumption.

The ideas are general and apply to most LLM coding agents. Pi is used as the concrete example throughout.

No API keys, secrets or machine-specific paths are required to understand this. Everything here is general and portable.

1. The effective system prompt is assembled, not fixed

The prompt a model sees is not a single static file. It is assembled from several building blocks at the start of every session.

graph TD
    A["Base SYSTEM.md<br/>(the core instructions)"] --> C[Assembled prompt]
    B["AGENTS.md / CLAUDE.md<br/>(global first, then project)"] --> C
    D[Skills section] --> C
    E[Current working directory] --> C
    C --> F[+ Full conversation history]
    F --> G[+ The new user message]
    G --> H[Single API request sent to the model]

The four static building blocks are:

Block Purpose
Base system prompt The built-in core: role, available tools, guidelines, documentation links.
AGENTS.md / CLAUDE.md Your instruction files. Loaded global first, then project (closer to the working directory = higher priority).
Skills Specialized, task-specific instructions.
Working directory The current working directory (cwd) so the agent knows where it runs.

The two dynamic parts — conversation history and the new message — are appended on top of the static base for every request.

SYSTEM.md vs AGENTS.md — what is the difference?

File Role
SYSTEM.md Replaces/extends the core system prompt (the agent’s base behavior).
AGENTS.md An appended context block with project/user rules. It does not replace the core — it is added to it.

Both end up in the same assembled prompt. AGENTS.md is simply appended as a <project_context> section after the base prompt.

2. When is the prompt sent?

Every single model request within a session carries the full assembled prompt. It is not sent once and forgotten — it is prepended to each request.

sequenceDiagram
    participant TUI as Client / TUI
    participant S as llama.cpp Server (KV cache slot)
    participant M as Model

    Note over TUI,M: Turn 1 (cold)
    TUI->>S: Request [system][history][msg1]
    S->>M: Prefill ALL tokens (cache_n = 0)
    M-->>S: Generate tokens
    S-->>TUI: Response + cache_n / prompt_n

    Note over TUI,M: Turn 2 and later (warm)
    TUI->>S: Request [system][history][msg1][msg2]
    S->>S: Reuse cached prefix (system + history)
    S->>M: Prefill only the new tokens (cache_n > 0)
    M-->>S: Generate tokens
    S-->>TUI: Response + high cache_n

The prompt is rebuilt from disk only on specific events, not per turn:

  • Session start
  • The set of active tools changes (new/removed extensions)
  • An extension overrides the prompt (session_system_prompt handler)
  • After compaction (when the context window fills)
  • Session reload / new session

Practical consequence: editing AGENTS.md or SYSTEM.md during a session does not take effect immediately. The content sits in a cached base prompt. It becomes active only after a rebuild event or a new session.

3. KV cache and prompt caching in llama.cpp

This is the part that most affects latency. llama.cpp stores the computed keys and values (KV cache) of every token in a slot. Because each entry is positional — it depends on everything before it — a new request can reuse the cached entries for any prefix it shares with the previous request.

The unit of reuse is a prefix. Tokens 0..k are reused only if they are the same tokens in the same order. From the first difference onward, every subsequent entry was computed against a context that no longer applies.

What the server reuses

With --cache-prompt (enabled by default on llama-server), the common prefix does not have to be reprocessed — only the suffix that differs.

The savings, quantified

Let R be the prefill rate in tokens/second, S the system-prompt tokens, and Q the new-message tokens:

1
2
3

cold prefill = (S + Q) / R
warm prefill = Q / R          # cache_n = S, prompt_n = Q
saved        = S / R

Worked example (R = 900 tok/s, S = 2000, Q = 40):

1
2
3

cold = 2040 / 900 = 2.27 s
warm =   40 / 900 = 0.04 s
saved          = 2.22 s per request

The saving is proportional to the shared prefix and independent of the answer length. That is why caching matters most where time-to-first-token dominates: a long instruction block with short questions against it.

Reading the cache hit on the server

The response timings object reports the numbers directly:

Field Meaning
cache_n Tokens reused from the KV cache
prompt_n Tokens actually processed (prefilled)
context prompt_n + cache_n + predicted_n

A high cache_n means the prefix (system prompt + history) is being reused efficiently.

4. What throws the cache away

Reuse is positional, so anything that changes an early token invalidates the whole prefix. This is more than a few milliseconds — it can invalidate the reuse of the entire conversation history.

graph TD
    Q[New request arrives] --> C{Prefix matches<br/>cached tokens?}
    C -- Yes --> W[Warm: reuse KV cache]
    C -- No --> X{What diverged?}
    X --> V[Volatile token at top<br/>timestamp / session id]
    X --> S[Different slot under concurrency]
    X --> O[Context window full<br/>eviction / re-prefill]
    X --> T[Chat template / format change]
    V --> R[Re-prefill the entire prompt]
    S --> R
    O --> R
    T --> R
    R --> C1[Cold: full prefill cost]
    W --> C2[Warm: only the suffix is prefilled]
What breaks the cache Why
A volatile token at the top (timestamp, session id) One differing token at position 12 invalidates positions 12 onward — the entire prompt.
A different slot (concurrency) The cache lives in the slot, so a request routed elsewhere starts cold.
Context window full Eviction forces a re-prefill. Resuming a long session near the context limit is especially slow.
Chat template / format change The template turns messages into tokens, so a different rendering invalidates every cached prefix.

--cache-reuse N softens the first rule: instead of giving up at the first divergence, the server reuses chunks of at least N tokens that appear after it (default 0 = off).

5. Best practices

These follow directly from the mechanics above.

  1. Keep instruction files short. A smaller system prompt S means a smaller one-time prefill cost and more KV-cache room for the actual conversation.
  2. No volatile material at the top. Never put a live timestamp, session id, or anything that changes per request at the beginning of SYSTEM.md. Put volatile content at the end — the suffix — so the prefix survives.
  3. Stable, deterministic rules. Keep the prompt content fixed turn over turn; do not reword it per request.
  4. Beware the context limit. As the conversation grows, eviction eventually forces a re-prefill. Compaction or a fresh session restores efficiency.
  5. Prefer purpose-built tools over shell. This is both a visibility/review benefit and a prompt-caching benefit (deterministic tool output).

6. Measuring the cache hit rate

Many TUIs show a cache hit rate in the footer. It is computed from the server usage stats as:

1
2

promptTokens = input + cacheRead + cacheWrite
cacheHitRate = (cacheRead / promptTokens) * 100
  • cacheRead — tokens reused from the cache
  • cacheWrite — tokens newly written to the cache
  • input — regular (uncached) prompt tokens

A reading of 96 %+ means 96 % of the prompt tokens in the latest request came from the KV cache — the system prompt and history are being reused, and the server is not reloading everything each time.

The value reflects the latest request, so it can fluctuate turn over turn. It is the server-side KV-cache hit rate reported by llama.cpp.

Summary

  • The effective prompt is assembled from a base system prompt, your AGENTS.md/CLAUDE.md files, skills, and the working directory.
  • It is sent on every request, but only rebuilt on specific events.
  • llama.cpp reuses the KV cache prefix across turns — the system prompt is prefilled once, not every request.
  • The cache breaks on volatile early tokens, slot changes, and context overflow. Keep instruction files short and stable to maximize the hit rate and minimize latency.