Cache hit rate is a harness quality metric

We build production GenAI for operations, knowledge, and data. That sentence still constrains the loop. It also constrains the bill. On a long agent session the model is not only answering; it is re-reading the same system blocks, the same tool schemas, and the same workflow pack on every turn. If those blocks stay byte-stable, the gateway can reuse what it already paid to encode. If they churn, you pay again. Cache hit rate is not a purity contest. It is a harness quality metric.

Teams talk about agents as if the model invents efficiency. It does not. Efficiency shows up when the harness stops rewriting the prefix, stops renaming tools mid-run, and stops stuffing a new memoir into every message. Prompt stability, tool-trace shape, and cache reuse are three sensors on the same product. Score them. Or keep guessing why the floor feels slow while the token meter climbs.

Cache hits are a harness score

A prompt cache rewards sameness at the front of the request: stable system instructions, stable tool definitions, and a stable role layout. Variable work (the ticket, the diff, the schema row) belongs later, where it belongs. When the front stays fixed and the tail carries the job, repeated enterprise workflows start to look like the same conversation to the encoder. That is how you earn reuse without faking a benchmark.

This is not another essay arguing that an agent is a harness. You already know the loop: gather, act, verify. The extension is narrower. Once the loop exists, the way you lay out messages and tools becomes a measurable surface. Hit rate. Prefix churn. Whether the tool schema in turn 17 still matches turn 1, and whether the observability rail can show a reviewer which block stayed hot and which block forced a miss.

We already see the consequence on repeated enterprise workflows: about 68% prompt-cache hit when reusable context sits behind an in-network gateway, alongside 42% lower token spend and 2.4× useful completions per 1k tokens. Those numbers are not a reason to skip verify. They are what happens when the expensive model sees a slim, stable pack instead of a wandering chat. Treat cache hit as a KPI when your tooling exposes it. Do not treat a lab anecdote as a Sunrise scorecard.

Read the figure the way a reviewer will read a cost spike. The left column is boring on purpose: fixed blocks, named schemas, a message order that puts the stable prefix first. The right column looks busy and helpful. It rewrites the system prompt because someone added a tip, renames a tool because the JSON looked prettier, and shoves the ticket into the system block because that seemed convenient at 11pm. The encoder sees a new prefix. Miss. The loop still has to gather, act, and verify. You just paid twice for the privilege.

What an engineered layout actually locks

Stability is not poetry. It is a short list of surfaces you refuse to mutate mid-run.

System blocks stay versioned files. STACK, PROJECT, TASK, CONVENTIONS. Or whatever names your estate already uses. The point is that the agent loads a known pack instead of inventing a new preamble every turn. When you launch a coding agent, prefer inspect, then plan, then the smallest correct change, then verify. That sequence is not theater. It keeps the expensive model from rewriting the world before it has read the file.

Tool schemas stay deterministic. Same name, same argument shape, same failure shape. If the tool contract changes, bump a version and treat it as a deploy, not as a chat edit. A reviewer should be able to open the trace and see that read_file in turn 40 is still the contract from turn 1. Drift in the schema is a silent cache killer and a silent permission killer.

Message layout stays cache-friendly. Put the durable instructions and tool definitions where the encoder expects a stable prefix. Put the ticket, the diff hunk, and the local evidence in the variable tail. Slim context packs already taught this lesson for knowledge work: the model should see what the job needs, not the estate memoir. Cache-friendly layout is the same idea applied to the wire format.

The loop still ends in verify. Cache hits do not certify correctness. A hot prefix can still drive a bad edit. Numeric gates on coding diffs, citation checks on retrieval, schema scores on mapping queues: those gates stay. Mapping work still only shows up to 60% less time on schema and mapping, and 3× throughput on attribute mapping, when the model proposes inside a scored contract. Speed without a gate is just a faster wrong answer.

What a vibe harness burns

A vibe harness looks busy for about one afternoon.

Someone pastes a better system prompt into the live session, and the cache misses for every open thread that shared the old prefix. Someone adds a tool "just for this ticket" with a slightly different name. The schema fingerprint flips. Someone merges yesterday's failure notes into the system block instead of into a skill pack or a guide file, so the prefix grows and every later turn pays for the growth again.

Then the bill arrives. Token spend climbs while useful completions stay flat. The floor asks why the same workflow that was cheap on Tuesday is expensive on Thursday. The answer is almost never "the model got worse." The answer is that the harness stopped being a product and became a scratchpad.

Incident desks feel this as latency and as trust. A ranked hypothesis with evidence and a person on the last step is still how you get 75% lower mean time to recovery. A vibe harness that rewrites its own sensors mid-incident does not get you there. It gets you a long trace of re-explained context and a late page.

Three sensors worth putting on the rail

If you only watch latency and final success, you will misdiagnose harness bugs as model bugs.

Prompt-cache hit rate. When the gateway or provider exposes it, log it per workflow, not as a single vanity number. A repeated mapping job should look different from a one-off research chat. Track the hit. Track the miss reason when you can: prefix change, tool schema change, forced refresh.

Prompt stability. Hash the durable blocks. If the hash flips between turns without a deliberate version bump, that is a defect. Stability is not "never change the prompt." Stability is "change it like software, with a diff a person can approve."

Tool-trace shape. Named tools. Bounded retries. Argument shapes that match the schema id. A trace that looks like a random walk of anonymous function calls is a miss waiting to happen, even when the cache is warm. Shape is how a reviewer reconstructs the run after a miss.

These sensors sit beside the same observability rail we already demand for permissions and evidence. They do not replace skills, slim packs, or numeric gates. They tell you whether the harness is earning reuse or burning it.

What breaks when cache is an afterthought

The mutating preamble. Every turn appends a tip, a warning, or a pep talk to the system block. The model sounds careful. The encoder sees a new world every time.

The anonymous tool zoo. Thirty tools, soft names, optional arguments that mean different things on different days. The schema never fingerprints cleanly. Permissions become a story instead of a list.

The prefix stuffed with the job. The ticket body, the full schema dump, and last week's chat sit in the system section because someone wanted them "always visible." Visibility without layout discipline is just unpaid re-encoding.

The hot miss that still ships. Cache hit looks green. Verify is weak or missing. Leadership celebrates efficiency while the floor backs out a bad write. Cache is not a test gate.

The lab number as a product claim. Someone quotes an extreme hit rate from a private stack and pins it on your roadmap. You do not have their harness. Measure yours.

Each of these is a harness bug. Buying a larger context window will not fix them.

Tradeoffs you actually have to make

Locking layouts costs design time. Versioning tool schemas costs process. Putting job facts in the tail means the model must gather instead of assuming the world is already in the system prompt. That can feel slower on the first turn.

Pay it. The alternative is repay on every turn.

Another trade: freshness versus hit rate. Sometimes the runbook changed an hour ago and the prefix must bump. Bump it on purpose. Write the version into the rail. A deliberate miss is cheaper than a silent stale instruction that ships the wrong remediation.

A third trade: one shared prefix versus many workflow-specific packs. A single mega-prefix is easier to cache and harder to reason about. Prefer small, named packs the gateway can route; composition beats a monolith that nobody dares to edit.

We still do not know how every provider will expose hit diagnostics next year. The rule that does not move is simpler. If you cannot point to a stable prefix, a schema id, and a verify step, you are not managing cache. You are hoping.

What production demands

Production wants reuse you can defend in a review.

Keep durable instructions and tool schemas in versioned files. Lay messages so the stable prefix stays stable. Put the ticket and the local evidence in the variable tail. Run inspect, plan, small change, verify. Log hit rate, prefix hash, and tool-trace shape when the tooling exposes them, and refuse to call a cache hit a done signal.

At sunrisegenai.com the work is still the harness. The model is how the harness speaks. Cache is how the harness stops paying twice for the same speech. If your hit rate is a mystery and your tool schemas drift by accident, you do not have a quality-engineered loop yet. You have a vibe with a receipt.