The day was about building something specifically to make cost visible, and then watching it report a cost of zero on every request it saw. I stood up a mixed-tier model proxy on the first node: four tiers, two mapped to subscription-billed models and two to a third-party provider with hard spend caps, $20 per 30 days on one and $10 on the other. It's backed by Postgres rather than run database-less, on purpose — the entire point of the thing is a spend record and an enforceable cap, and without the database neither exists. Twenty-one real requests went through the capped tiers. Every single one logged at 0.
The config was correct. The cost fields were in place, mapped to the right tiers, nothing missing. The actual fault was one layer down: the systemd unit's stop step ran, reported success, and did not stop the container. systemctl restart returned clean every time, but the container's own start time never moved, so the config that should have loaded on restart just sat there unread. The service was healthy. The thing the service was supposed to be managing hadn't been touched in who knows how long.
I found it by checking the container's uptime instead of the service's status, which is the only reason this got caught at all — systemctl status had nothing to say about it, because from its own point of view nothing was wrong.
A health check that checks the wrong layer
The lesson worth keeping is specific: a health check that only asks the service manager whether the service is running proves nothing about the container underneath it. systemctl was telling the truth about the unit and nothing about the workload. Once the container was actually restarted, spend started accumulating correctly and the numbers matched the requests going through.
The second half of the lesson is the one that's bitten this lab before: a zero is not a reading. It usually means a broken query, and a broken query looks exactly like "nothing has happened yet" — which is the one shape of failure that doesn't announce itself, because idle and broken produce the same number. Twenty-one requests through a system built to make cost visible, and the visible number was indistinguishable from silence.
Testing the cache instead of reasoning about it
Somewhere in the same day I'd also assumed a roughly 4,000-token prompt prefix was failing to cache, based on nothing more than "it doesn't look cheaper." Instead of reasoning about it further I tested it properly, across three variants — prefix in the system message, prefix in the user message, and an explicit caching header — at matched timestamps.
Caching works. It runs at about 91% of input tokens once the prefix clears roughly 10.7k tokens. The 4k prefix I'd been suspicious of was simply under the threshold the whole time, which is a less interesting bug than a broken cache and a more honest answer. I cross-checked it against the provider's own billing rather than trusting the proxy's own accounting a second time in one day: the more expensive tier billed at roughly 3.6× cheaper than its declared rate, which is the caching discount showing up as real money rather than as a number I generated myself. The 10.7k figure is now written down instead of remembered as a feeling.
The hostname that only breaks when something moves
Three containers had to migrate off the first node to make room for the proxy VM, 40 to 120 seconds of downtime each, one held up by a snapshot that had to be deleted to allow the move and recreated right after as a rollback point. I checked the public-facing tunnel container was serving alone from its new home before calling any of it done — two public hostnames, both 200.
What the move actually broke was somewhere else entirely: an update-rehearsal pipeline, because four separate scripts had the old node's hostname written directly into them. Three turned up right away. The fourth surfaced later, on its own schedule, which is the part worth naming — a hostname hardcoded into a script is a landmine that sits inert until the day something moves, and then it goes off with no warning attached to the move itself. It's also why the operational runsheets got renamed off host-in-filename this week: the name in the filename was recording something that changes, and every migration since has meant renaming documentation instead of just updating a field inside it.
A separate, smaller version of the same mistake showed up during a vault restructuring the same week: a bulk path rewrite that was right about the pattern and wrong about the boundary, and it caught the session directory of the editor doing the rewriting because the path merely looked similar to the intended target. Reverted, re-run with a tighter pattern.
One thing stayed open at the end of the day. One tier's routing name is still unmapped on the client side — a display name the proxy itself doesn't recognize — and a provider key that got exposed into an agent's transcript during the caching tests was rotated the same day, the old one dead. Both are closed enough to leave alone until tomorrow, not closed enough to call finished.