Writing
Evidence-backed notes on agent harnesses, evaluation, and verification — directional case studies, not leaderboards.
I am the bottleneck
From VS Code panes to an agent multiplexer, I'm still the message bus between my agents. What breaks when agents multiply, and a hypothesis for a StarCraft-inspired control plane that shows the work and sends people only the decisions.
Don't judge an agent by its pass@1
A single pass@1 score collapses runs that differ on three axes it can't show: how a failure was shaped, whether a success repeats, and what either one cost. One model concluded a secret-safety task in 4 turns; two others avoided the leak but never concluded, burning ~77× the tokens. Then the reliability metric I reached for turned out to move in coarse sixths, and inverted when I reran the identical matrix. What to log, and how far to trust it.
Is your harness driving your model crazy?
One model went from a 0.10 to a 0.75 success rate without anyone touching the model. The fault was a tool-schema bug in the harness, and the only reason I caught it was reading the agent's trajectory instead of the scoreboard. A case study on why agent evals score the whole system, and how to tell which part you're actually measuring.
When is an agent truly done?
Many agent loops still treat a model's completion message as terminal, but stopping isn't the same as being correct. On moving the authority for 'done' out of the model and into a verifier, and what that one move makes measurable.