When is an agent truly done?

Many agent loops still treat a model’s completion message as terminal. You ask for a change, it edits some files, says “Done, I’ve implemented the feature,” and the loop closes. Whether the change is actually correct is a separate question the agent never had to answer.

That gap bothered me enough to build around it. If “done” just means “the model decided to stop,” then the most load-bearing word in the entire loop is a claim the model grades itself on. A self-graded claim is one you can’t measure, can’t trust, and can’t learn from.

Done is a claim, not an event

So I built a small coding-agent harness on a single inversion: the model proposes actions; the harness disposes of them. The model can read files, edit them, run tests, but it cannot end the task. A run is only complete when an external verifier says so: the tests pass, the diff actually exists, the linter is clean. “Done” stops being something the model announces and becomes something the harness has to confirm.

The verifier isn’t a general-purpose judge of correctness. It runs a specific check fixed before the model starts editing: a plan the harness owns and freezes up front, either a test command you configure or one discovered deterministically from the repo. The model can propose “done”; it can’t choose, edit, or soften the check that disposes of it.

It’s a deliberately small idea. Most of the machinery around it, like how state is tracked and how permissions gate which tools are live, I’ll leave to the repository; it isn’t the point. The point is the one move: take the authority for “done” out of the model, and give it to something that can actually be checked.

What the verifier makes visible

The principle is easy to state; what it catches is the interesting part. Here is one run, taken straight from the journal: the append-only log of every step, recording the action the model submitted, the tool output it got back, and the verifier’s verdict. It is the observable decision-and-evidence trail, not a reconstruction of the model’s hidden reasoning.

If you have never seen one, here is a whole short run from a different task (building a command-line chatbot), trimmed to one field per line so the shape is visible:

{"type": "agent_start",      "goal": "Create a runnable Python command-line chatbot in chatbot.py ..."}
{"type": "model_decision",   "action_type": "tool_call",    "action": "list_files({'glob': '**/*'})"}
{"type": "tool_end",         "tool": "list_files", "success": true, "summary": "0 file(s) matching '**/*'"}
{"type": "model_decision",   "action_type": "tool_call",    "action": "write_file({'path': 'chatbot.py', ...})"}
{"type": "model_decision",   "action_type": "final_answer", "action": "Created chatbot.py, a runnable CLI chatbot using the openai SDK ..."}
{"type": "verification_start"}
{"type": "verification_end", "passed": true, "summary": "verification passed"}
{"type": "agent_end",        "outcome": "success"}

Read it top to bottom: the model proposes an action, the harness records the tool result it got back, and at the very end a verifier (not the model) stamps the run passed and the loop closes. The run below is the same shape, but the verdict in the middle is the interesting one.

I gave a weak model (gpt-oss-20b) a small, concrete job. The workspace held a file, data.csv, whose three columns were mostly numbers with a few text values mixed in:

A,B,C
-1913,-347,-299
1.5,2.5,6.0
foo,3,4
...

and a helper function it was told to use, col_sum:

def col_sum(values):
    try:
        total = values[0]
        for v in values[1:]:
            total = total + v
        return total
    except TypeError:
        return "".join(str(v) for v in values)

The task: for each row, add a column D equal to col_sum of A, B and C, and write the rows out. The catch lives in that helper. Cells read from a CSV are text, and in Python "10" + "20" is "1020", not 30. To get a real sum you first have to turn the numbers into numbers, and only let the string behavior through for the genuinely non-numeric rows like foo,3,4.

I ran the harness on this task with the verifier wired to a grader the model could not see. The exact invocation, the same one written up in the harness repo, was:

export AVATAR_MODEL=openai/gpt-oss-20b
export AVATAR_WORKSPACE_ROOT="$SCRATCH/workspace"
export AVATAR_TEST_COMMAND="pytest -q $SCRATCH/contract/grade.py"
export AVATAR_MAX_REPAIR_ATTEMPTS=2
uv run avatar --task-kind edit --log "$SCRATCH/journal.jsonl" \
  "Read data.csv. Write a script pipeline.py that adds a new column D holding the sum of
   columns A, B and C for each row, and writes all rows to out.csv. Use the col_sum
   function from summation.py. Give your final answer when you are confident it is correct."

--task-kind edit hands the run the strict edit contract: the model can read and write only workspace/, while the grader and its answer key sit in a separate contract/ folder its file tools cannot reach. After a rejection it gets two repair attempts. Everything below comes from the journal.jsonl that invocation produced.

The model wrote a script and, without ever running it, reported success:

Created pipeline.py that reads data.csv, computes column D using col_sum, and writes out.csv. The script converts numeric values to float, handles non-numeric by string concatenation, and writes all rows with the new D column.

Confident, specific, and every sum in that file was correct. It still wasn’t done. The harness ran the verifier, and the verdict went into the journal:

verification_end   passed: false   "verification failed: ['tests']"

The model couldn’t just reassert itself. It re-ran the failing check from inside the loop, and the journal caught both the grader’s complaint and the model’s own wrong output side by side:

run_tests   exit=1
E   ValueError: could not convert string to float: 'foo34'

out.csv (excerpt)        the D column the model actually wrote
1.5,2.5,6.0,10           ok, a pure-number row
foo,3,4,foo3.04.0        should be foo34
12,bar,7,12.0bar7.0      should be 12bar7

The bug is hiding in the line the model was proudest of. Converting every value to a float turned 3 into 3.0, so on the rows that mix text with numbers, foo,3,4 came out as foo3.04.0 instead of foo34. The arithmetic was flawless. The one case the model never thought to question was wrong, and nothing in its own summary hinted at it.

So it fixed the conversion, kept whole numbers whole, and declared done again, this time to a verifier that agreed:

verification_end   passed: true   "verification passed"

That is the whole arc, and it is the point of the inversion: the model presumed it was done, an external check disproved it, and the model used that result to make the output actually correct. None of it relied on the model catching its own mistake, which it never did. It relied on something outside the model checking, and feeding the verdict back into the loop.

The full run, in strict edit mode with gpt-oss-20b and the frozen check pytest -q grade.py supplied up front, along with the fixture and a step-by-step reproduction, is in the harness repo.

What I’m not sure of yet

I want to be honest about the edges. This is one harness, early in its life, exercised on a handful of tasks: directional, not a verdict on anything or anyone. Making the verifier the authority is clean when “correct” looks like “the tests pass.” It gets genuinely hard for work that has no test-shaped definition of done, and I don’t claim to have solved that.

There’s also a deeper problem I keep circling. A verifier is only as trustworthy as it is hard to fool, and the moment an agent can influence the very thing that grades it, the whole arrangement starts to wobble. I don’t have that fully worked out. It’s one of the threads this blog exists to pull.

So, when is an agent truly done?

Not when the model says so. The small, slightly stubborn answer is: when something other than the model can show that it is. The verifier isn’t outside the agent; it’s part of the same harness, just walled off from the part that’s motivated to declare victory. And the payoff for insisting on that separation isn’t only trust. A lot that used to be invisible (why a run failed, when a model was bluffing, what a score was hiding) becomes something you can measure, read, and argue with.

Most of what I’ll write here starts from a journal this harness produced and follows it until it tells me something I didn’t expect. More of that soon.