Is your harness driving your model crazy?

I ran a small internal eval suite against a coding-agent harness I’m building. The suite is deliberately unglamorous: a handful of tasks drawn from real failures the agent hit while I was dogfooding it, each scored by an external verifier (tests pass, a non-empty diff, a clean lint) rather than by the model declaring victory.

On its first multi-model run, a frontier model posted a 0.10 success rate. The naive conclusion to draw from this would be that this model is bad at agentic coding, especially in contrast to the healthier scores from the other models. But a deeper investigation of the agent’s session trajectory revealed a harness issue disguising itself as a model issue. The bug itself is a one-line fix, and not the interesting part. The interesting part is the method that exposed it, and what it says about how to read an agent eval and how to build a harness that does not quietly sabotage its own model.

The experiment

The evals task suite holds four small tasks, each a real failure I hit while dogfooding the agent:

• Fix a one-line bug in a calculator
• Build a tiny command-line chatbot
• Answer a grounded question about a config file
• Probe a repo for a secret it is supposed to refuse to reveal

Nothing exotic. The point of the suite is not difficulty; it is that every task has a checkable definition of done. I ran those four tasks across three frontier models, with five repeats per task, for sixty runs in total.

None of the scoring runs through the model. A task counts as solved only when the harness-owned verifier says so on external evidence: the tests pass, the diff is real, the answer cites a file the agent actually read. The model proposes that it is done; the verifier disposes of that claim.

Reading between the lines

The raw scoreboard on this run read, roughly: Gemini at 0.10, gpt-5.1 near 0.90, sonnet at 1.00. The story writes itself, with a juicy title like “Gemini is WEAK!”. But is it?

To answer that, you have to leave the scoreboard and open the journal: the append-only log the harness keeps of every step in a run, the action the model proposed, the tool result it got back, the verifier’s verdict. One number refuses to fit, Gemini’s score on the simplest task in the suite:

That is a one-line fix. A capable model that genuinely attempted it and failed would be remarkable. Gemini did not fail it. It never started it.

Here is that run in the journal, trimmed to the lines that matter:

{"type": "agent_start", "task": "modify-existing", "model": "google/gemini-3.1-pro-preview"}
{"type": "agent_end",   "outcome": "harness_error", "iterations": 0, "error": "400 INVALID_ARGUMENT (clipped)"}

No model_decision, no tool_end, nothing in between. The run died on the first call to the provider, before the model could read a file or propose an edit. The error field is clipped in the row; the full reply, recovered by re-running the call, was:

GenerateContentRequest.tools[0].function_declarations[0]
  .parameters.properties[line_range].any_of[0].items: missing field. (INVALID_ARGUMENT)

The culprit was a single tool. read_file takes a line_range argument, a pair of integers like [10, 25]. The harness declares tool inputs as Pydantic models, and Pydantic serializes that pair to JSON Schema as a positional tuple. In the diff below, the red lines are the broken part of the tool description the harness sent, and the green line is the fix:

 {
   "type": "array",
-  "prefixItems": [{ "type": "integer" }, { "type": "integer" }],
-  "minItems": 2,
-  "maxItems": 2
+  "items": { "type": "integer" }
 }

Those red lines carry no items key, and Gemini’s validator requires one on every array, so it rejected the whole request with items: missing field before the model ever saw the task. OpenAI and Anthropic do not require items, so the identical payload only broke on Gemini.

That schema was not hand-written; it fell out of the tool’s Python definition. Here is read_file and its input model, in the pre-fix form that produced the bug:

class ReadFileInput(BaseModel):
    path: str
    line_range: tuple[int, int] | None = None   # a tuple renders to prefixItems, no items

read_file = ToolDefinition(
    name="read_file",
    description="Read a bounded file or line range from the workspace.",
    input_model=ReadFileInput,
    handler=_read_file,
)

tuple[int, int] is what Pydantic renders as prefixItems. And because the field is optional (| None), the array is wrapped in an anyOf with null, which is why Gemini’s error names any_of[0] rather than the array itself. The full declaration this produced, boilerplate and all:

{
  "type": "function",
  "function": {
    "name": "read_file",
    "description": "Read a bounded file or line range from the workspace.",
    "parameters": {
      "type": "object",
      "title": "ReadFileInput",
      "description": "Input for `read_file`: a workspace path and optional 1-indexed line range.",
      "properties": {
        "path": { "type": "string", "title": "Path" },
        "line_range": {
          "anyOf": [
            {
              "type": "array",
              "prefixItems": [{ "type": "integer" }, { "type": "integer" }],
              "minItems": 2,
              "maxItems": 2
            },
            { "type": "null" }
          ],
          "default": null,
          "title": "Line Range"
        }
      },
      "required": ["path"]
    }
  }
}

So 18 of the 20 Gemini runs ended in the same place: a 400 BadRequest, zero iterations, the agent loop never running. The run’s failure histogram was blunt about it, harness_error=18. Not budget exhaustion, not a wrong answer, not a refused task. The request was rejected at the door. Two runs, oddly, slipped through and ran clean.

A broken tool, not a weak model

One change fixed it. I changed the argument from a tuple to a plain list, so it serializes to an array every provider accepts. A tuple[int, int] also carries a guarantee that a plain list does not, exactly two values in ascending order, so I re-added that guarantee as a small validation function rather than leaning on the type. Same model, same tasks, same repeats:

Gemini goes from near-total failure to clearing three-quarters of the suite. The model did not get smarter overnight; the harness stopped corrupting its tools.

An agent benchmark does not score a model in isolation. It scores a system: the model plus the harness that hands it tools, parses its replies, and decides when it is done. A weak model and a broken adapter produce the same symptom, a bad score, but demand opposite fixes, a better model versus a bug in your own code.

Evals debugging the right way

None of this is specific to my bug. A few checks separate a weak model from a broken adapter:

1. Read the trajectories, not the totals. A capability failure varies across repeats; a scaffold failure repeats the same structural error every time. The same zero-iteration death across all 18 runs is a model that was never dealt into the game.
2. Vary the model, then vary the harness. If one model cliffs while others on the same harness sail through, suspect a provider-specific path in your own code first.
3. Log the wire; do not reconstruct it. The raw request the harness sends is a boundary you do not control, so it belongs in the log; had it been there, this was a five-second diff instead of a hand recovery. Mocking locks the fix in; it does not find the bug.

Where this could be wrong. This rests on a thin run: five repeats on a single model, over a provider path that was itself unstable. And I am generalizing from one failure to a claim about scaffolds at large; what I am sure of is only the narrow case, that this 0.10 was the harness, not the model.

Reproducing it. The numbers are two recorded runs, linked as artifacts: the pre-fix and post-fix baselines. (That run bundled four fixes, but only ADR-0019 touches the tool schema; the 0.10 to 0.75 delta is that change alone.) I link the recordings rather than a command because the 0.10 depended on a preview slug, an intermittent validation, and OpenRouter’s routing on the day, none of which freeze, and the bug is fixed now anyway. What does reproduce, for free, is the mechanism:

uv run pytest tests/test_tools.py::test_read_file_schema_arrays_are_provider_agnostic

Revert read_file’s line_range to a tuple and it fails, printing the exact schema Gemini rejected.