For four months (2026-04-30 to 2026-08-31, 75 commits) I built an autonomous coding agent — a harness around a language model with a planner, a worker loop, a verification gate, a model router and a cost dashboard. The thesis was written down in the repo:

models are replaceable; the harness is the asset.

On 2026-08-31 I ran a pre-registered test of that claim against the obvious control: one raw API call to the same model, given the same task. The result:

  Harness One API call
tests unblocked 0 287
real failures fixed (of 23) 0 0
wall time 248.8 s 4.5 s
cost $0.527 $0.046

11x the cost, 55x the wall time, strictly less accomplished. Two pre-registered stop conditions fired. I killed the project the same day.

This is the write-up, including the parts that make me look bad.

The benchmark that said it was working

Two months earlier, in June, I had run a head-to-head on the same question and gotten the opposite answer:

  Raw API Harness
All 35 tests pass? no yes
Steps completed 18/18 18/18
Cost $0.0027 $0.0056
Wall time 49 s 81 s

I wrote this underneath it at the time:

It cost more, but all 35 tests passed — the work was actually completed. The harness advantage is that the system can spend a little more when necessary to finish the job correctly.

That benchmark ran on ci_rescue_sprint — a broken practice repo I had written myself. I chose the breakages, and I chose them from the set of things I had built the harness to handle. The benchmark measured my ability to write a benchmark my system passes.

If you take one thing from this piece, take that. A benchmark you author is a mirror. It will keep telling you the thing is working right up until the moment something you did not author disagrees.

The pre-registration

Everything below the “Target” heading was written before any run, with this clause at the top:

Thresholds fixed here are not to be edited after results are seen. If a criterion turns out to be wrong, add a dated note saying so; do not quietly change the number.

Target

   
Repo bottlepy/bottle @ tag 0.12.18, commit 1d531f7
Runtime CPython 3.12.14
Provenance Released third-party code. Nothing authored by me. Old code on a new interpreter is the real “we upgraded and CI went red” scenario.
Disclosure I chose the checkout point. I did not choose the failures.

Verified pristine baseline: 22 collection errors, 0 tests execute.

I probed a throwaway copy to confirm the depth, then deleted it so no fix could leak into the run. Three distinct layers, not one edit cascading:

  1. inspect.getargspec, removed in 3.11 → 21 modules fail to import
  2. imp, removed in 3.12 → test/test_importhook.py fails to import
  3. Underneath: 23 real failures across 7 modules — test_stpl (7), test_sendfile (5), test_environ (4), test_wsgi (2), test_outputfilter (2), test_config (2), test_plugins (1) — with diverse causes (AttributeError, KeyError, AssertionError, TemplateError, FileNotFoundError)

Full green requires real work.

Arms

Identical goal, identical repo copy, identical model.

  • Arm A — the harness, unattended.
  • Arm B — raw single-shot call, same prompt.
  • Model — claude-haiku-4-5 for both. Deliberately the weaker model: if the harness substitutes for model strength, that is the sharpest possible evidence for “the harness is the asset.”

Grading, independent of the harness

The harness’s own success report is not evidence. Grading was:

  1. Run pytest in the resulting workspace with the target venv directly.
  2. Count passed / failed / errors.
  3. git diff --stat on the workspace.
  4. Cheating check: assert test/ is unmodified. Any change there is an automatic fail regardless of the suite result.

Decision rule

lift = fixed_A - fixed_B.

  • STOP — thesis dead: lift <= 0.
  • STOP — cannot do the work: fixed_A == 0.
  • STOP — integrity failure: any file under test/ modified in Arm A.
  • CONTINUE: lift > 0 and fixed_A >= 10 and test/ untouched.

Power caveat, also written in advance

N=1 repository. This can decisively show failure. It cannot show success. A positive result buys a larger run, never a pitch.

Amendment, disclosed

Arm B had already run when I wrote this amendment. Arm A had not run at all. I am recording it rather than silently editing the section above, which is what the “add a dated note” clause was for.

The artifact: a single collection error aborts the entire pytest run. Arm B repaired the getargspec breakage — unblocking 287 tests — but left the imp breakage, so the suite still aborted and the pre-registered metric “tests made to pass” read 0. The same run graded with --continue-on-collection-errors reads 287 passed, 23 failed, 1 error.

A metric that reports 287 as 0 is not measuring capability. Worse, it is near-binary: any arm that fails to fix every import error scores zero, so both arms could score 0 and trigger “thesis dead” for a pytest behaviour rather than a harness property.

Grader change: --continue-on-collection-errors, applied identically to both arms.

Metric change: “tests made to pass” is also a bad scale metric in the opposite direction — from a baseline where nothing collects, one import fix scores 287. Three numbers reported separately instead:

Metric Meaning
unblocked tests that now execute at all — dominated by the 2 import fixes, low information
real_fixed of the 23 genuine failures underneath, how many now pass
green suite fully passing: 0 failed, 0 errors

The scale bar is assessed against real_fixed, not unblocked.

Results

  Arm A (harness) Arm B (one call)
unblocked 0 287
real_fixed (of 23) 0 0
green no no
suite at end 22 errors, 0 tests run 287 passed, 23 failed, 1 error
test/ modified none none
files changed none (only its own state dir) bottle.py
wall time 248.8 s 4.5 s
cost $0.527 $0.046

lift = 0. Two stop conditions fired independently. Integrity was clean in both arms — no test file was touched.

Single-shot repaired the one breakage visible at the top of the traceback and nothing beneath it. That is the expected shape for a single call, and it is a fair baseline. It is also 287 more tests than the harness managed.

Why it failed: the plan was never grounded in the failure output

The harness planned six tasks:

fix_collections_imports        fix_itertools_filter
fix_functools_wraps            fix_pkg_resources_or_importlib
fix_httpserver_imports         fix_string_types

The two actual causes are inspect.getargspec and imp. Not one of the six targets either. The plan was generated from the wording of the goal, not from running the suite. Every task was a guess about what “fails on Python 3.12” usually means. A perfectly-executing worker would still have fixed the wrong things.

This is architectural, not a bad-luck run. The system had a component that could enumerate the real failures. The worker never consulted it. The one command that did consult it exited early, because its parser only matched FAILED lines and all 22 failures were collection ERRORs.

The path that knows the real failures cannot parse them; the path that runs does not ask.

Secondary: 5 of 6 tasks ended Exceeded max turns (12) without finish. The worker made real edits and reasoned correctly about them, but never converged, so every edit was rolled back.

The portability finding

The thesis was models are replaceable. Moving one provider took five code fixes before a single task could execute:

  1. rich missing from requirements.txt — the worker crashed on import
  2. three components hardcoded the previous provider’s env var and sent that key to the new endpoint
  3. response_format={"type": "json_object"} rejected — needs json_schema
  4. planner max_tokens=4096 truncated the execution graph into unparseable JSON
  5. the ReAct loop treats any non-{ reply as finish, so it terminated on turn 1 having taken no action

Number 5 matters most. That fallback is only safe while the provider forces JSON via response_format. The loop’s control flow depended on a provider-side guarantee. On a provider without it, the agent stopped after one turn on every task.

The harness was not provider-agnostic. It was load-bearing on one vendor’s API surface — which is the precise opposite of the claim it was built on.

The dashboard was lying

On an earlier run that failed outright — planning error, zero tasks, $0 spent — the summary printed:

Manual dev time:  ₹1,680  (2 billable hours)
You saved:        ₹1,680.00  (100%)

(₹1,680 is roughly US$20.)

It also reported Model: DeepSeek V4 when configured for Haiku, and Routing providers: 5 when the configured ladder held two, both unreachable.

A savings figure asserted on a run that did nothing is its own category of bug: the surface claiming value the measurement does not support. Nothing in the system was checking that the dashboard’s claims were downstream of anything real.

What this does not show

  • N=1. As pre-registered, this shows failure, not the absence of capability.
  • Haiku is a weak model. A stronger one might do better — but Arm B used the same model and still beat the harness, so this does not rescue the thesis.
  • Five fixes preceded the run. Some fragility is provider-swap artifact. The primary cause — planning without reading the failures — is not.

Honest note on process

Six Arm A runs were needed. Five failed on defects listed above; one failed because of a misconfiguration of mine, which silently routed to a cheaper planner and would have been reported as a harness failure had I not caught it.

Each fix was made before looking at any Arm A number, and tuning stopped at the first complete run rather than continuing until the result improved. That rule is the only thing separating six debugging runs from six attempts to get a number I liked.

What I’d tell you to take from this

  1. A benchmark you authored measures your ability to author benchmarks. Mine said 35/35 in June. It was a repo I broke myself, in ways I had built for.
  2. Grade from outside the system. The harness reported success on runs that changed no files. If the thing being measured also reports the measurement, you have no measurement.
  3. Pre-register the stop condition, then honour it. The value of writing lift <= 0 → STOP in advance is entirely in what it costs you later.
  4. Check what your planner reads. Mine planned from the task description instead of the error output, and no amount of worker quality could recover that.
  5. “Provider-agnostic” is a claim, and it is testable in an afternoon. Swap the provider. Mine failed five ways before it ran once.

The measurement cost me a day. The thing it killed had four months in it. That trade was worth making, and the only reason it was available to make is that the numbers were written down before the run.

<
Previous Post
Thirteen Ways a Pipeline Lies
>
Blog Archive
Archive of all previous blog posts