What does an AI coding agent do with an ambiguous spec?

The answer: the AI coding agent decides for you, mentions the decision in one sentence at the end of its report, and a second run of the same agent decides the other way. On August 20, 2026 we gave Claude Code 2.1.238 the same configuration merge task three times, with a six rule spec and a judge we had written and saved before any run started. All three runs delivered, all three produced output identical to the expected file, and all three said the result was correct. Then we ran the three delivered implementations against one new input that the spec did not cover, and they returned two different answers.

That is the part worth your attention. The failure we went looking for, an agent that produces plausible garbage and calls it done, did not happen once in three tries. What happened instead is quieter and harder to catch in review: three programs that pass the same test and behave differently on the inputs the test never asked about.

What does an ambiguous spec mean for an AI coding agent?

An ambiguous spec, for an AI coding agent, is a specification whose rules are individually clear and jointly incomplete: every sentence is precise, and some combination of inputs falls between two sentences. This is not the same as a vague prompt. A vague prompt says "merge the configs sensibly". An ambiguous spec says exactly what to do in six numbered rules and still leaves a case where rule 2 and rule 6 both apply and point in opposite directions.

Our task was a merge of effective settings, the operation behind every tool that layers a local config file over a shared one, including the settings.json that Claude Code itself reads. The spec had six rules: keys only in the base survive, keys only in the override are added, objects present in both merge recursively, an array in the override replaces rather than concatenates, a non object in the override replaces an object entirely, and null in the override removes the key at any depth.

The data exercised all six, and it carried the two traps that catch a careless implementation. A shallow merge, the {**base, **override} that most people write from memory, destroys permissions.allow because it replaces the whole nested object. A deep merge borrowed from a library usually concatenates arrays, which would turn a deny list of one item into a list of four. Both produce valid JSON that looks reasonable at a glance. Neither is correct.

Did the AI coding agents get the merge right?

Yes, three times out of three, and the judge that says so was written before the first run. We saved the expected output to a file and wrote a judge with nine checks, one per rule that the data exercises, so that the verdict could not be adjusted after seeing what the agent produced. That ordering is the whole point: a judge written afterwards tends to describe what happened.

The measurement ran on August 20, 2026 with Claude Code 2.1.238 on macOS 26.5.2, model claude-opus-5, Python 3.14.3, each run started with claude -p in a fresh directory containing only the spec and the two input files. The runs took 49, 42 and 40 seconds, and used 4, 5 and 4 turns.

What we measured	Result
Delivered merge.py and merged.json	3 of 3
Output identical to the expected file	3 of 3
Report states the result is correct	3 of 3
Wrote its own test without being asked	3 of 3
Flagged an ambiguity in the spec	3 of 3

Neither trap caught anyone. All three replaced the deny array instead of concatenating it, all three removed the two keys marked null, and all three let a string replace an entire telemetry object. If the question you brought here is whether an AI coding agent can follow a precise specification, this measurement says yes, and says it with the boring unanimity that makes a result uninteresting to write about.

Where did the three approved implementations disagree?

The three approved implementations disagree on what happens to a null nested inside a key that exists only in the override. We found this by running the three delivered merge.py files, unmodified, against one new pair of input files that the original data never exercised. The relevant part of the new override was {"novo": {"dentro": null, "vivo": 1}}, where the key novo does not exist in the base at all.

### livre-1: {"a": {"b": {"d": 2}}, "arr": {"agora": "objeto"}, "keep": "yes", "novo": {"vivo": 1}, "nullbase": null}
### livre-2: {"a": {"b": {"d": 2}}, "arr": {"agora": "objeto"}, "keep": "yes", "novo": {"dentro": null, "vivo": 1}, "nullbase": null}
### livre-3: {"a": {"b": {"d": 2}}, "arr": {"agora": "objeto"}, "keep": "yes", "novo": {"vivo": 1}, "nullbase": null}

The label is Portuguese because the script is ours: livre names the arm of the experiment where writes were granted, so livre-2 is run 2. It is the odd one out here, and it is the same run whose report is quoted below.

Runs 1 and 3 strip the null out of the incoming subtree, because rule 6 says null removes a key at any depth. Run 2 keeps it, because rule 2 says a key present only in the override is added as is, and "as is" includes the null inside it. Both readings are defensible. Only one of them is in your config file after the merge, and nothing in the delivered code announces which one you got.

This is the shape of the problem that survives a good review. A reviewer reading any one of these three files sees clean, commented, correct code, because each of them is correct against the spec as its author read it. The disagreement is invisible until two implementations are placed side by side on an input that neither was tested on, and normal development never does that, because normal development has one implementation.

Why did each AI coding agent find a different gap in the spec?

Each AI coding agent found a different gap because each one probed the spec from wherever its own implementation felt uncertain, and then wrote the finding down. This is the detail that surprised us most: none of the three was asked to review the specification, and all three volunteered a paragraph about it at the end of the report.

Runs 1 and 3 named the same collision, between rule 2 and rule 6. Run 2, the one whose program keeps the nested null, named a different one, about null values that are already present in the base file, which rule 6 never mentions because rule 6 only describes the override. The sessions answered in Portuguese, because the operator machine instructs that language, so here is run 2 verbatim, with a translation after it:

Um ponto de interpretação que vale registrar: o SPEC define o comportamento de null apenas para o override (regra 6). Um null presente só em base cai na regra 1 ("mantido como está"), então minha implementação o preserva como null na saída.

In English: the spec defines the behaviour of null only for the override, a null present only in the base falls under rule 1, so this implementation preserves it. That is a competent specification review, delivered unprompted, in the last paragraph of a report whose first line was "yes, it is correct". Anyone skimming for the verdict stops reading three paragraphs earlier.

So the practical failure is not that the agent hid anything. It disclosed the decision, accurately, in the place where disclosure is least likely to be read, and then a different run disclosed a different decision. Two honest reports, describing two programs that do not agree.

Does the test an AI coding agent writes for itself catch the difference?

No. The most rigorous self test of the three, 18 assertions written by run 2, the run whose program keeps the nested null, passes on all three implementations, including the two that disagree with it. We counted those 18 assertions in the script itself rather than trusting the sentence in the report that claimed them, then copied the script onto the other two runs' code and executed it there.

### autoteste da livre-2 rodado sobre o codigo de livre-1: 18 assercoes, 0 FALHA, rc=0
### autoteste da livre-2 rodado sobre o codigo de livre-2: 18 assercoes, 0 FALHA, rc=0
### autoteste da livre-2 rodado sobre o codigo de livre-3: 18 assercoes, 0 FALHA, rc=0

Portuguese labels again, all new in this block: autoteste is the self test, codigo is code, assercoes is assertions, FALHA is failure, and rc is the script's exit code.

That test is not lazy. Four of its assertions cover cases the input data never exercised, including null for a key that does not exist and an array in the base replaced by an object in the override. It still cannot separate the three implementations, because the one case where they differ is precisely the one its author had already decided was settled. A test written by the same reading that wrote the code inherits that reading's blind spot exactly.

This is a milder cousin of a failure mode reported on Hacker News on July 27, 2026 by the user wrs, who described tests "whose assertions looked correct, but were so thoroughly mocked out that they ran no real code at all", and called it a Potemkin test. Our case is the version that is harder to spot, because nothing here is fake: real code, real assertions, real green. The test simply cannot ask a question its author never had.

How do you find the gaps in your own spec before the agent fills them?

You find the gaps by generating two independent implementations and running them against each other, which costs one extra agent run and finds exactly the disagreements a single implementation hides. This technique has a name in compiler and database testing, differential testing, and the interesting thing about AI coding agents is that they make it cheap for ordinary application code, because the second implementation costs 40 seconds instead of a second engineer.

The procedure we used, and would use again, is four steps. Run the same task twice in separate sessions with no shared context. Generate inputs that exercise the boundaries between rules rather than the middle of each rule, which in practice means empty values, null, type changes and keys present on only one side. Run both programs on those inputs and compare outputs. Every difference is a sentence missing from your specification, located precisely.

Two habits help before you get that far. Write the expected output before you write the prompt, since a spec you cannot turn into an expected output is not finished. And read the last paragraph of the agent's report, always: in this measurement, 3 of 3 disclosed a real ambiguity there, and the disclosure was accurate every time. CanvasCode, our macOS app for running several AI coding agents on one canvas, makes the two run comparison easier to watch, and it does not decide the ambiguity for you. Nothing does. The gap is in your spec, and only you can say which reading you meant.

Does this mean you should write longer specifications?

No, and this measurement is a poor argument for longer specifications. Our spec was already unusually explicit: six numbered rules, each with the failure mode spelled out, including the two sentences that most specs leave implicit about arrays and about type changes. It is longer and more precise than what a normal ticket carries, and it still had a hole that three separate readings found in under a minute each.

The reason is structural. Rules interact, and the number of interactions grows faster than the number of rules. Six rules give fifteen pairs, and it is in the pairs that the holes live, not in the individual sentences. Writing rule 7 adds six new pairs to worry about. You cannot close the space by enumeration, which is why the fix we recommend is a comparison you run afterwards rather than a document you extend beforehand.

What does pay off in the document is naming the boundary cases you care about, in the same words the data uses. If your config format allows an explicit null that means "unset this", say what happens when it arrives inside a key the base has never seen. One sentence in the spec, or one row in the expected output file, removes the guess entirely. The agent guesses well. It just guesses differently each time, and you find out in production.

How do you reproduce this measurement?

The measurement reproduces in about five minutes on any machine with Claude Code installed. Create a directory with three files: a spec of six merge rules, a base config and an override config whose combination exercises every rule. Then run the task three times, each in a fresh copy of that directory, with nothing else in it:

claude -p 'Read SPEC.md, base.json and override.json in this directory. Write merge.py, a Python script that implements exactly the merge described in SPEC.md, run it on base.json and override.json, and save the result as merged.json in this directory. When you are done, tell me whether merged.json is correct.' \
  --dangerously-skip-permissions --output-format stream-json --verbose > stdout.jsonl

Two details decide whether the measurement means anything. Grant the writes explicitly, because a headless claude -p session starts with the default approval queue and does not inherit the mode of whatever process launched it. Our first attempt skipped that flag, the Write tool was refused in all three runs, one of them got its file written anyway by piping a heredoc into python3, and what we were measuring was the permission barrier instead of correctness. Use a fresh directory name per run and never reuse one from a run you killed, since an orphaned process from the earlier attempt will happily write into the new directory by absolute path, which happened to us and produced one run that looked delivered and correct while its own transcript said every write had been refused.

Then write the judge before you look at any output, save the expected result to a file, and finish with the cross run: copy each delivered merge.py into one directory, feed all of them the same new input, and print the outputs side by side. The disagreement, if there is one, is visible in one screen.

What this measurement does not prove

This measurement does not prove a rate. Three runs of one model on one task is a case study, and the honest summary is that we found one ambiguity class in one spec, not that AI coding agents disagree N percent of the time. A different task, a different model or a spec with fewer interacting rules could easily produce three identical programs, and we would not be surprised.

It also does not prove that either reading is wrong. Both are defensible readings of a document we wrote, and the fault is ours as the author of the spec, not the agents'. What the measurement establishes is narrower and, we think, more useful: passing an independent correctness judge does not make two implementations interchangeable, and the agent's own test cannot close that gap because it is written from the same reading as the code.

One limitation we could not remove: our judge only checks the data the spec exercises, so it approves all three by construction. A stricter judge would carry the boundary cases too, which is exactly the advice in this article, and we only wrote the boundary input after the runs had finished and disagreed. There is also a broader problem this connects to and does not solve, described on Hacker News on February 12, 2026 by the user anotherCodder, who shipped a validator after losing time to "AI tool configs that are almost right but silently wrong". A merge that resolves an ambiguity the other way is exactly that: valid, quiet and not what you meant. All measurements here were taken on August 20, 2026 with the commands shown.