How do you verify what an AI coding agent says it did?

The answer: check the claim against a source the agent does not control, and before that, ask whether the action it claims leaves a durable record or only a last state. On August 15, 2026 we audited the 15 most recent claims our own agent made about acting outside its own server. Ten hold up when checked at the source. Five cannot be verified by us or by anyone, because the source keeps only the most recent event and overwrites the rest. None of the 15 was shown to be false, and five of them never could be, which is not the same result.

That distinction is the one that changed how we work. We went in expecting to catch a lie and came out with something more useful: a whole class of agent claim that is unauditable by construction, and which reads exactly like a claim you could check.

What does verifying an AI coding agent's report actually mean?

Verifying an AI coding agent's report means confirming, in a system the agent has no write access to, that the effect it described actually landed. It is a different activity from reading its code. When an AI coding agent such as Claude Code, Codex or Cursor finishes a task, its output usually contains two kinds of statement, and only one of them is a diff. "I refactored the parser" points at code you can read. "I ran the test suite", "I pushed the branch", "I restarted the dev server", "I opened the pull request" and "I sent the notification" point at events in the world, and the diff says nothing about them.

The second kind is where trust quietly accumulates. Reviewing a diff is a habit the industry already has, with tooling and culture around it. Auditing an assertion about an executed action has neither, so the agent's sentence is usually the only artifact anyone ever looks at. A commenter on Hacker News, writing on August 15, 2026, put the failure mode plainly:

You have to be able to write the software yourself in order to judge the results and get good software out. Otherwise the system claims the goals are met, the tests pass, and the human driving the system puts up a new PR.

So the practical question is not whether the agent is truthful. It is which of its sentences are attached to a source you can query, and what that source is capable of remembering.

Why the agent's own log is not evidence

An agent's own log is a record of what it believed it did, produced by the same process that did it, which makes it a description rather than a confirmation. If the action silently failed, the log will still say it succeeded, because the log is written from intent and from a response the agent may have misread.

We learned this in the least comfortable direction. Our content pipeline keeps a journal, and on August 15, 2026 one run read a previous run's entry claiming a cross post to dev.to, checked whether that post existed, concluded it did not, and recorded that the earlier entry was false. The accusation was wrong. The post had been live since 11:04:24 UTC that morning. What had failed was the checking method, not the journal, and the cost was real: acting on the wrong conclusion, that run published a second cross post and blew the daily quota.

The lesson generalises past our case. When an agent's report and your check disagree, there are three candidates for the error and most people only consider one. The agent may be wrong, the source may be lying by omission, or your query may be asking the wrong question. We assumed the first and it was the third. Before treating an agent's claim as false, reproduce the check with a different query against the same source, because a verification method has no more authority than the agent it is auditing.

Which agent claims can be verified at all?

An agent's claim can be verified when the effect it describes leaves a durable record, and cannot be verified when the effect only updates a last state. This distinction decides everything downstream, and it is worth sorting the claims before writing any tooling.

Durable record means the action creates an object that persists and carries its own identifier: a published page that keeps returning 200, a post with a permanent slug, a commit reachable by SHA, a CI run with a run id, a released artifact with a checksum. You can ask for it by name years later. Last state means the system stores only the most recent occurrence: a sitemap submission, a cache invalidation, a service restart, a webhook fired at an endpoint that keeps no inbox. The action leaves a mark, but the mark is overwritten by the next one, so it can answer "when did this last happen" and can never answer "did it happen on Tuesday".

Applied to an AI coding agent, this sorts a normal session in seconds. "I pushed" is durable, because the remote has the SHA. "I opened the pull request" is durable, because the pull request has a number. "I ran the tests" is durable only if the run happened somewhere that keeps run records, and is otherwise pure last state, which is why it is the most commonly repeated unverifiable sentence in the category. "I restarted the service" is last state almost everywhere. A claim in the second column is not a lie. It is a sentence that no amount of diligence can promote to evidence after the fact, and the only fix is to make the action leave a record before it runs.

What did the audit of our own agent return?

Our audit returned 10 verified out of 15, and the five that failed all failed for the same structural reason. The subject was our content pipeline's journal: of its 100 most recent entries, 15 assert an effect outside the server that stores them, logged between August 14, 2026 at 00:14 UTC and August 15, 2026 at 17:11 UTC. We wrote down what we expected to find before querying anything, which is the only way an audit can disagree with the person running it.

The six publication claims verified 6 of 6. Each names a slug, and each slug exists in three languages, so the check was 18 requests against the live site, all returning 200:

curl -s -o /dev/null -w '%{http_code}\n' "https://canvascode.app/en/news/undo-ai-coding-agent-changes"

The three cross post claims verified 3 of 3, each present with the correct canonical URL pointing back at our site. We used the account endpoint /api/articles/me/all, which returned 401 without an API key and 200 with one, rather than the public listing, for reasons the next section explains.

The six sitemap ping claims verified 1 of 6. The Search Console response for our sitemap carried a single submission on record, lastSubmitted at 2026-08-15T17:08:02.065Z, with no earlier ones beside it. That confirms the most recent ping and leaves the other five indistinguishable from pings that never happened. The same response carried lastDownloaded at 2026-08-15T17:08:04.364Z, and that field is the more interesting half: two seconds after our submission, Google fetched the file. That is the difference between evidence that you spoke and evidence that the other side acted, and it is the strongest confirmation on this list.

Why does an empty API response not prove the agent did nothing?

An empty API response does not prove absence because a listing endpoint answers the question you typed, not the question you meant, and it will answer a malformed question with a confident, well formed nothing. We reproduced three separate ways to be misled by the same dev.to API on August 15, 2026, and each returned HTTP 200.

The first is identity. Querying the public listing with a username that does not exist returns [] and status 200. Our brand name is not our handle there, and asking for the brand produced a clean empty list, which looks precisely like an agent that published nothing.

The second is caching by URL. With the correct handle, the same endpoint returned different totals depending on a parameter that should only widen the window:

curl -s "https://dev.to/api/articles?username=hassekf&per_page=30"

That returned 8 articles, while per_page=10, per_page=20 and per_page=50 each returned 10. We ran all four values three times and got the identical pattern every time, so this is not flakiness, it is a cache keyed on the full URL. A listing that shrinks when you ask for more cannot be used as a census.

The third is the fix for both. Asking for the item by its identifier, /api/articles/hassekf/<slug>, returned 200 for the post whose existence we had wrongly denied earlier that day. That endpoint needs no key; it simply answers about one named thing instead of enumerating. When you want to know whether one specific thing exists, ask for that thing by name, and enumerate only when you need a count.

A counter that reads zero while the same source reports 39 impressions

The same Search Console response that verified our sitemap ping also carried a number we nearly published as a crisis. Alongside the submission timestamps, the contents block reported 579 URLs submitted and 0 indexed. Read alone, that says Google has indexed nothing we have ever written.

It is contradicted by the same product. Pulling the search analytics for the trailing 28 days from that identical property returned 23 distinct queries and 39 impressions, with 0 clicks. A page cannot be shown 39 times in search results while being absent from the index, so the two numbers cannot both describe reality, and the one to distrust is the counter rather than the report. Our visitor log agrees with the report: it records arrivals referred from google.com in the same window.

We are stating what we measured and not why the counter behaves this way, because we have no source from Google explaining it, and inventing one would be the exact failure this article is about. The operational rule we take from it is narrower and does not need the explanation: a single field is not a measurement until a second field from the same source agrees with it. A counter that contradicts its own product's report is a reason to look for the second field, not a reason to raise an alarm. We would have raised one.

The check we ran on ourselves, and lost

While auditing our agent we caught one of our own unverified beliefs, which is worth reporting because it is the same failure in miniature. Our internal notes say that fetching canvascode.app pages with a bare script gets rejected by the probe defence that guards the site, so every verification in this audit was written with a full browser user agent, out of habit rather than measurement.

During this audit we finally tested the habit, requesting the same page three ways: with no user agent header at all, with curl/8.7.1, and with a browser string. All three returned 200. The precaution was unnecessary for this kind of request, and the note that justified it did not survive contact with a command, which is why we now check canvascode.app with plain curl and no user agent flag, not the browser string we used out of habit before.

The belief was not invented. It came from a real earlier observation, and the reading we take is that we generalised a specific case into a rule and then stopped testing it. That is exactly what an agent does when it writes "done" into its log: it converts one observation into a standing fact. The difference between us and the agent, on that particular day, was three seconds of curl.

How do you make an agent's report checkable before it runs?

You make an agent's report checkable by requiring it to record the identifier the source gave back, rather than the fact that it acted. This is a change to the instructions you give the agent, and it costs nothing at the moment of writing, which is the only moment when it is still possible.

Three requirements cover most of it. First, the report must carry an identifier that can be looked up independently: a commit SHA, a pull request number, a CI run id, a published URL, a message id. "Pushed the fix" is unverifiable; "pushed 4f2a91c to origin/main" can be checked by anyone with the repository. Second, it must carry the source's own timestamp, not the agent's, because the agent's clock only proves when it believed it acted. Third, it must include the exact command or call it made, so a reviewer can replay the check instead of inventing one, which is how our own bad verification happened in the first place.

For last state actions, the only honest option is to create the record yourself. If the agent restarts a service, have it capture the new process start time immediately afterward and put that in the report. If it runs tests locally, have it write the summary to a file with a timestamp, or run them somewhere that keeps run records. CanvasCode, our macOS app for running several AI coding agents on one canvas, helps with the part of this that is about seeing many agents at once, and it does not solve this problem: an agent whose report has no identifier in it is unverifiable no matter how good your view of it is. The fix lives in what you require the agent to write down, not in the tool you watch it through.

What this audit does not prove

This audit does not prove that AI coding agents report their actions accurately, and the sample is small enough that we would not defend a rate from it. Fifteen claims from one agent over 41 hours is a case study. It also does not prove anything about the five unverified claims: they were not shown to be false, they were shown to be unfalsifiable, which is a different and in some ways worse result.

The largest limitation is what the audited agent is. Our pipeline writes and publishes content; it is not an AI coding agent, and it does not touch a repository. What transfers is the classification of claims, not the numbers: publishing a page and pushing a commit are both durable record actions, and pinging a sitemap and restarting a service are both last state actions, but our 10 of 15 says nothing about what a coding agent's ratio would be in your repository. We would like to see someone run the same audit against a coding agent's session log, because the method is cheap and the result is checkable.

One more piece about the scoring. We counted a claim as verified when an independent source confirmed the effect, but for the six published pages the independent source is our own website, which our own pipeline wrote to. A stricter audit would confirm them somewhere we do not own, such as a third party archive. All measurements in this article were taken on August 15, 2026 with the commands shown.