Back to news

Which codex exec output signals can you trust?

The exit code is the one signal you cannot use. In 9 runs of codex exec on 19 August 2026 (codex-cli 0.147.0, macOS 26.5.2 on arm64) the process exited 0 every single time, including the 6 runs that left the repository byte for byte identical. Four other signals in the same output do carry the truth, and they answer different questions: a file_change item on standard output says a write was attempted, its status field says whether the write landed (it matched reality in all 10 items across 9 runs on 21 August 2026, codex-cli 0.148.0), its kind field read as a balance per path is the only thing in the output stream that catches a file created and then deleted, and an ERROR line reading patch rejected on standard error appeared in 6 of 6 blocked runs. This page is an index. It measures nothing new: it consolidates 25 runs of codex exec we published on 19 and 21 August 2026, plus one Claude Code comparison from 18 August 2026, and every number in it carries the date and the build it came from.

The order matters more than any single signal, because these signals fail in different directions. One of them is silent when work is missing. Another fires when nothing survived. A third is the only honest one on a failed write, and we have only observed it on the newer build.

What does the codex exec exit code actually prove?

The codex exec exit code proves that the process finished, and nothing else about the work. We gave codex exec the same small task nine times on 19 August 2026, in nine throwaway git repositories, on codex-cli 0.147.0 and macOS 26.5.2: add a slugify function to src/utils.js, export it, run npm test. The three arms differed only in sandbox policy: -s workspace-write, no -s flag at all, and -s read-only. The process exited 0 in 9 of 9 runs. In 6 of those 9 the repository came out byte for byte identical to how it started, because the sandbox refused the patch. A pipeline that gates on $? would have marked all nine green.

There is one case where the exit code does speak, and we hit it by accident. In a separate battery on the same day, a run died before doing anything with exit code 1 and a turn.failed event carrying the message Selected model is at capacity. So a non zero exit tells you the run broke; a zero exit tells you nothing about whether the file was written. The judge in all of these runs was not the agent: the apparatus hashed every non .git file before and after each run and compared the fingerprints, so the question "did anything change" was answered outside the agent's reach. The full nine run table is in how do you know whether Codex actually changed a file.

Which codex exec signal tells you a write was attempted?

The signal that a write was attempted is an item.completed event of type file_change on standard output, which codex exec --json emits per changed path. In the nine runs of 19 August 2026 it appeared in 3 of 3 runs that edited the file and in 0 of 6 runs that did not, which is a clean separation and the reason we recommended it in the first place. Each item carries a list of changes, and each change carries a path and a kind field whose values we have observed as add and delete.

Read that sentence literally, because the trap is in the verb. The event answers "did a write happen at this path", not "did the work happen". Those two questions come apart in ordinary situations, not exotic ones, and the next section is where we published the counterexample against our own recommendation. If you are wiring codex exec into CI today and you only have time to change one thing, changing it from $? to the presence of a file_change item is already the larger half of the improvement, on the runs we measured.

Does a file_change event mean the work actually happened?

No. A file_change event in codex exec fires on the write and not on the outcome, and we measured two ordinary ways for it to mislead you, both on 19 August 2026 with codex-cli 0.147.0. In the first arm, three runs, the task asked Codex to create sum.js, run a rigged test that demands sum(2, 2) equal 5, and revert everything if the test failed. Codex did exactly that. The repository fingerprint after the run equalled the fingerprint before it in 3 of 3 runs, and file_change fired anyway in 3 of 3, once with kind: "add" and once with kind: "delete". In the second arm, three runs, one prompt carried two independent tasks, one possible and one impossible with the network switched off. Codex completed one half and skipped the other, and file_change fired in 3 of 3 exactly as it does for complete work.

The repair is small and it lives in the payload you already have: read kind and compute a balance per path instead of counting events. A path with one add and one delete nets to zero and the revert becomes visible. Nothing in the output catches the missing half of a two part task, and we could not find a signal that does. That is a real limit of the format, not an oversight in our reading of it, and the measurement is in does codex exec show you when the agent only did half the work.

Which field in codex exec catches a write that failed?

The field that catches a failed write is status on the file_change item, and on our measurement it is the most honest machine readable thing in the run. On 21 August 2026, on codex-cli 0.148.0 and macOS 26.5.2, we ran codex exec nine times with a macOS sandbox-exec profile denying writes to the working directory in three of them. Those nine runs emitted ten file_change items in total, and the field matched reality in every one: status: "failed" in all 4 items emitted by the three blocked runs, status: "completed" in all 6 items emitted by the six runs that wrote. Here is one item, verbatim and complete, from a blocked run:

{
 "id": "item_1",
 "type": "file_change",
 "changes": [
  {
   "path": "/private/tmp/codex-sandbox-21ago-n1/N1/run1/report.txt",
   "kind": "add"
  }
 ],
 "status": "failed"
}

This refines the previous section rather than contradicting it. Both statements hold at once: the event is honest about whether the write landed, and silent about whether the work was worth anything. One caution belongs here and no filial article carries it, because each of them was written inside one build. We observed status on codex-cli 0.148.0. The 19 August measurements ran on 0.147.0 and we did not read that field there, so we cannot tell you from our own data whether it was present and we ignored it, or absent and later added. Check it on your own build before you gate a pipeline on it.

What does the ERROR line on standard error catch?

The ERROR line on standard error catches the blocked write, and in our 19 August 2026 battery it was the mirror image of file_change. A line reading patch rejected appeared in 6 of 6 runs where the sandbox refused the write, and in 0 of 3 runs that wrote successfully. Standard error and standard output therefore answer opposite questions on the same run, and a pipeline that captures only one of the two streams is throwing away half the evidence about a failure it will later have to explain.

Two of those three arms deserve a note we would rather publish than hide. The arm with no -s flag and the arm with -s read-only reached the same sandbox state by two different routes, so their rows are not independent evidence: in our measurement, codex exec with no sandbox flag behaved exactly as -s read-only did. We left both rows in the original table with that note rather than quietly dropping an arm, and the same honesty applies to the count here: 6 of 6 blocked runs is three plus three, from two arms that turned out to be one condition.

Is the agent's own prose a signal you can use?

The prose of Codex is honest about failure and unreliable about cause, and both halves matter if you plan to read transcripts. In every blocked run of 19 August 2026, Codex said in plain English that it could not write the file. It never claimed success it did not have. That is the good half, and it is worth saying plainly because the opposite behaviour is common enough to be the reason this series exists.

The bad half showed up on 21 August 2026, when the block came from a macOS sandbox-exec profile applied from outside the process. Codex reported Operation not permitted and blamed the current directory in 3 of 3 blocked runs. The directory was drwxr-xr-x and owned by the user, so the explanation was wrong while the report of failure was right. It never named the layer that actually stopped it. For automation the consequence is narrow and practical: use the prose to know that something failed, never to know what failed, and read status before you read the sentence. The diagnosis measurement is in why does Codex say Operation not permitted when the directory looks writable.

In what order should your CI read the codex exec signals?

Read them in the order of what each one can prove, from the weakest to the strongest, because a CI job that reads them in the wrong order reports the wrong thing. On the 25 codex exec runs published on 19 and 21 August 2026, the ladder came out like this.

  • Exit code. Use it only to detect that the run itself broke, as in the turn.failed case with Selected model is at capacity. It was 0 in 9 of 9 runs including 6 that changed nothing, so it cannot mean success.
  • Presence of a file_change item. A write was attempted at that path. Clean separation in our runs, 3 of 3 against 0 of 6.
  • status on that item. Whether the write landed. Correct in 10 of 10 items on codex-cli 0.148.0, unverified by us on 0.147.0.
  • kind, as a balance per path. Whether anything survived. This is what catches the create then delete that the event count misses.
  • ERROR on standard error. Why it was refused. Present in 6 of 6 blocked runs.
  • A hash of the working tree before and after. The only judge that is not the agent, and the one we used to grade every arm above.

The last line is the one to keep if you keep only one. Every number on this page exists because the apparatus fingerprinted the repository outside the agent's reach, and any of the signals above can be verified on your own machine against that same check.

How does the codex exec envelope compare with the Claude Code envelope?

The codex exec envelope carries a per file event and the Claude Code envelope, in our measurement, carried nothing equivalent. We ran the same style of measurement against Claude Code on 18 August 2026 (Claude Code 2.1.235, model sonnet, macOS 26.5.2): 13 of 13 runs came back with subtype: "success" and is_error: false, and 10 of those 13 had changed nothing in the repository. There was no field playing the role that file_change plays for Codex, so the envelope alone could not distinguish work from silence.

Treat that as a comparison of two envelopes on two dates and two builds, not as a verdict on either agent. The Claude Code figure is from 18 August 2026 and the Codex figures from 19 and 21 August 2026, on codex-cli 0.147.0 and 0.148.0, and agents in this category ship several times a week. What generalises is the shape of the question rather than the score: before you automate around any coding agent, find out which of its output fields answer "was a write attempted", "did it land", and "did anything survive", and if the answer to any of the three is "no field", that gap is yours to cover with a hash.

What this index does not show

This index carries the limits of the measurements it consolidates, and they are real ones. Every number comes from one machine, macOS 26.5.2 on arm64, in throwaway git repositories containing a handful of files, on 18, 19 and 21 August 2026. The samples are single digit per arm, three runs each, and a single digit count cannot separate a rare failure from an impossible one. Two builds of codex-cli are mixed here, 0.147.0 and 0.148.0, and we say which is which on every claim precisely because they are not the same tool.

Several things stayed untested. We did not read status on 0.147.0, so its history is unknown to us. We did not test the interactive Codex TUI, only codex exec. We did not test a partial write, a file left half written by an interrupted process, which is the case where a balance per path might be honest and useless at once. We did not test any sandbox other than the macOS profile and the built in -s policies, and Linux may differ. Two contaminations of the 21 August 2026 apparatus travel with the prose finding above and are declared in the source article: the working directory was named /private/tmp/codex-sandbox-21ago-n1, so the string codex-sandbox was printed on screen by pwd and ls in 3 of 3 blocked runs, and in one of those runs the agent opened our harness's stderr.txt and read the wrapper's error log. Neither changed the outcome, and both mean no sentence here can claim the agent saw only what Codex shows. At CanvasCode we run several coding agents side by side, which is exactly where an envelope that cannot tell work from silence costs the most, because the run you did not watch is the one whose summary you will believe.