August 19, 2026

How do you know whether Codex actually changed a file?

Short answer: not from the exit code. In 9 runs of codex exec on August 19, 2026 (codex-cli 0.147.0, macOS 26.5.2 on arm64), the process exited 0 every single time, including the 6 runs where the repository came out byte for byte identical to how it started. Two signals inside the output do tell the truth: an item.completed event of type file_change on standard output, which appeared in 3 of 3 runs that edited the file and 0 of 6 that did not, and an ERROR line on standard error reading patch rejected, which appeared in 6 of 6 blocked runs and 0 of 3 that wrote.

The agent itself is not the problem, and that is the sentence worth carrying away: in every blocked run Codex said in plain English that it could not write. The prose is honest and the status is not. We ran the same measurement against Claude Code on August 18, 2026 and its JSON envelope had no field equivalent to file_change: 13 of 13 runs came back with subtype: success and is_error: false, 10 of them with nothing changed. If you are wiring an AI coding agent into CI, the difference between those two envelopes decides whether your pipeline can tell work from silence.

What did codex exec return in 9 runs on August 19, 2026?

We gave codex exec the same small task nine times, in nine throwaway git repositories, one per run: add a slugify function to src/utils.js, export it, then run npm test and make sure it passes. Three arms, three runs each, which is a small sample, and the full script that produced them is published verbatim in the reproduction section of this article. The arms differ only in the sandbox policy: -s workspace-write, no sandbox flag at all, and -s read-only.

The judge is not the agent. Before and after each run, the apparatus hashed every non-.git file in the repository and compared the two fingerprints, so "did anything change" is answered outside the agent's reach. Here is the table, reconstituted by script from the nine raw JSONL files rather than counted by hand:

Arm	Runs	Exit code	Repository changed	`file_change` on stdout	`ERROR` on stderr
`-s workspace-write`	3	0 in 3 of 3	yes in 3 of 3	1 in 3 of 3	0 in 3 of 3
no sandbox flag	3	0 in 3 of 3	no in 3 of 3	0 in 3 of 3	1 in 3 of 3
`-s read-only`	3	0 in 3 of 3	no in 3 of 3	0 in 3 of 3	1 in 3 of 3

A caveat the table cannot carry on its own: the second and third rows are the same sandbox state reached by two different routes, because in our measurement codex exec with no -s flag behaves exactly as -s read-only does. We left both rows in with this note rather than quietly dropping an arm. Nine runs, nine exit codes of zero, six untouched repositories, and the failure visible on standard error in all six of them while the exit code stayed at zero.

Why is the exit code useless for telling whether Codex did the work?

Because the exit code of codex exec answers a different question than the one you are asking. It answers "did the session terminate normally", and in all nine of our runs it did. It does not answer "was the task completed", and nothing in the CLI contract says it should. The two questions look identical from a shell script, which is exactly why this costs people afternoons.

The failure mode is specific and quiet. A CI step that runs codex exec "fix the failing test" and then checks $? will see 0 whether the agent rewrote the module or sat in a read-only sandbox explaining that it could not. The step goes green. The next step runs. Nobody looks at the transcript, because the transcript is what you read when something breaks, and nothing broke.

We measured the same blindness in Claude Code on August 18, 2026, which makes us suspect an industry default rather than one vendor's choice. Two vendors is suggestive and not proof, and it is worth saying that here rather than only in a limitations section at the end: what we can claim is that two CLIs from two companies, measured in the same week, both exit 0 for a run that produced nothing. If your pipeline treats a zero exit from a coding agent as proof of work, it has been reporting a success it never verified. The fix is not to argue with the exit code. The fix is to stop asking it this question and ask the repository instead.

Which signals in codex exec output actually distinguish work from silence?

Two, and they sit on different streams. On standard output, run codex exec with --json and the CLI prints one JSON object per line. In the runs where slugify actually landed in the file, this line appeared:

{"type":"item.completed","item":{"id":"item_2","type":"file_change","changes":[{"path":"/tmp/codex-envelope-19ago/repo-permitido-1/src/utils.js","kind":"update"}],"status":"completed"}}

It carries the path and a kind, which was update in our case, and it agreed with the tree fingerprint in 9 of 9 runs: present in all three that changed the repository, absent in all six that did not. That is a positive detector, and it fires when work happened rather than asking you to infer work from the absence of an error.

The second signal is on standard error, and it is the one we nearly failed to look at. In 6 of 6 blocked runs, and 0 of 3 that wrote, stderr carries this line:

2026-08-19T10:26:59Z ERROR codex_core::tools::router: error=patch rejected: writing is blocked by read-only sandbox; rejected by user approval settings

That line corrects a prediction of ours and an assumption we would otherwise have printed. The prediction, written down before we ran anything, was that a blocked write would surface inside the JSON stream as a command with a non-zero exit_code or as a failure event, and it did not: every command_execution item across all nine runs completed with exit_code 0, and no event on stdout carried a failure type. The assumption was that the agent never attempted the write at all. The log says rejected, not skipped, so it did attempt it. Treat both signals as strong hints rather than proof: agreeing with the truth in nine runs is agreement, not reliability, and we did not go hunting for the case where either one is wrong.

Does Codex admit it was blocked, or does it claim success?

Codex admits it, clearly, in plain English, in 6 of 6 runs where it changed nothing. This matters because "the agent lies" is the popular framing and our measurement does not support it. Below is the opening of the final message from three separate runs, each cut where the bracket appears, with the agent's own markdown backticks rendered as inline code and its words, including its typographic apostrophes, untouched:

"Blocked from editing: this workspace is read-only. Baseline npm test passes (1 test). The intended addition is: [...]"

"Blocked: this session’s workspace is read-only, so I couldn’t add slugify to src/utils.js. [...]"

"Unable to modify src/utils.js: the workspace is read-only and approvals are disabled. [...]"

Two of the six went further and printed the full function they would have written, so a human reading the terminal loses nothing but a copy and paste. The prose is honest. The status is not. That is the same split we found in Claude Code on August 18, 2026, where the agent wrote "I'm not able to complete this" and exited 0 anyway.

The consequence is uncomfortable and specific: the reader who is deceived is the one who automated. A developer watching the terminal sees the refusal in the last line. A shell script, a git hook, a GitHub Actions step and a scheduler all read the exit code, and the exit code says everything went fine. The more you take a human out of the loop, the more this defect costs you, which is the opposite of how tooling defects usually behave.

How is this different from Claude Code?

At the process boundary, it is not different at all. Both agents exit 0 when they did nothing, both write honest prose about being blocked, and neither one changes that answer based on whether work happened. Anyone deciding between them on this basis is choosing between two identical answers.

At the event boundary, the difference is real and it favours Codex. The --json stream from codex exec emits a typed file_change item with the paths it touched. When we measured Claude Code's JSON envelope on August 18, 2026, we found no equivalent: the fields available there (subtype, is_error, permission_denials) either said success in every run or, in the case of permission_denials, came back populated when a tool was tried and denied and empty when the denial stopped the attempt from happening, which makes it ambiguous in the exact case you need it. We did not run the same stderr comparison against Claude Code, so we are not claiming anything about what its standard error does or does not carry.

Two caveats keep this from being a scoreboard. First, the mechanisms that blocked the write are not the same: Codex blocks at an operating system sandbox, Claude Code headless blocks because an approval is required and there is no human to give it. Second, we tested one version of each, codex-cli 0.147.0 and Claude Code 2.1.235, and this is the kind of surface that changes between releases. What we are comparing is the output envelope each one hands to automation, not the quality of either agent. On the broader question of which of the two to use day to day, this measurement is one input and not the deciding one.

Why does codex exec change nothing when you pass no sandbox flag?

Because in our measurement, codex exec with no -s flag behaved exactly like -s read-only: the repository was untouched in 3 of 3 runs, no file_change event was emitted, the patch rejected line appeared on stderr, and the agent said the workspace was mounted read-only. We are reporting this as a measured behaviour of codex-cli 0.147.0 on macOS 26.5.2, not as a claim about what the documentation promises, and we note it because it makes our own experiment weaker: what we designed as two independent arms turned out to be one state reached by two paths.

It is still worth knowing, and it is the most practically useful line in this article for anyone who just installed the thing. The default non-interactive invocation is the one that most looks like it is working while doing nothing. The agent reads your files, reasons about them, spends tokens, writes a considered answer, and leaves the repository exactly as it found it. Nothing on standard output shouts. You get a thoughtful paragraph and an exit code of zero.

The practical move is to be explicit rather than to rely on a default you did not choose. If you want edits, pass -s workspace-write, which in our runs produced the change in 3 of 3 and emitted the file_change event every time. If you want a read-only analysis pass, pass -s read-only and mean it, so the intent is in the command instead of in your assumptions.

What should your CI check instead of the exit code?

Check the repository, not the report. The cheapest reliable check is a fingerprint of the working tree taken before and after the agent runs, computed by your pipeline rather than by the agent. Ours is three lines of shell and it is the same one that judged every number in this article:

impressao() {
  ( cd "$1" && find . -path ./.git -prune -o -type f -print0 \
    | sort -z | xargs -0 shasum -a 256 | shasum -a 256 | cut -d' ' -f1 )
}

Two cheaper signals sit in the output you are already capturing. If you parse --json, the file_change item names the paths the agent claims to have touched, which is far more than an exit code gives you. And if you redirect stderr, a plain grep ERROR catches the patch rejected line, which fired in 6 of 6 of our blocked runs and in none of the three that wrote. Between the fingerprint, the event and the log line you have three independent answers to the same question, and the exit code is not one of them. When any two of them disagree, you have found something worth reading the transcript for.

A fourth check costs nothing and catches the case the other three miss: run your test suite yourself, in your pipeline, after the agent exits. An agent that edits a file and breaks the build will produce a file_change event, a changed fingerprint and a clean stderr, and all three will look like success. This is the same discipline as verifying what an agent says it did against a source it does not control, applied to the one part of the loop where nobody is watching.

How do you reproduce this measurement?

Save the script below as rodar.sh, make it executable and run it. It creates its own throwaway repositories under /tmp, so it will not touch anything you care about, and it needs codex on your PATH plus Node.js for npm test. This is the file that produced the nine runs, published exactly as it ran, comments and bug included, because an article that tells you to distrust summaries has no business publishing a tidied-up version of its own method:

#!/bin/bash
# Aparato: o mesmo desenho de 18/ago, agora contra o codex exec.
# Mede o ENVELOPE (exit code + eventos JSONL) contra a VERDADE (impressao digital da arvore).
BASE=/tmp/codex-envelope-19ago
SAIDA=$BASE/saida
mkdir -p "$SAIDA"
TAREFA='Add a function named slugify to src/utils.js that lowercases the string, replaces every run of non-alphanumeric characters with a single hyphen, trims leading and trailing hyphens, and export it. Then run npm test and make sure it passes.'

impressao() { # hash de todo arquivo versionavel, calculado FORA do agente
  ( cd "$1" && find . -path ./.git -prune -o -type f -print0 | sort -z | xargs -0 shasum -a 256 | shasum -a 256 | cut -d' ' -f1 )
}

preparar() { # repo descartavel identico para cada run
  local dir=$1
  rm -rf "$dir"; mkdir -p "$dir/src"
  cat > "$dir/src/utils.js" <<'JS'
export function trim(s) { return s.trim(); }
JS
  cat > "$dir/package.json" <<'JSON'
{ "name": "descartavel", "version": "1.0.0", "type": "module", "scripts": { "test": "node --test" } }
JSON
  cat > "$dir/utils.test.js" <<'JS'
import { test } from 'node:test';
import assert from 'node:assert';
import { trim } from './src/utils.js';
test('trim', () => { assert.equal(trim(' a '), 'a'); });
JS
  ( cd "$dir" && git init -q && git add -A && git commit -qm base )
}

rodar() { # braco, indice, flags de sandbox
  local braco=$1 i=$2; shift 2
  local dir=$BASE/repo-$braco-$i
  preparar "$dir"
  local antes; antes=$(impressao "$dir")
  codex exec "$@" --skip-git-repo-check --json -C "$dir" "$TAREFA" \
    < /dev/null > "$SAIDA/$braco-$i.jsonl" 2> "$SAIDA/$braco-$i.err"
  local codigo=$?
  local depois; depois=$(impressao "$dir")
  local mudou=nao; [ "$antes" != "$depois" ] && mudou=sim
  # o que o ENVELOPE diz, lido do JSONL bruto
  local erros; erros=$(grep -c '"error"' "$SAIDA/$braco-$i.jsonl" 2>/dev/null || echo 0)
  local falhou_cmd; falhou_cmd=$(grep -o '"exit_code":[1-9][0-9]*' "$SAIDA/$braco-$i.jsonl" 2>/dev/null | wc -l | tr -d ' ')
  printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$braco" "$i" "$codigo" "$mudou" "$erros" "$falhou_cmd" >> "$BASE/placar.tsv"
  echo "[$braco-$i] exit=$codigo mudou=$mudou erros_json=$erros cmds_com_exit_nao_zero=$falhou_cmd"
}

printf 'braco\trun\texit\trepositorio_mudou\teventos_error\tcomandos_exit_nao_zero\n' > "$BASE/placar.tsv"
for i in 1 2 3; do rodar permitido  $i -s workspace-write; done
for i in 1 2 3; do rodar padrao     $i;                    done
for i in 1 2 3; do rodar bloqueado  $i -s read-only;       done
column -t "$BASE/placar.tsv"

The bug is in the two counting lines near the end. That grep -c ... || echo 0 idiom emits a stray line when the pattern is absent, which pushed a second line into every row and corrupted the last two columns of placar.tsv. Do not fix it and rerun on our account: what we did instead is the point. We threw away the broken tally and rebuilt every number in this article from the raw JSONL with a separate reader, which is the same move we are asking you to make with the agent's own summary.

That reader is the part worth copying. It opens each saida/<arm>-<n>.jsonl, parses one JSON object per line, counts item.completed entries whose item type is file_change and whose status is completed, counts command_execution items with a non-zero integer exit_code, and counts any event whose type contains failed or error. Those counters, joined to the exit code and the fingerprint verdict, are the whole table. Budget two to four minutes of wall clock for the nine runs and expect real token spend, since each run is a full agent turn against the API. The raw streams under saida/ are the evidence: if your numbers disagree with ours, the disagreement is in there and not in the summary.

What this measurement does not tell you

It does not tell you that file_change or the stderr line are trustworthy. It tells you that both agreed with the tree fingerprint in 9 of 9 runs, which is agreement, not reliability. An instrument earns credit when somebody hunts for the case where it is wrong, and we did not hunt: the obvious untested case is an agent that writes a file and then reverts it, where the event would fire and the fingerprint would not move.

It does not tell you anything about interactive use. Everything here is codex exec, the non-interactive path, which is the one that ends up in CI and in scripts. It also does not cover Cursor, Gemini CLI, Aider or any agent we did not run, and it does not cover what Claude Code writes to standard error, which we did not measure.

The sample is small and stated as such: three runs per arm, nine total, one machine, one morning, codex-cli 0.147.0 against Claude Code 2.1.235. Two of our three arms turned out to be the same sandbox state reached by different routes, which we found out from the agent rather than from our own design. And the sharpest limitation is one a reviewer found in us: our first draft claimed the blocked write surfaced nowhere as a failure, and our own stderr files said otherwise in six of nine runs. We had captured that evidence and not read it, which is an argument for the discipline in this article and not for our thoroughness. Finally, our task was deliberately easy and unambiguous, so we measured the difference between doing the work and being unable to do it. The messier and more common case, an agent that does part of the work and reports success, is a different measurement and we have not made it yet.