how to tell if your AI agent's cron job silently died

Not from the exit code. That’s the whole problem. A scheduled agent run can report success having accomplished nothing, because an in-run failure comes back as content: Claude Code “prints the failure as the result on stdout,” and the docs never say that run exits non-zero. Check the artifact instead, then the error fields in the JSON result, then whether your alert path can fire at all.

why the exit code lies

It lies in both directions, which is what makes this failure so quiet.

In one direction, a dead run reports success. The headless documentation promises that “Claude Code exits with code 0 on success and a non-zero code when the run fails,” and then adds the sentence that undoes it for scheduled work: “When a failure happens inside the run, such as missing authentication, Claude Code prints the failure as the result on stdout.” Missing auth isn’t an error your script can catch. It’s a string. Your wrapper tests whether the output is non-empty, the string is very much non-empty, and the job records a win.

In the other direction, a successful run reports failure. Interpreter crashes land after the work is finished as happily as before it. 139 is SIGSEGV via bash’s documented 128+N convention, 137 is SIGKILL (usually the OOM killer), 143 is SIGTERM — Claude Code specifically documents exit 143 when a supervisor stops a run — and 124 is GNU timeout’s own code for “I killed it.” None of those tell you whether the job did its job first. Retry on a bare non-zero code and you’ll cheerfully duplicate completed work.

So the only honest success test is the artifact. Did the file appear, did the row land, did the message send. Not the status, and not the log line that says it sent.

what a silently dead run actually looks like

Four shapes, three of them with public receipts, in rough order of how hard they are to notice.

It never started. Ken Imoto put seven agents on cron and wrote up the result: “Two of them stopped running on day one. I noticed on day eighteen.” The cause was a renamed Python helper: the wrapper died at exit 1 under set -euo pipefail, with stderr redirected to /dev/null and no MAILTO= in the crontab. The status was non-zero the entire time and nobody was positioned to read it. (64 is the code he assigned that failure afterwards, in the exit-code contract he bolted on.) His tracing missed it too, because there was no trace to collect — nothing ran. One person’s write-up rather than an incident report, but the failure mode is exact.

The ticker itself died. Hermes issue #20302, open since May 2026, is titled “cron: no health monitoring for cron ticker — can silently stop for hours” and reports that “a syntax error caused the cron ticker to stop for 7.5 hours with zero alerts.” The gateway stayed up and answered normally the whole time. Everything looked healthy because almost everything was.

One wedged turn starved the rest. botnexus #4688, filed October 6 and closed October 14, is the cleanest write-up I’ve seen of the agent-flavored version: every agent-prompt cron job belonging to one soul-enabled agent collapses onto a single-turn Agent instance, so one wedged turn stopped roughly 20 of them. The reporter measured “a 15h19m total cron outage that was discovered by a human, not by monitoring.” Worth knowing that this is a two-star personal lab, not a popular framework — the receipt is specific, the sample size is one host.

The error got delivered as content. This is the one I shipped myself, so I’ll own it. A scheduler I run sends a periodic message to an actual person. The engine lost its credentials and started returning the string Not logged in · Please run /login, the only validation in the path was if message:, and that string is truthy. It went out ten times over four weeks before I read what was actually being sent rather than confirming that sending had happened. Exit code 0 every time. The monitoring was perfect and measured the wrong thing.

is this actually AI-specific?

Partly, and less than the “LLM turn blocks the scheduler loop” story going around suggests. I went looking for that mechanism and mostly didn’t find it.

OpenClaw hardcodes DEFAULT_CRON_MAX_CONCURRENT_RUNS = 8 in cron-limits.ts, and the resolver takes no config argument at all. The cron block in its config schema is a strict object, so cron.maxConcurrentRuns — the setting half the tutorials tell you to tune — now fails validation rather than being quietly ignored. Hermes dispatches due jobs into a persistent thread pool that its own docstring calls “unbounded” by default, which Python then silently caps at min(32, cpu_count + 4) — six workers on a two-core box. (HERMES_CRON_MAX_PARALLEL=1 restores serial execution.) Self-hosted n8n ships with concurrency control disabled. Serial starvation is real in botnexus today, and OpenClaw had its own flavor of it until a nested-lane bottleneck got closed out. It’s an implementation detail to check in your own stack, not a law of agent schedulers.

What is genuinely AI-specific is the lying status. A classic cron job that can’t authenticate dies with a non-zero code. An agent that can’t authenticate writes you a polite sentence about it and exits clean.

check 1: judge success by the artifact

Write the success condition against the thing the job exists to produce, and make it stricter than existence:

# -s, not -f: an empty file is a failed run, not a successful one
[ -s "$OUT" ] || { notify "run produced no output"; exit 1; }

The -f version of that line is a bug I also shipped. The engine crashed after truncating its output file, the file existed, and the fallback notice stayed quiet.

check 2: read the JSON result, not the text

--output-format json gives you fields that can’t be faked by a plausible sentence. Two of them are worth more than the rest, and the TypeScript reference has a detail that tells you everything about this class of bug: is_error is a boolean on both arms of the result type, including the one whose subtype is literally "success", and api_error_status — “the HTTP status code of the API error that terminated the conversation” — sits on that same success arm. The docs then describe another field as present only “on a success result with is_error false,” conceding that the other kind exists.

A result labeled success can carry the status code of the error that ended it. Read is_error and api_error_status, not result.

jq -e '.is_error == false and .api_error_status == null' <<< "$RESPONSE"

With --output-format stream-json you also get system/api_retry events whose error field is a fixed category — authentication_failed, billing_error, account_on_hold, rate_limit, overloaded, invalid_request, cloud_credential_error and friends. Those are the machine-readable version of the sentence that otherwise ends up in your output file. Or in somebody’s inbox.

And if the output goes to a human: validate it before it ships. Length floor plus a blacklist of not logged in, /login, credit balance, rate limit. If it trips, alert yourself and send nothing.

check 3: read your alert path backwards

Ask one question about the branch that notifies you: under what state does this not fire? If the answer is “a genuine failure,” you’ve found the bug.

Mine sat under elif [ "$FAILED" -eq 0 ]. The fallback notice that existed specifically to tell me the job produced nothing was gated behind the engine having succeeded. Routine failures were covered. Real ones were silent.

OnFailure= in systemd deserves the same question. It activates units “when this unit enters the ‘failed’ state” — a state, not a judgement about output. An agent that exits 0 with an error inside never enters it.

the systemd defaults that keep it quiet

Four traps, all documented, all defaults:

• TimeoutStartSec= is disabled by default for Type=oneshot — which is exactly what scheduled agents are. A hung run hangs forever.
• RuntimeMaxSec= has no effect on Type=oneshot. The docs say so outright, because oneshot units terminate when activation completes. People reach for it and get nothing.
• StartLimitBurst=5 within StartLimitIntervalSec=10s. A crash-looping unit exhausts its burst and systemd stops auto-restarting it, with no new alert. A timer or socket can still start it once the interval passes; systemctl reset-failed flushes the counter immediately. Both numbers are manager defaults, so don’t hunt for them in the unit file.
• AccuracySec= defaults to 1min. Your 09:00 timer fires somewhere in a one-minute window, wider if you set RandomizedDelaySec=. Size your monitor’s grace time accordingly.

Two things worth adding. Persistent=true on an OnCalendar= timer fires one catch-up run after downtime — one, not every slot you missed. And systemctl show -p LastTriggerUSec <timer> gives you the last fire time in a form a script can read, with a trap that belongs in this article: a timer that has never fired and a unit that does not exist both return an empty value and exit 0. Test the string, not the status. systemctl list-timers lists only timers currently in memory and hides inactive ones unless you pass --all, so absence there reads exactly like “nothing due.”

how do I get alerted when it dies?

Something outside the box has to notice that a ping didn’t arrive. That’s a dead man’s switch, and the free tiers are enough for a personal fleet.

Service	Free tier	Paid	Notable
Healthchecks.io	20 checks	$20/mo for 100	BSD-3, self-hostable, /start + /fail + /<exit-status>
Cronitor	5 monitors, 5-min floor	$2/mo per monitor + $5/user	sends duration, status_code, custom metrics
Dead Man’s Snitch	1 snitch	$19/mo for 100	the free tier is a demo

Healthchecks wins on the free tier, and the /start suffix is the part that matters here. Ping /start before the run and the success URL after, and a run that starts and never reports back trips the alert once grace time expires. That catches the hang, not just the no-show. The full contract is three lines:

curl -fsS -m 10 "https://hc-ping.com/$UUID/start"
./run-agent.sh; RC=$?
[ -s "$OUT" ] || RC=1          # artifact decides, not the exit code
curl -fsS -m 10 "https://hc-ping.com/$UUID/$RC"

Exit status 0 reads as success there and anything non-zero as failure, so that last line reports the real verdict rather than the engine’s opinion of itself. If you self-host Healthchecks, note that alerts come from a long-running sendalerts management command: forget it and your monitor fails silently too. Watchmen all the way down.

A loud failure costs you an alert. A quiet one costs you eighteen days, and you find it because somebody finally asks why the reports stopped arriving — which is roughly how the people in the receipts above found theirs.