The day /dev/null broke every scheduled job.

What happened

In October 2026 every scheduled command in our production Laravel app started failing, and each run ended in about 2 ms. The jobs themselves were fine. Laravel starts each scheduled event with its output redirected to /dev/null, and /dev/null on the server had become crw-r--r-- owned by www-data. The app user could no longer write to it, so the shell refused to start every command with "cannot create /dev/null: Permission denied". Only the jobs that logged to a real file kept running.

The first window ran from the afternoon of 22 October to late on 23 October. It came back on the evening of 24 October, and that time I did not catch it until the morning of 28 October, three and a half days later. In the second window the scheduler logged more than 43,000 failed runs: scheduled orders, payment watchers, item re-enabling, campaign dispatch, the support desk tick, presence recording and more. The at daemon could not start either, so any at jobs queued before the fix may have been lost.

How I found it

It surfaced while I was moving our cron jobs to Go. The run history showed every Laravel job failing within milliseconds, which pointed at something below the jobs rather than in them. The error message led to /dev/null, and its change time showed exactly when it had been altered. I checked my own deploy scripts from around that time and they were not the cause. The second time, the same symptom took me straight to ls -la /dev/null.

The fix

I put /dev/null back to crw-rw-rw- root root with one standard chown and chmod. The next scheduler run was clean, all jobs done.

Then I went through the fallout job by job. One job that uses an overlap lock stayed blocked for about 8 hours, because the failed run never released its lock. A membership expiry sweep that had missed its runs was run by hand. The rest of the missed runs were safe to skip.

While digging I found two more problems. Production had two schedulers, a systemd service running schedule:work and a root crontab line running schedule:run, so some jobs ran twice; I commented out the duplicate crontab line. And daily jobs without an explicit timezone were running in UTC, 5 hours 45 minutes late. I built a fix that sets the scheduler timezone to Asia/Kathmandu, reviewed each job against it, proved it on our test environment, and it now runs in production.

What changed

I still do not know what changed /dev/null. My best guess is a root command that copied a file with its owner and mode onto it, but I never proved it, and I would rather say that than invent a cause.

What I changed is how fast it gets caught. A small guard now checks /dev/null every minute; if the owner or mode is wrong, it puts it back and writes a log line. It has already earned its place: on 1 November /dev/null was changed a third time, and the guard repaired it within a minute, before a single scheduled job failed. "Every job fails in milliseconds" also now means checking /dev/null first.

The bigger lesson was the three and a half days. A scheduler that fails silently is worse than one that crashes, so a job that has not succeeded in its expected window should page someone. The guard fixes this one cause; that alert is what catches the next one.