Gateway timeouts on the home screen.

What happened

On 30 May 2026, at about 3:08 PM local time, the customer app's home screen showed a "Gateway Time-out" toast. At that moment about 14 home screen endpoints returned 504 together. The same limit had been hit three times in two weeks, always at peak hours, so it was going to get worse as we grew.

How I found it

The obvious suspect was the database, so I ruled it out first. The MySQL slow log was empty and iowait was close to zero. The CPU told a different story: load went from 5.9 to 9.8 on a 4 CPU machine, at 87% CPU, from the home screen's many parallel calls plus rider app traffic sending GPS points and polling current orders.

The access logs looked clean, which was confusing until I understood why. The Apache error log showed that Apache reached its MaxRequestWorkers limit of 150 at 15:06:34. With no free Apache thread, nginx could not even finish the TLS handshake to Apache, and its error log had "upstream timed out while SSL handshaking". The 504s never appeared in the access log because those requests never reached Apache.

Under that was the real squeeze. The production PHP-FPM pool was set to ondemand with max_children = 8: eight PHP processes at most, none kept warm, behind 150 Apache threads. Apache threads piled up waiting for PHP, and once they were all taken, nginx had nowhere to go.

The fix

I changed the pool to pm = dynamic with max_children 30, start_servers 8, min_spare_servers 6 and max_spare_servers 16, and kept max_requests at 4000. Each worker used about 110 MB, so 30 workers is roughly 3.3 GB, which fit in the free memory with room to spare. I reloaded PHP-FPM gracefully instead of restarting it.

To check it, I fired 14 parallel home screen requests. All returned 200 in 0.28 to 0.68 seconds. I kept a copy of the old pool file so a rollback would be one copy and one reload.

What changed

The pool file is generated by the hosting control panel, and a panel rebuild of the site quietly puts it back to eight on-demand workers. I wrote that down next to the rollback steps and moved on. That was the mistake.

A rebuild did undo it. By late September the pool was back to eight workers, and the 504s returned at busy hours on three days within a week. I raised it again in October. Then in November the control panel updated itself, rebuilt the site and reset the pool once more, at night, with nobody watching.

So the setting now lives where the panel cannot lose it: a custom PHP-FPM template for this site, which the panel uses every time it rebuilds. A fix that a routine rebuild can undo is only half a fix, and I had to learn that twice. Now anything I change in a generated config gets the same question: what regenerates this file, and will my change survive it?

I also found a small app bug while reading the logs. The app's error handler looked for "gateway timeout" without the hyphen, but the real reason phrase is "Gateway Time-out", so a 504 showed the raw phrase to customers instead of our friendly error message. We decided not to ship an app update only for that, so it stayed a known issue, and it is still open: when the app moved to native code, the same check came along with it. It is a one-line fix, on my list for the next build. It reminded me that the error path is also part of the product, and it needs tests like everything else.