← Writing

One queue worker was not enough.

Splitting Laravel's background jobs into an order-critical queue and a low-priority one, with a systemd worker pool, so a GPS flood can't delay dispatch.

Sakshyam Baral··4 min readinfralaayo

Laayo's Laravel app does a lot of work in background jobs. Some of it is urgent: telling nearby riders about a new order, sending reminders when a restaurant hasn't accepted, working out delivery distance. Some of it is not: storing batches of rider GPS points, nightly summaries, fraud checks on fuel claims, making WebP copies of uploaded images.

For a long time all of it went through one queue and one worker. The worker was started by cron every minute, guarded by a file lock so only one ran at a time, and told to exit when the queue was empty. It was simple and it worked when the app was quiet.

The problem was ordering. A queue is first in, first out. GPS points arrive in large batches, and those jobs went into the same line as "notify riders about this order". A big enough burst of GPS work would make the order notification wait behind it. Nothing would fail. Dispatch would just get slower, and a rush is exactly when there is the most GPS data and the most orders at the same time.

Two queues, by urgency

The fix was to separate jobs by how much a delay costs, not by what they do. There are now two queues: default for anything on the order path, and low for background work that can wait.

I moved four job classes to low: GPS log ingestion (the main source of floods), the fuel fraud check, the daily fuel summary and the WebP image copies. One of those was also dispatched explicitly onto default at its call site, so I fixed that too. Everything on the order path stayed where it was. I deliberately didn't touch those classes, because they sit next to the money code and the change didn't need them.

One Laravel detail matters here. Setting the queue as a class property does not work:

class IngestRiderGpsLogsJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
 
    public $queue = 'low'; // fatal: conflicts with the Queueable trait
}
class IngestRiderGpsLogsJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
 
    public $queue = 'low'; // fatal: conflicts with the Queueable trait
}

The Queueable trait already declares that property, and PHP refuses a class that redeclares it with a different default. The working version sets it in the constructor:

public function __construct(public array $rows)
{
    $this->onQueue('low');
}
public function __construct(public array $rows)
{
    $this->onQueue('low');
}

A worker pool under systemd

Then the workers. The cron-started worker went away, replaced by systemd services that stay running:

  • a template unit for default workers, so I can run a small pool of them as numbered instances
  • a separate unit for the low queue

The important property is that the default workers never take jobs from low. A GPS flood can fill the low queue as much as it likes, and the order queue still has its own workers waiting for it. Each worker runs one attempt per job, restarts automatically if it dies, and exits on its own after a while so that systemd starts a fresh process. That periodic recycle keeps memory leaks in long-running PHP from building up.

I also kept the old cron line written down, together with the commands to stop the new units, so the rollback was one copy and paste away.

The function the host had disabled

The first start failed immediately. Laravel's queue worker calls pcntl_signal() to handle timeouts and graceful shutdown, and it does so unconditionally. The PHP configuration on my server, a hosting panel default, lists all the pcntl_* functions in disable_functions. So the worker crashed on start.

The old cron worker had the same problem and already ran with an override, so the fix was the same: start the worker with disable_functions cleared for that process only.

ExecStart=/usr/bin/php -d disable_functions= artisan queue:work redis --queue=default
ExecStart=/usr/bin/php -d disable_functions= artisan queue:work redis --queue=default

That re-enables pcntl for the queue worker, and since nothing else was in the disabled list, it re-enables nothing else. Web requests still run with the panel's settings.

Two details for anyone who hits the same wall: passing a zero timeout to the worker does not avoid the call, and running in --once mode does avoid it, but then you are back to starting a process per job.

Checking it worked

I didn't want to trust the configuration, so I watched it. I dispatched a GPS ingestion job and checked that it landed on the low queue in Redis, not on the default one. I watched the low worker pick it up while the default workers stayed idle. Both queues drained to empty, the site kept answering, and the server load stayed low.

Background jobs are part of the Go migration too. New jobs are already written as outbox rows, picked up by a Go worker from NATS, and each existing job will move over with its own switch, the same way routes do. Until then, the Laravel pool keeps the order path out of the way of everything else.

Get new posts by email

One email when something new goes up. No tracking pixels, unsubscribe in one click.