Background Jobs and Workers: When to Move Work Out of the Request Cycle

The practical rule is simple: any work that takes more than a fraction of a second, and whose result the user is not waiting to see right now, belongs outside the request cycle in a background queue handled by a separate worker. Sending a welcome email, generating a thumbnail, building a PDF report, calling a slow third-party API, each of these needlessly stretches page response time when run inside the request. The idea is that the user's request writes a "job" to a queue and returns immediately in around 50 milliseconds instead of 3 seconds, while a worker running in the background picks up that job and processes it quietly. The direct result is an interface that stays fast under load, heavy tasks processed in parallel, and a temporary failure in an external service that does not break the user experience because the worker retries automatically.

In Laravel there are three components: a Job that describes the work, a queue driver that stores it, and a worker, a long-lived process that pulls jobs and runs them one after another. You add ShouldQueue to the class and it goes to the queue instead of running inline. The simplest driver is the database, where a jobs table holds pending work, and that is enough for small loads. Once volume grows, or you want thousands of jobs per second, Redis is far faster because it lives in memory. This design fits exactly the companies with custom systems we build alongside on the cloud server, where queues separate the fast interface from heavy processing.

What Deserves a Queue and What Stays Inline

Not every task needs a queue. The rule is to ask: is the user waiting on this result right now? If they need the invoice number immediately after payment, compute it in the request. Emailing a copy of that invoice, though, goes to the queue, because the user is not standing at the screen waiting for the message to arrive. The table below shows the practical split:

Work	Inline in request	In the queue
Form validation	Yes	No
Sending email or WhatsApp	No	Yes
Generating or compressing thumbnails	No	Yes
Building a large PDF report	No	Yes
Calling an external payment API	Depends	Often yes
A price shown to the user instantly	Yes	No

The deciding factor is perceived response time. If running the work inline raises page time from 80 milliseconds to 2 seconds, you pay a steep price in every user's experience for work that could have been deferred without anyone noticing.

Choosing the Driver: Database or Redis

The database driver writes each job as a row and the worker reads it with a lock. It is easy to set up, needs no extra service, and covers tens or hundreds of jobs per minute. Its weakness is that every job pull is a disk query, so under high load the table itself becomes the bottleneck. Redis keeps the queue in memory, so the worker pulls a job in fractions of a millisecond and handles thousands per second without weighing down your primary database. The practical rule: start with the database if your load is light, and move to Redis once you pass a few hundred jobs per minute or you see the queue falling behind. A Growth server with 4GB of RAM comfortably holds Redis alongside your application, and each typical job uses only a small amount of memory.

Failure, Retries, and the backoff Policy

External services fail temporarily: the payment gateway lags, the mail server refuses the connection for a few seconds, a third-party API returns a 503. The queue's great advantage is that it retries instead of dropping the work. You set the attempt count with tries and the per-attempt limit with timeout, and you delay each retry with an escalating backoff, say 10 seconds, then 60, then 300, so you do not hammer a struggling service with back-to-back attempts. When all attempts are exhausted, the job moves to the failed_jobs table instead of vanishing silently. Watch that table daily, because it is the first sign of a real problem: a sudden pile-up means a service went down or a job carries a code bug. You requeue failed jobs with a single command once the cause is fixed.

The idempotency Principle

Because a job may run more than once during retries, write it so that repeating it does no harm. Do not create a new invoice on every run, check first whether one exists. Make the outcome the same whether the job runs once or three times, and you are protected from duplicate messages or double charges.

Keeping Workers Alive: supervisor After Every Deploy

A worker is a long-lived process that does not end with the request, and if you stop it the queue stops. So do not run queue:work by hand in a terminal, put it under a process manager like supervisor that restarts it automatically if it crashes or the server reboots. You tune worker count through the number of processes: two or three cover medium loads, and you add more when queue wait time grows. A point many overlook: a worker loads the code into memory when it boots, so it will not see your changes after a deploy until you restart it. Make a final step in your deploy script run queue:restart so workers pick up the new code. Add an upper timeout per job so a worker never hangs forever on a broken task.

Monitoring Health by the Numbers

Track three numbers: queue length (how many jobs wait right now), wait time (how long a job sits before a worker takes it), and failure rate. If queue length climbs steadily, you have fewer workers than needed or your jobs run slower than they arrive. A wait time that jumps from seconds to minutes at peak means you need extra workers or priority queues, a fast queue for notifications and a slow one for heavy reports, so a huge report never blocks an urgent alert.

Priority Queues and Ordering the Work

As your app grows you discover that not all tasks are equally urgent. A notification telling a customer their payment succeeded should go out within seconds, while a huge monthly report can wait minutes. If you mix them in one queue, the urgent notification may sit behind a report that takes two minutes and arrive late for no reason. The fix is to split work across named queues: a high queue for urgent notifications, a default queue for ordinary work, and a low queue for reports and heavy tasks. Then you distribute workers so some serve the fast queue first. The practical rule: split by expected run time and urgency, not by feature type, because what matters to the user is when the effect reaches them, not the name of the job that produced it.

Also watch each job's memory footprint. A job processing a high-resolution image may use tens of megabytes while running, so if you run ten parallel workers on such jobs, consumption multiplies by ten. On a server with 4GB of RAM, this means balancing worker count against job size, not just adding workers because the queue is long. Sometimes two workers processing steadily beat eight competing for memory that runs out and gets them killed by the system.

Timing Deferred Jobs

A queue does not only run work immediately, it also schedules it for the future. You delay a follow-up message to an hour after signup, or schedule a daily report at three in the morning when load is low. This turns the queue from a load-shedding tool into a smart timing tool, spreading heavy work over quiet hours instead of crowding the daytime peak. Combine that with monitoring queue length, and you see clearly when you need an extra worker and when what you have is enough.

Testing Jobs Before Deploy

A queued job is code like any other, and it deserves a test before you rely on it in production. Test that it does the right work, that it behaves safely when repeated, and that it fails cleanly when the external service fails. The common mistake is writing a job that works on the first try, then discovering in production that it duplicates an email or charges twice on a retry. Testing the job on both success and failure paths saves you embarrassing incidents in front of your customers, and makes your trust in the queue rest on evidence rather than hope.

Conclusion: Practical Steps

Start by moving every task the user is not waiting on into a job that carries ShouldQueue. Use the database driver if your load is light, and switch to Redis as you scale. Set tries and an escalating backoff, write jobs so repetition is harmless, and watch the failed_jobs table daily. Run workers under supervisor, and add queue:restart at the end of every deploy. If you are building a custom system that needs reliable queues on a Saudi server with roughly 8ms response time from Riyadh, review the pricing and plans and pick a server that fits your app and Redis together, or build your AI on AI hosting with a ready n8n template to run background flows without code.