Multiple Queues in Laravel

Question

I am creating a web application in laravel in which bidding is being done by users in multiple games. Bidding is being performed by front end users and by cron job as well. Cron job do bid on each game after each second. Therefore some collision was happening between bids when same row was accessed at same time. To resolve concurrency issue I decided to use laravel queues for bidding. But I am having multiple games and therefore I want simultaneously bids of each game. I don't want bids of same game to be process at same time because then concurrency issue can be occur. I want to know about multiple queues system in laravel. After having search on internet I got to know about multiple queues like

php artisan queue:work --queue=myJobQueue, myJobQueue1, myJobQueue2,..myJobQueue7

But I am not sure how it works. Please someone explain me in detail that all 7 queues work simultaneously or one by one.

Solution 1

Are looking for the queue:listen command?

queue:work will process all pending jobs that are stored by the queue driver, whereas queue:listen will be waiting for jobs to be thrown at it to execute them as they come.

If you do php artisan queue:listen --queue=myJobQueue, myJobQueue1, myJobQueue2,..myJobQueue7, 7 queues are being created and listening to new tasks on their own.

In your code, you can dispatch jobs like the following:

dispatch((new MyJob)->onQueue('myJobQueue'));

You might want to use a tool like Supervisor to make sure queue:listen is always running in the background.

Hope this helps!

Solution 2

php artisan queue:work --queue=myJobQueue, myJobQueue1, myJobQueue2,..myJobQueue7 sets the priority in which queues will be executed. So with this all jobs on myJobQueue will be executed before moving to execute jobs on myJobQueue1 then to myJobQueue2 in that order.

However if you want jobs on these queues to be executed simultaneously, you could run each queue in the background.

php artisan queue:work --queue=myJobQueue & php artisan queue:work --queue=myJobQueue1 & php artisan queue:work --queue=myJobQueue2 &

This will run each queue as single processes in the background.

Solution 3

Like Ben V said, it is highly recommended to use Supervisor to keep the workers active at all times, especially if you want to run one or more workers per queue, or if you want the queues to be processed simultaneously.

Here is an example Supervisor configuration file:

[program:laravel-worker-myJobQueue]
process_name=%(program_name)s_%(process_num)s
command=php artisan queue:work --queue=myJobQueue
numprocs=8
autostart=true
autorestart=true

[program:laravel-worker-myJobQueue1]
process_name=%(program_name)s_%(process_num)s
command=php artisan queue:work --queue=myJobQueue1
numprocs=1
autostart=true
autorestart=true

The above configuration creates 8 workers for myJobQueue, and one worker for myJobQueue1, since multiple workers can help speed things up, but can cause trouble for jobs that try to access the same row in the database, in which case you want to limit things to 1 worker only.

You then simply dispatch jobs to the correct queue using

dispatch((new MyJob)->onQueue('myJobQueue'));

or

dispatch((new MyJob)->onQueue('myJobQueue1'));

Solution 4

This might be old but just in case, all of the answers are on point, but you must set the .env variable QUEUE_CONNECTION to something else than sync,if your configuration is set to sync it will take every job in order of entry to the queue (thus finishing one and then starting the next one), if it's set to database or redis it will be taking jobs in parallel if needed (which is the idea of setting priorities) you should check this article (it helped me) https://medium.com/hexavara-tech/optimize-laravel-with-redis-caching-made-easy-bf486bf4c58 also you will need to configure your queues in config/queue.php like such for example in the 'connections' array:

    'database' => [
        'driver' => 'database',
        'table' => 'jobs',
        'queue' => ['default','another_queue'], //this is just 'default' by default
        'retry_after' => 90,
    ],

this applies for all the other configurations in this file.

View more solutions

Comments

[Pratik]

unfortunately I used same type of supervisor config but it seems my job is being processed in queue after worker1 is done. It doesnt happen simultaneously.

[Paul Preibisch]

Btu how do we configure myJobQueue, and myJobQueue1,2... where is the config for that?

[habib]

remember to remove space after comma, otherwise the command give error (I experience it on windows) command should be: php artisan queue:work --queue=myJobQueue,myJobQueue1