Laravel Jobs are not asynchronous

Question

I need a way to run some tasks asynchronously as the execution time varies between each task and I want to run the in an asynchronous way using Laravel Jobs and database as the driver.

I created to test jobs using the command line: php artisan make:job TestOne php artisan make:job TestTwo

TestOne.php

<?php

namespace App\Jobs;

use App\Jobs\Job;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;

class TestOne extends Job implements ShouldQueue
{
    use InteractsWithQueue, SerializesModels;

    /**
     * Create a new job instance.
     *
     * @return void
     */
    public function __construct()
    {
        //
    }

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        sleep(5);
        foreach (range(1, 10) as $item)
            \Log::info("TestOne: item #" . $item);
    }
}

TestTwo.php

<?php

namespace App\Jobs;

use App\Jobs\Job;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;

class TestTwo extends Job implements ShouldQueue
{
    use InteractsWithQueue, SerializesModels;

    /**
     * Create a new job instance.
     *
     * @return void
     */
    public function __construct()
    {
        //
    }

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        foreach (range(1, 10) as $item)
            \Log::info("TestTwo: item #" . $item);
    }
}

I simply log some messages in laravel's log file, and since TestOne is sleeping for 5 seconds, TestTwo should log the messages first

HomeController.php

<?php

namespace App\Http\Controllers;

use Queue;
use App\Jobs\TestOne;
use App\Jobs\TestTwo;

class HomeController extends Controller
{
    public function index()
    {
        $this->dispatch(new TestOne());
        $this->dispatch(new TestTwo());
        die("stop");
    }
}

However TestTwo job still waits until TestOne job is done:

[2017-03-04 17:00:30] local.INFO: TestOne: item #1  
[2017-03-04 17:00:30] local.INFO: TestOne: item #2  
[2017-03-04 17:00:30] local.INFO: TestOne: item #3  
[2017-03-04 17:00:30] local.INFO: TestOne: item #4  
[2017-03-04 17:00:30] local.INFO: TestOne: item #5  
[2017-03-04 17:00:30] local.INFO: TestOne: item #6  
[2017-03-04 17:00:30] local.INFO: TestOne: item #7  
[2017-03-04 17:00:30] local.INFO: TestOne: item #8  
[2017-03-04 17:00:30] local.INFO: TestOne: item #9  
[2017-03-04 17:00:30] local.INFO: TestOne: item #10  
[2017-03-04 17:00:30] local.INFO: TestTwo: item #1  
[2017-03-04 17:00:30] local.INFO: TestTwo: item #2  
[2017-03-04 17:00:30] local.INFO: TestTwo: item #3  
[2017-03-04 17:00:30] local.INFO: TestTwo: item #4  
[2017-03-04 17:00:30] local.INFO: TestTwo: item #5  
[2017-03-04 17:00:30] local.INFO: TestTwo: item #6  
[2017-03-04 17:00:30] local.INFO: TestTwo: item #7  
[2017-03-04 17:00:30] local.INFO: TestTwo: item #8  
[2017-03-04 17:00:30] local.INFO: TestTwo: item #9  
[2017-03-04 17:00:30] local.INFO: TestTwo: item #10 

I am running the jobs with php artisan queue:listen

What am I doing wrong here? I really need these tasks to run asynchronously, just like, say, a JS AJAX request would work like.

I am using Laravel 5.2. Again, I am using "database" as the queue driver and yes I have migrated the jobs table. Is it not possible using the database as the driver?

Solution 1

To process jobs in parallel you'd have to split them across different queues as @dparoli pointed out.

This way you can not only categorize them but also priorize how they'll be processed by your queue workers.

When dispatching a job, you'll specify which queue it belongs:

$this->dispatch((new TestOne())->onQueue('queue1'));
$this->dispatch((new TestTwo())->onQueue('queue2'));

That way you can spawn multiple queue workers to process jobs separately:

php artisan queue:work --queue=queue1
php artisan queue:work --queue=queue2

You can also use a single queue worker which priorize how queues are processed, so you can give a higher or lower precedence for some jobs over others:

php artisan queue:work --queue=queue2,queue1

By using a process monitor like Supervisor you can even spawn a single worker in multiple processes as detailed in the documentation.

It's worth noting that a single queue worker which prioritizes its queues will still process their jobs by taking the FIFO precedence in addition to the given queue priority. To achieve a better parallelism you'll want to spawn multiple queue workers.

This holds true for all queue drivers.

Solution 2

Asynchronous mean that the jobs won't hold the execution of the controller method. For instance, if you add sleep(5); to One and sleep(10); to Two, die('stop'); will still happens instantaneously when you request your controller. In a Synchronous execution, it would take 15 seconds for die to be reached.

Queue kind of holds of the notion of FIFO (first in, first out). When you go to the Supermarket, it doesn't matter if you have a lot more items (too much processing time) than the 2nd person, if you're the first in line, the 2nd will have to wait you to finish. That's how Queue works.

For you to achieve what you want, I suggest a simple test exercise.

• Stop the queue:listen
• Call the Controller (causing both jobs to be queued);
• Call php artisan queue:work & from the terminal
• Press up arrow and issue the command again real fast.

Since & will send the process to the background, you'll be free to issue queue:work twice almost instantaneous. This should bring the behavior you expect.

This was my output

[03:01 PM]-[root@php7]-[/var/www/html/jobs]
php artisan queue:work &
[1] 2456

[03:02 PM]-[root@php7]-[/var/www/html/jobs]
php artisan queue:work &
[2] 2458

[03:02 PM]-[root@php7]-[/var/www/html/jobs]
[2017-03-04 18:02:33] Processed: App\Jobs\TaskTwo
[2017-03-04 18:02:37] Processed: App\Jobs\TaskOne

The point I'm trying to make is:

• Controller will not have to wait the jobs to finish (this is what Asynchronous means)
• queue:listen will run one job at a time and will only start the next after the 1st finished;
• queue:work will start the 1st job in line and mark it as reserved (column reserved_at) so the next queue:work can take the next job not reserved.

Solution 3

Push jobs on different queues, i.e. queue1, queue2 etc.

For each queue you have defined you should have a worker:

php artisan queue:work --queue=queue1
php artisan queue:work --queue=queue2

You can use supervisord to monitor queue workers as per documentation.

With this solution each queue run async respect others queues, but two jobs on the same queue are not async, they respect FIFO precedence.

Comments

[mike.void]

Thanks for the answer. Ok, so if Asynchronous is wrong term in this context, what would you call it then? How would I use queue:work in a real life application? I also use queues to send emails as my application depends heavily on sending emails.

[Marco Aurélio Deleu]

Your controller is asynch, your queue is not. Why do you need asynch queue? You probably don't.