Laravel queue rate limiting or throttling

Question

I am working on an app that requires fetching data from a third-party server and that server allows max 1 request per seconds.

Now, all request send as job and I am trying to implement Laravel "Rate Limiting" to release 1 job per second but unable to figure out why it should be implemented and there is no real-life example in the web.

Did anyone implement it?

Any hint of this?

Solution

I'm the author of mxl/laravel-queue-rate-limit Composer package.

It allows you to rate limit jobs on specific queue without using Redis.

1. Install it with:

   $ composer require mxl/laravel-queue-rate-limit:^1.0
   

2. This package is compatible with Laravel 5.5+ and uses auto-discovery feature to add MichaelLedin\LaravelQueueRateLimit\QueueServiceProvider::class to providers.

3. Add rate limit settings to config/queue.php:

   'rateLimit' => [
       'mail' => [
           'allows' => 1,
           'every' => 5
       ]
   ]
   

   These settings allow to run 1 job every 5 seconds on mail queue. Make sure that default queue driver (default property in config/queue.php) is set to any value except sync.

4. Run queue worker with --queue mail option:

   $ php artisan queue:work --queue mail
   

   You can run worker on multiple queues, but only queues referenced in rateLimit setting will be rate limited:

   $ php artisan qeueu:work --queue mail,default
   

   Jobs on default queue will be executed without rate limiting.

5. Queue some jobs to test rate limiting:

   SomeJob::dispatch()->onQueue('mail');
   SomeJob::dispatch()->onQueue('mail');
   SomeJob::dispatch()->onQueue('mail');
   SomeJob::dispatch();
   

Comments

[Pathros]

How would you limit the number of emails sent (jobs) per day?

[namal]

->allow(30)->everyMinutes(60*24)

[Pathros]

Nice! And what about the pending jobs? How to dispatch them in the next day (24 hours after) and so on?

[namal]

You can use supervisor to keep running queue:worker command. then the rest jobs will be dispatched automatically accordingly.

[namal]

laravel.com/docs/9.x/queues#supervisor-configuration