Processing large amounts of data in PHP without a browser timeout

14,080

Solution 1

I would write two scripts:

File index.php:

<iframe src="job.php" frameborder="0" scrolling="no" width="1" height="1"></iframe>
<script type="text/javascript">
    function progress(percent){
        document.getElementById('done').innerHTML=percent+'%';
    }
</script><div id="done">0%</div>

File job.php:

set_time_limit(0);                   // ignore php timeout
ignore_user_abort(true);             // keep on going even if user pulls the plug*
while(ob_get_level())ob_end_clean(); // remove output buffers
ob_implicit_flush(true);             // output stuff directly
// * This absolutely depends on whether you want the user to stop the process
//   or not. For example: You might create a stop button in index.php like so:
//     <a href="javascript:window.frames[0].location='';">Stop!</a>
//     <a href="javascript:window.frames[0].location='job.php';">Start</a>
// But of course, you will need that line of code commented out for this feature to work.

function progress($percent){
    echo '<script type="text/javascript">parent.progress('.$percent.');</script>';
}

$total=count($mobiles);
echo '<!DOCTYPE html><html><head></head><body>'; // webkit hotfix
foreach($mobiles as $i=>$mobile){
    // send sms
    progress($i/$total*100);
}
progress(100);
echo '</body></html>'; // webkit hotfix

Solution 2

I'm assuming these numbers are in a database, if so you should add a new column titled isSent (or whatever you fancy).

This next paragraph you typed should be queued and possibly done night/weekly/whenever appropriate. Unless you have a specific reason too, it shouldn't be done in bulk on demand. You can even add a column to the db to see when it was last checked so that if a number hasn't been checked in at least X days then you can perform a check on that number on demand.

Processing of the data involves checking mobile number type (e.g CDMA), assigning unique ids to all the numbers for further referencing, check for network/country unique charges, etc.

But that still leads you back to the same question of how to do this for 50,000 numbers at once. Since you mentioned cron jobs, I'm assuming you have SSH access to your server which means you don't need a browser. These cron jobs can be executed via the command line as such:

/usr/bin/php /home/username/example.com/myscript.php

My recommendation is to process 1,000 numbers at a time every 10 minutes via cron and to time how long this takes, then save it to a DB. Since you're using a cron job, it doesn't seem like these are time-sensitive SMS messages so they can be spread out. Once you know how long it took for this script to run 50 times (50*1000 = 50k) then you can update your cron job to run more/less frequently.

$time_start = microtime(true);
set_time_limit(0);

function doSendSMS($phoneNum, $msg, $blah);

$time_end = microtime(true);
$time = $time_end - $time_start;
saveTimeRequiredToSendMessagesInDB($time);

Also, you might have noticed a set_time_limit(0), this will tell PHP to not timeout after the default 30seconds. If you are able to modify the PHP.ini file then you don't need to enter this line of code. Even if you are able to edit the PHP.ini file, I would still recommend not changing this feature since you might want other pages to time out.

http://php.net/manual/en/function.set-time-limit.php

Solution 3

If this isn't a one-off type of situation, consider engineering a better solution.

What you basically want is a queue that your browser-bound process can write to, and than 1-N worker processes can read from and update.

Putting work in the queue should be rather inexpensive - perhaps a bunch of simple INSERT statements to a SQL RDBMS.

Then you can have a daemon or two (or 100, distributed across multiple servers) that read from the queue and process stuff. You'll want to be careful here and avoid two workers taking on the same task, but that's not hard to code around.

So your browser-bound workflow is: click some button that causes a bunch of stuff to get added to the queue, then redirect to some "queue status" interface, where the user can watch the system chew through all their work.

A system like this is nice, because it's easy to scale horizontally quite a ways.

EDIT: Christian Sciberras' answer is going in this direction, except the browser ends up driving both sides (it adds to the queue, then drives the worker process)

14,080

elf1984

Web developer..

Updated on April 04, 2020

Comments

elf1984 about 4 years

I have array of mobile numbers, around 50,000. I'm trying to process and send bulk SMS to these numbers using third-party API, but the browser will freeze for some minutes. I'm looking for a better option.

Processing of the data involves checking mobile number type (e.g CDMA), assigning unique ids to all the numbers for further referencing, check for network/country unique charges, etc.

I thought of queuing the data in the database and using cron to send about 5k by batch every minute, but that will take time if there are many messages. What are my other options?

I'm using Codeigniter 2 on XAMPP server.
elf1984 about 13 years

Wonderful answer, thank you so much. // keep on going even if user pulls the plug (Very funny)
elf1984 about 13 years

@Christian The solution works better on FireFox & IE, but google Chrome freezed, any other way out?
Christian about 13 years

Safari Webkit (also use din Chrome) has some issues with the PHP progress function. I found a fix though, I'll get back to you with it... Edit: Fixed.
Ross about 13 years

@Christian Sciberras could you please explain the "hotfix"? Does Chrome just require a valid doctype and layout? Great answer though, will be bookmarking this !
elf1984 about 13 years

@Christian it works like bad, i will use css and make the progress bar look clean. thank you so much
Christian about 13 years

@elf1984 - Well yeah, I can't see this working without a proper progress bar :). I've made an installer using this - you can pass a message with progress() which describes what it is currently doing.