What's the best approach to sending email to hundreds of recipients from a Zend Framework application?

Question

I'm trying to implement a mailing list system for my application. I'm currently using Zend_Mail_Transport_Smtp('localhost') as my transport, looping through my list of subscribers, and sending a new Zend_Mail to each one. However, I am noticing that the length of time that it takes for the script to complete increases as the number of subscribers increase.

I'm sure there must be a more professional approach to doing this, involving the queuing of emails. I suppose the ideal approach would be for the user to fill out the form, click send, and immediately get a response saying that the emails are being sent, rather than waiting for the hundreds of emails to finish sending.

I understand that Zend_Mail does not do any sort mail queuing. Could anyone who has experience with this, give me an overview of how this can be done? I don't know anything about cron/crontab/cronjobs, so if it involves that, please explain the process.

Solution 1

In order to reliably send a large number of emails using PHP you have to use a queueing mechanism. As suggested by others, the process of using a queue looks something like this:

• Loop over your set of users, creating an email for each one and possibly customizing the content
• Pass each mail object to the queue which will delay the sending of the email until later
• In some sort of cron script, send out the contents of the queue a few hundred at a time. Note: you'll want to tweak the number of emails you are sending by watching the logs for errors coming back from the actual sending process. If you try to send too many, I've noticed it reaches a point where the mail transport will no longer accept connections (I'm using qmail)

There are a few libraries out there you can use to do this, PEAR Mail Queue (with Mail_Mime) and SwiftMailer both allow you to create and queue emails. So far, Zend Mail only provides for creation of emails, not queueing (more on that later).

I have experience primarily with PEAR Mail Queue and there are a few gotchas. If you are trying to queue up a large number of emails (for instance, looping over 20,000 users and trying to get them into the queue in a reasonable time), using Mail Mime's quoted-printable encoding implementation is very slow. You can speed this up by switching to base64 encoding.

As for Zend Mail, you can write a Zend Mail Transport object that puts your Zend Mail objects into the PEAR Mail Queue. I have done this with some success, but it takes a bit of playing to get it right. To do this, extend Zend Mail Transport Abstract, implement the _sendMail method (which is where you will drop your Zend Mail object into the Mail Queue) and pass the instance of your transport object to the send() method of your Zend Mail object or by Zend Mail::setDefaultTransport().

Bottom line is that there are many ways you can do this, but it will take some research and learning on your behalf. It is a very solvable problem, however.

Solution 2

NOTE: when I first read your question, I thought it said hundreds of thousand emails at once. When I double checked, I noticed it actually said hundreds to thousands. I'm too lazy to change my post now, so here are some caveats: From my experience, you can probably run fine without a commercial tool to about 40K. At about 10K you'll want to be following the 'minimum' list to prevent major pain when you start reaching larger list sizes. I do recommend implementing it all right away though.

I've said this before, there are two sides to sending email:

1. The technical side -- basically all of the RFC's around the smtp protocol, email formats, DNS records, etc. This is mildly complicated but solvable.
2. The magical side -- email delivery management is voodoo. You will get frustrated, things will break for no apparent reason and you will consider leaving for another job that doesn't involve email.

I recommend not writing your own bulk sender. I'm sure PHP can do a fine job, but you should probably spend your time elsewhere. The two products I've used in the past and recommend are Strongmail and PowerMTA. Be warned -- they have a high price tag, but I can almost guarantee that you'll spend more building your own solution in the long run.

One area that you'll get nailed with writing your own in PHP is throttling/tar pitting. Mail servers will start adding in sleep(30)'s after you've sent a few messages to slow you down and stop you from spamming.

Typically, these commercial bulk senders run the SMTP protocol for queueing. You would continue to use Zend_Mail, but hard code it to connect to your server. It'll queue the mail just about as fast as you can send it, then use it's own engine to send the mail to their destinations.

At a 100K list, you will have to employ email best practices. At a minimum, you'll need:

• SPF Records, possibly DKIM as well
• Multiple IPs to segment traffic over -- have 3 IP's, one for quality address you trust, one for medium risk IP addresses and one for high risk IP addresses. This design helps minimize the risk to getting mail to your best customers.
• Proper reverse DNS for sending IP addresses
• Use the feedback loops from AOL, hotmail, yahoo and others for processing spam complaints
• Unsubscribe and bounce management -- make sure you're pruning these addresses
• Having open/click tracking is also important -- if you're customers on the A list aren't opening your emails, you need to degrade them to the B list and so forth. This is important because ISP's will turn inactive accounts into a honeypot. Hotmail is famous for this.

Finally, if you're really serious about sending email, you'll want some other tools like Return Path.

Solution 3

From the PHP.net Documentation.

Note: It is worth noting that the mail() function is not suitable for larger volumes of email in a loop. This function opens and closes an SMTP socket for each email, which is not very efficient.
For the sending of large amounts of email, see the » PEAR::Mail, and » PEAR::Mail_Queue packages.

The Zend Mail class is probably pretty good (most of Zend's stuff is good) But if you want other options. Here they are.

Solution 4

Use Zend_Queue to place the emails in the queue for asychronous background processing. You will need a cron job to processes the queue in the background.

protected function _enqueueEmail(WikiEmailArticle $email)
{
    static $intialized = false; 

    if (!$initialized) {

        $this->_initializeMailQueue("wikiappwork_queue");
        $initialized = true;
    }

    $this->_mailQueue->send(serialize($email));  
}
protected function _initializeMailQueue()
{
    /* See: 1.) http://framework.zend.com/manual/en/zend.queue.adapters.html and
     *      2.) Zend/Queue/Adapter/Db/mysql.sql. 
     */

 $ini = Zend_Controller_Front::getInstance()->getParam('bootstrap')
                                            ->getOptions(); 

     $queueAdapterOptions =    array( 'driverOptions' => array(
    'host' => $ini['resources']['multidb']['zqueue']['host'],
    'username' => $ini['resources']['multidb']['zqueue']['username'],
    'password' => $ini['resources']['multidb']['zqueue']['password'],
    'dbname' => $ini['resources']['multidb']['zqueue']['dbname'],
    'type' => $ini['resources']['multidb']['zqueue']['adapter'] ),
    'name' => $ini['resources']['multidb']['zqueue']['queueName'] );

    $this->_mailQueue = new Zend_Queue('Db', $queueAdapterOptions);

 }

Then for the cron job, a script like

<?php
use \Wiki\Email\WikiEmailArticle;

// Change this define to correspond to the location of the wikiapp.work/libary
define('APPLICATION_PATH', '/home/kurt/public_html/wikiapp.work/application');

set_include_path(implode(PATH_SEPARATOR, array(
     APPLICATION_PATH . '/../library',
     get_include_path(),
 )));

// autoloader (uses closure) for loading both WikiXXX classes and Zend_ classes.
spl_autoload_register(function ($className) { 

  // Zend classes need underscore converted to PATH_SEPARATOR
  if (strpos($className, 'Zend_' ) === 0) {

        $className = str_replace('_', '/', $className );   
  }

  $file = str_replace('\\', '/', $className . '.php');

  // search include path for the file.
  $include_dirs = explode(PATH_SEPARATOR, get_include_path());

  foreach($include_dirs as $dir) {

    $full_file = $dir . '/'. $file;

    if (file_exists($full_file)) { 

        require_once $full_file; 
        return true; 
    }
  }

  return false; 
 }); 

// Load and parese ini file, grabing sections we need.
$ini = new Zend_Config_Ini(APPLICATION_PATH . 
                          '/configs/application.ini', 'production');

$queue_config = $ini->resources->multidb->zqueue;

$smtp_config = $ini->email->smtp;

$queueAdapterOptions =  array( 'driverOptions' => array(
                                        'host'      => $queue_config->host,
                    'username'  => $queue_config->username,
                    'password'  => $queue_config->password,
                    'dbname'    => $queue_config->dbname,
                    'type'      => $queue_config->adapter),
                'name' => $queue_config->queuename);

$queue = new Zend_Queue('Db', $queueAdapterOptions);


$smtp = new Zend_Mail_Transport_Smtp($smtp_config->server, array(
                'auth'      => $smtp_config->auth,
        'username'  => $smtp_config->username,
        'password'  => $smtp_config->password,
        'port'      => $smtp_config->port,
        'ssl'       => $smtp_config->ssl
        ));

Zend_Mail::setDefaultTransport($smtp);

$messages = $queue->receive(10); 

foreach($messages as $message) {

        // new WikiEmailArticle.     
    $email = unserialize($message->body);

        try {

            $email->send();

        }  catch(Zend_Mail_Exception $e) {

               // Log the error?
               $msg = $e->getMessage();
               $str = $e->__toString();
               $trace =  preg_replace('/(\d\d?\.)/', '\1\r', $str);
        } // end try

$queue->deleteMessage($message);

} // end foreach

Solution 5

You should be fine using PHP up into the thousands of recipients, although avoid the mail() as others have noted. I've seen a few systems designed for large amounts of mail (100,000+ recipients) kick over to bypassing the standard mailing functions and trying to work more directly with the MTA. Even then it's not been clear to me that was required.

Making email professional is more about making sure the formatting is good (HTML and plain text whenever possible), people can unsubscribe easily, bounces are handled correctly, the mail server has all the right DNS records are in place, and the server configuration doesn't violate rules of any major blacklist system. The language you write the application in isn't a major factor at a few hundred or even a few thousand messages.

View more solutions

Comments

[rick]

PEAR::Mail is slow in my experience. PHPMailer and swiftmailer are excellent.

[rick]

This is a confusing answer since if not using cron, script timeouts when sending emails will be an issue before traffic is of concern.

[rick]

You know your stuff. Unfortunately, your answer is overwhelming. How would one use Strongmail or PowerMTA with Zend Mail and cover all the bullet points you listed?

[grossvogel]

I think Phil's suggesting that you use cron to throttle the sending of the emails. For example, only send 100 at a time, every 30 minutes, until the list is exhausted.

[rick]

But it seems he's suggesting crontab should be used as a solution to high traffic? Anyway, we should all be so lucky as to generate too much traffic with a marketing campaign. Chances are, that isn't an issue.

[Alister Bulman]

A couple of useful tips for speeding up the queuing with PEAR_Mail is make the *_seq a MEMORY table in Mysql. I also wrapped it in a transaction per 100 entries

[vaske]

You can tie you application with Strongmail, when I say application I mean on data storage, this means that you don't need to implement any line of code, you just share your user details with Strongmail server and send the millions of emails :)