How to loop a Magento collection?

Question

Basically, I need to get a CSV file of my customers, generated automatically in a script every day. I've tried several ways, but they are too slow or actually exhausted of memory.

*1) foreach through collection resource *

$collection = Mage::getResourceModel('customer/customer_collection')
->addAttributeToSelect('email')
->addAttributeToSelect('created_at')
->joinAttribute('billing_company', 'customer_address/company', 'default_billing', null, 'left')
->joinAttribute('billing_street', 'customer_address/street', 'default_billing', null, 'left')
->joinAttribute('billing_postcode', 'customer_address/postcode', 'default_billing', null, 'left')
->joinAttribute('billing_telephone', 'customer_address/telephone', 'default_billing', null, 'left')
->joinAttribute('billing_city', 'customer_address/city', 'default_billing', null, 'left')
->joinAttribute('billing_region', 'customer_address/region', 'default_billing', null, 'left')
->joinAttribute('billing_country_id', 'customer_address/country_id', 'default_billing', null, 'left');

foreach($collection as $customer) {
echo $customer->getFirstname() . ",";
}

2) foreach and load customer

$collection = Mage::getResourceModel('customer/customer_collection');
foreach($collection as $customer) {
  $fullcustomer = Mage::getModel("customer/customer")->load($customer->getId());
  echo $fullcustomer->getFirstname() . ",";
}

Any ideas?

Thanks!

Solution 1

Try paging a large collection!

The idea is if you can load the collection in smaller chunks, you won't use as much memory. Load a chunk(page), then do something with it like saving it out to a text file and then load the next chunk. The result being, you've worked with the whole larger collection, but only incurred the memory cost of the largest page.

We use something similar to this to export orders from our store. I plugged in your collection and it seems to work.

<?php

 if(php_sapi_name()!=="cli"){
 echo "Must be run from the command line.";
 };

/**
 * Setup a magento instance so we can run this export from the command line.
 */

require_once('app/Mage.php');
umask(0);

if (!Mage::isInstalled()) {
    echo "Application is not installed yet, please complete install wizard first.";
    exit;
}

// Only for urls // Don't remove this
$_SERVER['SCRIPT_NAME'] = str_replace(basename(__FILE__), 'index.php', $_SERVER['SCRIPT_NAME']);
$_SERVER['SCRIPT_FILENAME'] = str_replace(basename(__FILE__), 'index.php', $_SERVER['SCRIPT_FILENAME']);

Mage::app('admin')->setUseSessionInUrl(false);
Mage::setIsDeveloperMode(true); ini_set('display_errors', 1); error_reporting(E_ALL);

try {
    Mage::getConfig()->init();
    Mage::app();   
} catch (Exception $e) {
    Mage::printException($e);
}
ini_set('memory_limit','500M');

$customerCount = 0;
try{
    //configure the collection filters.
    $collection = Mage::getResourceModel('customer/customer_collection')
    ->addAttributeToSelect('email')
    ->addAttributeToSelect('created_at')
    ->joinAttribute('billing_company', 'customer_address/company', 'default_billing', null, 'left')
    ->joinAttribute('billing_street', 'customer_address/street', 'default_billing', null, 'left')
    ->joinAttribute('billing_postcode', 'customer_address/postcode', 'default_billing', null, 'left')
    ->joinAttribute('billing_telephone', 'customer_address/telephone', 'default_billing', null, 'left')
    ->joinAttribute('billing_city', 'customer_address/city', 'default_billing', null, 'left')
    ->joinAttribute('billing_region', 'customer_address/region', 'default_billing', null, 'left')
    ->joinAttribute('billing_country_id', 'customer_address/country_id', 'default_billing', null, 'left');

    //Add a page size to the result set.
    $collection->setPageSize(100);
    //discover how many page the result will be.
    $pages = $collection->getLastPageNumber();
    $currentPage = 1;
    //This is the file to append the output to.
    $fp = fopen('/tmp/customers.csv', 'w');
    do{
         //Tell the collection which page to load.
         $collection->setCurPage($currentPage);
         $collection->load();
         foreach ($collection as $customer){
            //write the collection array as a CSV.
            $customerArray = $customer->toArray();
            //var_dump($customerArray); echo "\n\n";
            fputcsv($fp, $customerArray);
            $customerCount++;
         }
         $currentPage++;
         //make the collection unload the data in memory so it will pick up the next page when load() is called.
         $collection->clear();
    } while ($currentPage <= $pages);
    fclose($fp);
} catch (Exception $e) {
    //$response['error'] = $e->getMessage();
    Mage::printException($e);
}
echo "Saved $customerCount customers to csv file \n";
?>

I saved it as exportCustomers.php.

Then setup your cron task to run: php -f /path/to/your/magento/exportCustomers.php

Solution 2

Using a load() on each Customer object will significantly slow down your code. I would suggest that your first approach of adding the required attributes to the collection is the right one, but finish it with something like:

$collection->toArray([$arrRequiredFields = array()]);

That way, the individual customers are never loaded, but toArray() (doco here) will give you the fields you want, then you can iterate over the multi-dimensional array to produce your comma-separated string.

Alternatively, you could try $collection->toXml() and work with XML if you're comfortable with that. The Google will point you to xml->csv conversion methods.

Comments

[Corgalore]

Paging the collection is definitely the way to go. It's pretty fast and more memory efficient.

[Corgalore]

I did notice when using this, however, that each page takes longer to process than the previous. It's as if the collection->clear() is not truly freeing up memory or something else is going on.