How to free memory after array in PHP? unset and = null seems to not working

Question

I ve got a 16 MB size CSV file and try to parse it, and do some stuff, but script run out of memory after some time. I realize that this code generate around 200 MB of used space and unset is not working.

    $countRows = 1;
    var_dump("3. ".memory_get_usage()." beginDiff: ".(memory_get_usage() - $this->startingMemory));
    while(($row = fgetcsv($fp, 300000, ';', '"')) !== FALSE)
    {
        if ($row == '')
            continue;

        if($firstRow == true)
        {
            foreach($row as $k => $v)
            {
                $this->columnMapping[$k] = trim(mb_strtolower($v));
            }
            $firstRow = false;
            continue;
        }else
        {
            foreach($row as $k => $v)
            {
                $row[$this->columnMapping[$k]] = $v;
                unset($row[$k]);
            }
        }
    ...
        //$this->theirCategoriesToProducts[$row['kategorie']][]['kodproduktu'] = $row['kodproduktu'];
        $this->theirCategoriesToProducts[$row['kategorie']][] = $row;
    }
    var_dump("3,5.  ".memory_get_usage()." beginDiff: ".(memory_get_usage() - $this->startingMemory));
    ...
    var_dump("7. - before unset total: ".memory_get_usage()." beginDiff: ".(memory_get_usage() - $this->startingMemory));
    unset($this->theirCategoriesToProducts);
    var_dump("8. - after unset total: ".memory_get_usage()." beginDiff: ".(memory_get_usage() - $this->startingMemory));die;

Generating this output:

    string '3. 72417440 beginDiff: 34730040' (length=31)
    string '3,5.  292748528 beginDiff: 255061136' (length=36)
    string '7. - before unset total: 299039360 beginDiff: 261351984' (length=55)
    string '8. - after unset total: 297364432 beginDiff: 259677056' (length=54)

With setting that variable equals null is the output very similar. But switching comments between this two lines

    $this->theirCategoriesToProducts[$row['kategorie']][]['kodproduktu'] = $row['kodproduktu'];
    //$this->theirCategoriesToProducts[$row['kategorie']][] = $row;

Will output:

    string '3. 72417784 beginDiff: 34730040' (length=31)
    string '3,5.  81081984 beginDiff: 43394248' (length=34)
    string '7. - before unset total: 87256544 beginDiff: 49568824' (length=53)
    string '8. - after unset total: 85581520 beginDiff: 47893800' (length=52)

So its about 200 MB of "lost" memory (almost half of dedicated).

Recursive function unseting all parts of arrays eat more memory, than was able to free so crashed also.

In script is never used that array with & so there should be no references to other variables.

File is closing right after 3.5 dump.

Any other ideas, how to unset that array?

Solution 1

As of PHP > 5.3 there are some Garbage Collection mechanisms available, so theoretically you could think of something like the example in the docs

//Memory cleanup for long-running scripts.
gc_enable(); // Enable Garbage Collector
var_dump(gc_enabled()); // true
var_dump(gc_collect_cycles()); // # of elements cleaned up
gc_disable(); // Disable Garbage Collector

But unfortunately, in your case you have to bear in mind that (according to Can I trigger PHP garbage collection to happen automatically if I have circular references?) the garbage collector "will not run, for example, when the memory limit is about to hit. As a result, your script can still abort when hitting memory limit only because PHP is too dumb to collect the cycles in that case!".

In the end, you can try using GC, but it possibly won't solve your problem.

So, what else is there to try? Try splitting your master data array that you import into smaller chunks and import them sequentially one after another. Fetch the chunks in a loop always into the same variable and then loop through it to process the records.

Solution 2

You can use unset to delete variables allowing them to be garbage collected.

$foo = "bar";
unset($foo);
var_dump($foo); // null

Overall just keep track of what you're keeping references to. Maybe you don't need to keep track of everything. A while-loop allows you to be memory efficient, just carrying over what you need for each line.

Some scripts actually just need a lot of memory to run, it's not too crazy to increase the memory limit, but only do this if it's actually needed.

Solution 3

The PHP function fgetcsv is bad because the server need to store the complete file in the memory better read one line store it

And php arrays use alot of memory because array's in php arrays are implemented as "hashmaps" or a "hashtable" you could use splFixedArray (real C or C++ array) if you don't need string's as key

splFixedArray (you need atleast php 5.3 to use it) are known to use 40% off the total what php array needed.