FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory when processing large files with fs

Question

I have a nodeJs script that process a bunch of large .csv files (1.3GB for all). It run for a moment and throw this error:

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

I have tried to put #!/usr/bin/env node --max-old-space-size=4096 at the beginning of my file but this didn't solve my problem...

Tried to google it but nearly no pertinent info about my issue..

Can I flush memory once I do not need file content ? Do I need to allocate more memory to nodeJs ?

Thanks ;)

Here is my code sample:

fs.readdir(dirName, function(err, filenames) {
    if (err) console.error(err);
    else {
        filenames.forEach(function(filename) {
            fs.readFile(dirName + filename, 'utf-8', function(err, content) {
                if (err) console.error(err);
                else processFile(content);
            });
        });
    }
});

Solution 1

I have finally found the solution to the problem ! I needed to launch the process adding the --max-old-space-size=8192 param to node process like so:

node --max-old-space-size=8192 ./myScript.js

I processed my 1.3GB without a problem !!

Solution 2

You can increase the amount of memory allocated to the command by running the following command prior to running Snyk:

Linux/macOS

export NODE_OPTIONS=--max-old-space-size=8192 For example:

export NODE_OPTIONS=--max-old-space-size=8192 snyk test export NODE_OPTIONS=--max-old-space-size=8192 snyk monitor

Windows

From the control panel go to System -> Advanced system settings -> Environment Variables -> New (user or system)