How to increase Navigation Timeout when running Puppeteer tests with Jest

Question

I have a small suite of tests based on Puppeteer with Jest, and I can't get rid of the following problem: when I am running a single specific test (ex: yarn test myscenario.test.js) everything works just fine; the problem occurs when I am running the entire test suite (about 20 tests) using yarn test command, some of my tests are failing with following error:

Navigation Timeout Exceeded: 30000ms exceeded at Promise.then (node_modules/puppeteer/lib/NavigatorWatcher.js:73:21)

The thing is that all my tests already have a specific timeout already set (99999 ms !!) and a single test is executed in about 6-7 seconds. My guess is that when the entire test suite is running, there is a global navigation timeout which exceeds the limits of 30000ms.

Is there any way to override this global Navigation Timeout limit?

Here is how one of my tests (myscenario.test.js) looks like (which iterates through a JSON file and navigates to various URLs where executes some simple actions):

const puppeteer = require('puppeteer')
const propertiesReader = require('properties-reader')
const jsonReader = require("fs")
const prop = propertiesReader('./resources/config.ini')
const tealiumTags = JSON.parse(jsonReader.readFileSync('./resources/tealium-tags.json', 'utf8'))

let browser
let page

beforeEach (async () => {
    browser = await puppeteer.launch({headless: true, args: ['--start-maximized']});
    page = await browser.newPage();
    await page.setViewport({ width: 1920, height: 1080 })
  })

afterEach (() => {
    browser.close()
  })

describe('Decline all Tealium tags', () => {
 
    for (let brand in tealiumTags) {
      if (tealiumTags.hasOwnProperty(brand)) {
        
        let brandUrl = tealiumTags[brand].url

        test('Decline all Tealium tags for ' + brandUrl, async () => {
          
          console.log("---------------------------------------")
          console.log("Decline all Tealium tags for " + brandUrl)
          await page.goto("https://www." + brandUrl, { waitUntil: 'domcontentloaded' })
          await page.waitForSelector(prop.get('DECLINE_COOKIES_BUTTON'))
          await page.click(prop.get('DECLINE_COOKIES_BUTTON'))    
          await page.waitForNavigation({waitUntil: 'domcontentloaded'})

          let utag = await page.evaluate(() => window.utag["send"])

          expect(Object.keys(utag).length).toEqual(0)

        }, 99999)
      }
    }
})

Any hint on this matter would be much appreciated, thanks!

Solution 1

To address the actual question, "overriding the navigation timeout". You can pass the timeout option to the waitForNavigation method.

page.waitForNavigation( { timeout: 60, waitUntil: 'domcontentloaded' });

You can also disable the timeout by passing the value 0.

However, I think you have a bigger problem. It seems:

1. Running all tests files at the same time is causing multiple Chromium sessions causing long load times.
2. You page variable is being shared by tests trying to run at the same time.

You've defined 'page' at the top of the scope, therefore, every test case is going to share the instance of the page variable.

let browser;

beforeAll(async () => {
  browser = await puppeteer.launch({ headless: true, args: ['--start-maximized'] });
});

afterAll(() => {
  browser.close();
});

describe('Decline all Tealium tags', () => {
  for (const brand in tealiumTags) {
    if (tealiumTags.hasOwnProperty(brand)) {
      const brandUrl = tealiumTags[brand].url;

      test(`Decline all Tealium tags for ${brandUrl}`, async () => {
        // Setup the page in each test case allows you to run these concurrently.
        const page = await browser.newPage();
        await page.setViewport({ width: 1920, height: 1080 });

        console.log('---------------------------------------');
        console.log(`Decline all Tealium tags for ${brandUrl}`);
        await page.goto(`https://www.${brandUrl}`, { waitUntil: 'domcontentloaded' });
        await page.waitForSelector(prop.get('DECLINE_COOKIES_BUTTON'));
        await page.click(prop.get('DECLINE_COOKIES_BUTTON'));
        await page.waitForNavigation({ waitUntil: 'domcontentloaded' });

        const utag = await page.evaluate(() => window.utag.send);

        expect(Object.keys(utag)).toHaveLength(0);
      }, 99999);
    }
  }
});

You might need to run out of headless mode to isolate the problem.

Solution 2

Faced similar issue when running tests in parallel in docker, launching browser with below args greatly reduced navigation time and the tests are running fine now.

await puppeteer.launch({args: [
    '--disable-gpu',
    '--disable-dev-shm-usage',
    '--disable-setuid-sandbox',
    '--no-first-run',
    '--no-sandbox',
    '--no-zygote',
    '--single-process', // <- this one doesn't works in Windows
]})

More reference on issue here

Solution 3

The default in puppeteer timeout is 30 seconds. To use custom timeouts, you can use the setDefaultNavigationTimeout and setDefaultTimeout methods or the timeout property in the options parameter. The wait time in all cases is specified in milliseconds.

await page.setDefaultNavigationTimeout(60000);

e.g.

const page = await browser.newPage();            
await page.setDefaultNavigationTimeout(60000); //timeout 60 seconds now

Pass 0 to disable the timeout

await page.setDefaultNavigationTimeout(0);   

Github Official docs: https://github.com/puppeteer/puppeteer/blob/v10.0.0/docs/api.md#pagesetdefaultnavigationtimeouttimeout

Comments

[boldnik]

Did you try to use recursion with several attempts when you get TimeoutError ?

[Lucian Radu]

I've made the adjustments as you suggested but unfortunately the issue still persists. One thing is now clear, the problem occurs due to parallel execution of the tests. Just tried to run the tests sequentially using npm test -- --runInBand and the suite is executed properly with no more timeout errors. I will keep in mind your suggestion and will try to rethink the test structure a bit. I will let you guys know if any positive results :) Thanks for the help!

[Lucian Radu]

Each test is opening a new browser instance with a fresh tab, but yes, seems that the problem occurs due to parallel execution. Suite is executed correctly when using npm test -- --runInBand, no more timeout problems. I will try to investigate and understand how the test structure needs to be created in order to avoid that timeout problem when running in parallel. Thanks for the help!

[Matt Shirley]

Just out of curiosity, what happens if you change browser.close(); to await browser.close();?

[Lucian Radu]

Just tried now, unfortunately the same issue...I've added async to afterEach arrow function, put await browser.close() and downgraded puppeteer to 1.1.1 in order for the code to work again but the timeout issue is still present.

[Matt Shirley]

How many test cases are generated? I'm starting to think that you're running out of memory. You might need to continue using runInBand or set the maximum number of workers (facebook.github.io/jest/docs/en/cli.html#maxworkers-num).

[Lucian Radu]

Using yarn test command, 3 separated test scripts are executed simultaneously (from a total of 20 in the entire suite), but each test script is iterating through a test data file and performs same actions for multiple websites. So this is probably the reason I am getting that timeout because sometimes I've got the same error when executing tests sequentially.

[Lucian Radu]

Got rid entirely of this line await page.waitForNavigation({waitUntil: 'domcontentloaded'}) and seems the timeout is now gone..

[Lucian Radu]

Thanks, I will give it a try! :)

[java-addict301]

you probably meant "timeout: 60000"