Puppeteer | Wait for all JavaScript is executed

Question

I try to take screenshots from multiple pages, which should be fully loaded (including lazy loaded images) for later comparison.

I found the lazyimages_without_scroll_events.js example which helps a lot.

With the following code the screenshots are looking fine, but there is some major issue.

async function takeScreenshot(browser, viewport, route) {
  return browser.newPage().then(async (page) => {
    const fileName = `${viewport.directory}/${getFilename(route)}`;

    await page.setViewport({
      width: viewport.width,
      height: 500,
    });
    await page.goto(
        `${config.server.master}${route}.html`,
        {
          waitUntil: 'networkidle0',
        }
    );
    await page.evaluate(() => {
      /* global document,requestAnimationFrame */
      let lastScrollTop = document.scrollingElement.scrollTop;

      // Scroll to bottom of page until we can't scroll anymore.
      const scroll = () => {
        document.scrollingElement.scrollTop += 100;
        if (document.scrollingElement.scrollTop !== lastScrollTop) {
          lastScrollTop = document.scrollingElement.scrollTop;
          requestAnimationFrame(scroll);
        }
      };
      scroll();
    });
    await page.waitFor(5000);
    await page.screenshot({
      path: `screenshots/master/${fileName}.png`,
      fullPage: true,
    });

    await page.close();
    console.log(`Viewport "${viewport.name}", Route "${route}"`);
  });
}

Issue: Even with higher values for page.waitFor() (timeout), sometimes not the all of the frontend related JavaScripts on the pages were fully executed.

For some older pages where some JavaScript could change the frontend. F.e. in one legacy case a jQuery.matchHeight.

Best case: In an ideal world Puppeteer would wait till all JavaScript is evaluated and executed. Is something like this possible?

---

EDIT
I could improve the script slightly with the help from cody-g.

function jQueryMatchHeightIsProcessed() {
  return Array.from($('.match-height')).every((element) => {
    return element.style.height !== '';
  });
}

// Within takeScreenshot() after page.waitFor()
await page.waitForFunction(jQueryMatchHeightIsProcessed, {timeout: 0});

... but it is far from perfect. It seems I have to find similar solutions for different frontend scripts to really consider everything which happening on the target page.

The main problem with jQuery.matchHeight in my case is that it does process different heights in different runs. Maybe caused by image lazyloading. It seems I have to wait until I can replace it with Flexbox. (^_^)°

Other issues to fix:

Disable animations:

await page.addStyleTag({
  content: `
    * {
      transition: none !important;
      animation: none !important;
    }
  `,
});

Handle slideshows:

function handleSwiperSlideshows() {
  Array.from($('.swiper-container')).forEach((element) => {
    if (typeof element.swiper !== 'undefined') {
      if (element.swiper.autoplaying) {
        element.swiper.stopAutoplay();
        element.swiper.slideTo(0);
      }
    }
  });
}

// Within takeScreenshot() after page.waitFor()
await page.evaluate(handleSwiperSlideshows);

But still not enough. I think it's impossible to visual test these legacy pages.

Solution 1

The following waitForFunction might be useful for you, you can use it to wait for any arbitrary function to evaluate to true. If you have access to the page's code you can set the window status and use that to notify puppeteer it is safe to continue, or just rely on some sort of other ready state. Note: this function is a polling function, and re-evaluates at some interval which can be specified.

const watchDog = page.waitForFunction('<your function to evaluate to true>');

E.g.,

const watchDog = page.waitForFunction('window.status === "ready"');
await watchDog;

In your page's code you simply need to set the window.status to ready

To utilize multiple watchdogs in multiple asynchronous files you could do something like

index.js

...import/require file1.js;
...import/require file2.js;
...code...

file1.js:

var file1Flag=false; // global
...code...
file1Flag=true;

file2.js:

var file2Flag=false; // global
...code...
file2Flag=true;

main.js:

const watchDog = page.waitForFunction('file1Flag && file2Flag');
await watchDog;

Solution 2

async function takeScreenshot(browser, viewport, route) {
  return browser.newPage().then(async (page) => {
    const fileName = `${viewport.directory}/${getFilename(route)}`;

    await page.setViewport({
      width: viewport.width,
      height: 500,
    });
    await page.goto(
        `${config.server.master}${route}.html`,
        {
          waitUntil: 'networkidle0',
        }
    );
    await page.evaluate(() => {
      scroll(0, 99999)
    });
    await page.waitFor(5000);
    await page.screenshot({
      path: `screenshots/master/${fileName}.png`,
      fullPage: true,
    });

    await page.close();
    console.log(`Viewport "${viewport.name}", Route "${route}"`);
  });
}