How to run Puppeteer code in any web browser?

javascript node.js web-scraping puppeteer

15,261

EDIT: Since puppeteer removed support for puppeteer-web, I moved it out of the repo and tried to patch it a bit.

It does work with browser. The package is called puppeteer-web, specifically made for such cases.

But the main point is, there must be some instance of chrome running on some server. Only then you can connect to it.

You can use it later on in your web page to drive another browser instance through its WS Endpoint:

<script src="https://unpkg.com/puppeteer-web">
</script>

<script>
  const browser = await puppeteer.connect({
    browserWSEndpoint: `ws://0.0.0.0:8080`, // <-- connect to a server running somewhere
    ignoreHTTPSErrors: true
  });

  const pagesCount = (await browser.pages()).length;
  const browserWSEndpoint = await browser.wsEndpoint();
  console.log({ browserWSEndpoint, pagesCount });
</script>

I had some fun with puppeteer and webpack,

See these answers for full understanding of creating the server and more,

15,261

Author by

Admin

Updated on July 14, 2022

Comments

Admin almost 2 years

I'm trying to do some web scraping with Puppeteer and I need to retrieve the value into a Website I'm building.

I have tried to load the Puppeteer file in the html file as if it was a JavaScript file but I keep getting an error. However, if I run it in a cmd window it works well.

Scraper.js:

getPrice();
function getPrice() {
    const puppeteer = require('puppeteer');
    void (async () => {
        try {
            const browser = await puppeteer.launch()
            const page = await browser.newPage()              
            await page.goto('http://example.com') 
            await page.setViewport({ width: 1920, height: 938 })        
            await page.waitForSelector('.m-hotel-info > .l-container > .l-header-section > .l-m-col-2 > .m-button')
            await page.click('.m-hotel-info > .l-container > .l-header-section > .l-m-col-2 > .m-button')
            await page.waitForSelector('.modal-content')
            await page.click('.tile-hsearch-hws > .m-search-tabs > #edit-search-panel > .l-em-reset > .m-field-wrap > .l-xs-col-4 > .analytics-click')
            await page.waitForNavigation();
            await page.waitForSelector('.tile-search-filter > .l-display-none')
            const innerText = await page.evaluate(() => document.querySelector('.tile-search-filter > .l-display-none').innerText);
            console.log(innerText)
        } catch (error) {
            console.log(error)
        }

    })()
}

index.html:

<html>
  <head></head>
  <body>
    <script src="../js/scraper.js" type="text/javascript"></script>
  </body>
</html>

The expected result should be this one in the console of Chrome:

But I'm getting this error instead:

What am I doing wrong?

Admin about 5 years

Well, that's kinda hard in my opinion... I'm trying out another alternative that is called Goutte, it seems to fit better for my needs and it's easier when it comes to show the results in a web browser since it uses PHP and it doesn't require an external command. Anyway, I will save your answer in case I need it in the future. Thank you so much!!
Md. Abu Taher about 5 years

You are welcome. However, since your question was regarding puppeteer, and not about scraping in general, I am sure the answer fits your question perfectly :D .
TheMaster almost 4 years

Official links 404'd
tchan almost 4 years

Here's the old link from a past commit. It looks like they removed it in this pull request
Md. Abu Taher almost 4 years

Looks like I need to find an alternative for it soon and update the answers. :D Gimme a bit of time, I'll check after weekends.
Crashalot about 3 years

This is very interesting. Thanks for sharing. Is it possible to extract the part of puppeteer that creates a PNG/JPG/PDF screenshot of HTML and create a separate library that could run in the browser to allow for browser-based screenshots?