Puppeteer browser useragent list

Question

I would like to load random list of user-agents from my default location path , for example: 'agents.json' instead of adding direct and only 1 user-agent.

agents.json

["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36", "Mozilla/5.0 (iPad; CPU OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/79.0.3945.73 Mobile/15E148 Safari/604.1"]

My code:

const puppeteer = require('../core/puppeteer');
const referers = require('../core/referers.json');
const referers = require('../core/agents.json');
const viewVideosInBatch = async ({ targetUrls, durationInSeconds, port }) => {
  let browser;
  try {
    browser = await puppeteer.getBrowserInstance(port);
    const randomReferer = referers[Math.floor(Math.random() * referers.length)];
    const randomAgents = agents[Math.floor(Math.random() * agents.length)];
    const page = await browser.newPage();
    browser = await page.setUserAgent({ agents: randomAgents });
    page.setDefaultTimeout(PAGE_DEFAULT_TIMEOUT * 1000);
    page.on('error', handlePageCrash(page));
    page.on('pageerror', handlePageCrash(page));
    page.setExtraHTTPHeaders({ referer: randomReferer });

    await page.setViewport({
      width: 640,
      height: 480,
      deviceScaleFactor: 1,
    });
    const ipAddr = await getCurrentIP(page);
    const targetUrlsForAction = _take(_shuffle(targetUrls), VIEW_ACTION_COUNT);
    await watchVideosInSequence(page, ipAddr, targetUrlsForAction, durationInSeconds);
    await page.close();
  } catch (error) {
    logger.warn('Entire view action in a batch failed.');
    logger.debug(error);
  } finally {
    await browser.close();
  }
};

ERROR I'm getting:

(node:1) UnhandledPromiseRejectionWarning: Error: Protocol error (Network.setUserAgentOverride): Target closed.
view_4  |     at /app/node_modules/puppeteer/lib/Connection.js:183:56
view_4  |     at new Promise (<anonymous>)
view_4  |     at CDPSession.send (/app/node_modules/puppeteer/lib/Connection.js:182:12)
view_4  |     at next (/app/node_modules/puppeteer-extra-plugin-stealth/evasions/sourceurl/index.js:30:43)
view_4  |     at CDPSession.send (/app/node_modules/puppeteer-extra-plugin-stealth/evasions/sourceurl/index.js:46:18)
view_4  |     at Plugin.onPageCreated (/app/node_modules/puppeteer-extra-plugin-stealth/evasions/user-agent-override/index.js:75:18)
view_4  |     at runMicrotasks (<anonymous>)
view_4  |     at processTicksAndRejections (internal/process/task_queues.js:97:5)
view_4  |     at async Plugin._onTargetCreated (/app/node_modules/puppeteer-extra-plugin/dist/index.cjs.js:491:17)
view_4  | (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 79)

Solution

You seem to redeclare browser when you set the UA, which you shouldn't. Also you don't need to set it as an object: as page.setUserAgent accept a string: page.setUserAgent(userAgent).

await page.setUserAgent(randomAgent); will be just fine.

const browser = await puppeteer.getBrowserInstance(port);
const randomReferer = referers[Math.floor(Math.random() * referers.length)];
const randomAgent = agents[Math.floor(Math.random() * agents.length)]; // made it singular instead of plural
const page = await browser.newPage();
await page.setUserAgent(randomAgent); // like this
page.setDefaultTimeout(PAGE_DEFAULT_TIMEOUT * 1000);
page.on('error', handlePageCrash(page));
page.on('pageerror', handlePageCrash(page));
page.setExtraHTTPHeaders({ referer: randomReferer });

Note: I think it is only a mistake in your post, but you also have two const named referers, this one should be the agents:

const referers = require('../core/agents.json');

Comments

[Tran]

yes you totally right again :) haha but i dont understnd this: // made it singular instead of plural , can you explain?

[Tran]

also the json should be loaded with useragents in " " OR ' '

[Tran]

sorry can not acept answer for now, I get the exeption error: SyntaxError: /app/core/agents.json: Unexpected token ' in JSON at position 1 | at parse (<anonymous>)

[theDavidBarton]

by the plural to singular I meant: randomAgents => randomAgent without an 's' in the end, because it will be only one user agent value, not more agents. In JSON (if I understand correctly your 2nd comment) you should always use the double quotes " ", the unexpected token might be a syntax error due to single quotes.

[Tran]

hi, yes it was a single quotes issue, now if i added mobile ipad useragent the page browser crash and throw some error: Error: EvalError: Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'report-sample' 'nonce-4/lYxaZ4FmRH/FosJwePuw' 'unsafe-inline'". Any idea why? Thx again

[theDavidBarton]

not sure. I suggest to create a new question for this. I am sure it is something some experts can answer you in no time, and also it can be useful answer for others, would be wasted to hide the answer here in the comments :)