how do POST request in puppeteer?

Question

(async() => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto('https://www.example.com/search');
    const data = await page.content();
    browser.close();
    res.send(data);
})();

I do this code for send get request. I don't understand how I should send post request?

Solution 1

Getting the "order" right can be a bit of a challenge. Documentation doesn't have that many examples... there are some juicy items in the repository in the example folder that you should definitely take a look at.

https://github.com/GoogleChrome/puppeteer/tree/master/examples

Here is the example; place the following into an async block:

    // Create browser instance, and give it a first tab
    const browser = await puppeteer.launch();
    const page = await browser.newPage();

    // Allows you to intercept a request; must appear before
    // your first page.goto()
    await page.setRequestInterception(true);

    // Request intercept handler... will be triggered with 
    // each page.goto() statement
    page.on('request', interceptedRequest => {

        // Here, is where you change the request method and 
        // add your post data
        var data = {
            'method': 'POST',
            'postData': 'paramFoo=valueBar&paramThis=valueThat'
        };

        // Request modified... finish sending! 
        interceptedRequest.continue(data);
    });

    // Navigate, trigger the intercept, and resolve the response
    const response = await page.goto('https://www.example.com/search');     
    const responseBody = await response.text();
    console.log(responseBody);

    // Close the browser - done! 
    await browser.close();

Solution 2

There's a quirk with the way setRequestInterception and the 'request' event work. Once activated, Puppeteer will send the POST data to every resource on the page, not just the original requested page. I was having the issue that all of my page resources (scripts, CSS) were failing to load once I added POST data in Puppeteer.

Since I only want to apply POST data to the first request, this code worked for me:

// Used for serializing POST parameters from an object
const querystring = require('querystring');

// ...

const browser = await puppeteer.launch();
const page = await browser.newPage();

let postData = {a: 1, b: 2};

await page.setRequestInterception(true);

page.once('request', request => {
    var data = {
        'method': 'POST',
        'postData': querystring.stringify(postData),
        'headers': {
            ...request.headers(),
            'Content-Type': 'application/x-www-form-urlencoded'
        },
    };

    request.continue(data);

    // Immediately disable setRequestInterception, or all other requests will hang
    page.setRequestInterception(false);
});

const response = await page.goto('https://www.example.com/');

Solution 3

Here the complete example with Puppeteer 2.0.0 :

    const puppeteer = require("puppeteer");
    const devices = require("puppeteer/DeviceDescriptors");

    async function main() {
      const browser = await puppeteer.launch({
        args: ["--enable-features=NetworkService", "--no-sandbox"],
        ignoreHTTPSErrors: true
      });
      const page = await browser.newPage();

      await page.setRequestInterception(true);

      page.once("request", interceptedRequest => {
        interceptedRequest.continue({
          method: "POST",
          postData: "foo=FOO&bar=BAR",
          headers: {
            ...interceptedRequest.headers(),
            "Content-Type": "application/x-www-form-urlencoded"
          }
        });
      });

      const response = await page.goto("https://postman-echo.com/post");

      console.log({
        url: response.url(),
        statusCode: response.status(),
        body: await response.text()
      });

      await browser.close();
    }

    main();

Note that if you check response.request().method() it won't be updated (still GET)

Comments

[CodeGuru]

Is there anyway i can start the setRequestInterception(true) only on the last request? I'm trying to find a way to send my crawled data

[JBis]

Can you check if this still works, I copied directly from your answer and from the documentation but it is not working.

[Martijn Pieters]

@JBis did you maybe upgrade puppeteer without upgrading Chrome? Puppeteer 1.15 requires Chrome v75+, source: GitHub issue comment

[Matej J]

Is it possible to make request data different for each url?

[mopsled]

I had to add the "Content-Type": "application/x-www-form-urlencoded" header from Dmitry Sheiko's answer below for this to work.

[Neil Gaetano Lindberg]

My lint complains about const devices being defined and never used when pasting this sample. Then, I can't run it because Error: Cannot find module 'puppeteer/DeviceDescriptors'.

[Neil Gaetano Lindberg]

Wishing I had scrolled down about 5 hours ago. I found the disabling of request interception absolutely critical where I'm using puppeteer to make dynamic PDFs in an Express app. That one line was all I needed to move on!

[ggorlen]

You might also want to use .once rather than .on so the handler is removed after you send it. Ditto for page.setRequestInterception(false); after the request is sent.