"Clicking" button with requests

Question

I have this little website i want to fill in a form with the requests library. The problem is i cant get to the next site when filling the form data and hitting the button(Enter does not work).

The important thing is I can't do it via a clicking bot of some kind. This needs to be done so I can run in without graphics.

info = {'name':'JohnJohn',
        'message':'XXX',
        'sign':"XXX",
        'step':'1'}

First three entries name, message, sign are the text areas and step is I think the button.

r = requests.get(url)
r = requests.post(url, data=info)

print(r.text)

The Form Data looks like this when i send a request via chrome manually:

• name:JohnJohn
• message:XXX
• sign:XXX
• step:1

The button element looks like this:

<td colspan="2" style="text-align: center;">
    <input name="step" type="hidden" value="1">
    <button id="button" type="button" onclick="myClick();"
     style="background-color: #ef4023;  width: 80px; font-face: times; font-size: 14pt;">
        Wyślij
    </button>
</td>

The next site if i do this manually has the same adres.

Solution 1

As you might see from the snipped you posted, clicking the button is triggering some JavaScript code, namely a method called myClick().

It is not straightforward to click on this thing using pythons requests library. You might have more luck trying to find out what happens inside myClick(). My guess would be that at some point, a POST request will be made to a HTTP endpoint. If you can figure this out, you can translate it into your python code.

If that does not work, another option would be to use something like Selenium/PhantomJS, which give you the ability to have a real, headless and scriptable browser. Using such a tool, you can actually have it fill out forms and click buttons. You can have a look at this so answer, as it shows you how to use Selenium+PhantomJS from python.

Please make sure not to abuse such methods by spamming forums or [insert illegal or otherwise abusive activity here].

Solution 2

In such a situation when you need to forge scripted button's request, it may be easier not to guess the logic of JS but instead perform a physical click and look into chrome devtools' network sniffer which gives you a plain request made which, in turn, can be easily forged in Python

Comments

[Padraic Cunningham]

Can you share the link?

[Repcak]

How could i figure out what is happening inside myClick()? Also isn't Selenium something like emulating physical clicks so i would need to have a browser open etc? I would like to avoid that. It's mostly for educational purposes. I'm interested with bots (Good ones) and webscraping.

[Dave]

PhantomJS is a headless browser, meaning that there is no window open, it's just a running process. A chrome process can be started in a similar fashion. Since you are getting served the .js files that are required for the page, just open the Chrome Dev Tools and find the right one in the "Resources" Tab.

[Kranthi Kiran]

How do I know where the myClick() is making a POST request? @Dave

[BallpointBen]

@KranthiKiran You have to find the definition of myClick and see what it does.

[Dave L]

@KranthiKiran I use Firefox (and I assume chrome devtools has this option) but if you look at the network tab, you can see the requests being made after clicking javascript elements. I have a question (similar to the author's) that I am trying to tackle without selenium or PhantomJS because I want my app to be light (and I want to learn more about requests).

[Julian]

Can anyone provide an example for a "Load more" button? I need to understand the research process behind that, and so far I'm stuck at dev tools on the browser > Network > XHR > a Post request and viewing the Response. New to AJAX / JS scraping.