How can I simulate onclick event in python?

Question

I am working on a small project where I have to submit a form to a website. The website is, however, using onclick event to submit the form (using javascript).

1. How can the onclick event be simulated in python?

2. Which modules can be used? I have heard about selenium and mechanize modules. But, which module can be used or in case of both, which one is better?

I am new to web scraping and automation.So,it would be very helpful.

Thanks in advance.

Solution 1

Ideally you don't even need to clicks buttons in these kind of cases.

All you need is to see at what webservice does the form sends request when clicked on submit button.

For that open your developer's control in the browser, Go to the Network tab and select 'preserve log'. Now submit the form manually and look for the first xhr GET/POST request sent. It would be POST request 90% of times.

Now when you select that request in the request parameters it would show the values that you entered while submitting the form. Bingo!!

Now all you need to do is mimic this request with relevant request headers and parameters in your python code using requests. And Wooshh!!

Hope it helps..

Solution 2

There is no silver bullet in simulating onclick events on a web page. It is pretty much use-case specific, but here are some points and guidelines.

In general, there are two approaches:

• use browser developer tools, open the network tab, make the click and see what request is being sent to the server. Then, simulate this request in Python, with, for example, requests.
• use selenium which would fire up a real browser where you would find the specific element and click via .click() method

mechanize would not execute/trigger the onclick function, because executing onclick requires executing javascript which mechanize cannot do.

Comments

[Greg]

This can be done with selenium. I don't know about mechanize.

[JeffC]

What have you tried and what was the result? Please read the help topics on how to ask a good question. You need to research your own issue, find code samples, etc. and write your own code to solve the issue. If you do all that and still can't figure it out, then come back and edit your question and add notes from the research you did, the code you have tried reduced to a minimal reproducible example, and what the result was... any error messages, etc. It's also very important to include any relevant HTML and properly format the HTML and code.