Python: Clicking a button with urllib or urllib2

Question

I want to click a button with python, the info for the form is automatically filled by the webpage. the HTML code for sending a request to the button is:

INPUT type="submit" value="Place a Bid">

How would I go about doing this? Is it possible to click the button with just urllib or urllib2? Or will I need to use something like mechanize or twill?

Solution 1

Use the form target and send any input as post data like this:

<form target="http://mysite.com/blah.php" method="GET">
    ......
    ......
    ......
    <input type="text" name="in1" value="abc">
    <INPUT type="submit" value="Place a Bid">
</form>

Python:

# parse the page HTML with the form to get the form target and any input names and values... (except for a submit and reset button)
# You can use XML.dom.minidom or htmlparser
# form_target gets parsed into "http://mysite.com/blah.php"
# input1_name gets parsed into "in1"
# input1_value gets parsed into "abc"

form_url = form_target + "?" + input1_name + "=" + input1_value
# form_url value is "http://mysite.com/blah.php?in1=abc"

# Then open the new URL which is the same as clicking the submit button
s = urllib2.urlopen(form_url)

You can parse the HTML with HTMLParser

And don't forget to urlencode any post data with:

urllib.urlencode(query)

Solution 2

You may want to take a look at IronWatin - https://github.com/rtyler/IronWatin to fill the form and "click" the button using code.

Solution 3

Using urllib.urlopen, you could send the values of the form as the data parameter to the page specified in the form tag. But this won't automate your browser for you, so you'd have to get the form values some other way first.

Comments

[imm]

@Dan Dobint, What is automatically filling those form values? Javascript? Your browser's form autocomplete function? Are there default values for this form that are specified in each input element's value attribute?

[Dan Doe]

There is only one value and it appears that it is just pre-specified in the source code as a default value unless you change it manually.

[imm]

@DanDobint, see docs.python.org/library/urllib.html . The URL to use for urlopen is whatever the form tag's action parameter is. You may specify that single field in urlopen's data parameter.