How to pass parameter to Url with Python urlopen

Question

I'm currently new to python programming. My problem is that my python program doesn't seem to pass/encode the parameter properly to the ASP file that I've created. This is my sample code:

import urllib.request

url = 'http://www.sample.com/myASP.asp'
full_url = url + "?data='" + str(sentData).replace("'", '"').replace(" ", "%20").replace('"', "%22") + "'"
print (full_url)
response = urllib.request.urlopen(full_url)
print(response)

the output would give me something like:

http://www.sample.com/myASP.asp?data='{%22mykey%22:%20[{%22idno%22:%20%22id123%22,%20%22name%22:%20%22ej%22}]}'

The asp file is suppose to insert the acquired querystring to a database.. But whenever I check my database, no record is saved. Though if I do copy and paste the printed output on my browser url, the record is saved. Any input on this? TIA

Update: Is it possible the python calls my ASP File A but it doesn't call my ASP File B? ASP File A is called by python while ASP File B is called by ASP File A. Because whenever I run the url on a browser, the saving goes well. But in python, no saving of database occurs even though the data passed from python is read by ASP File A..

Solution 1

In case anybody stumbles upon this, this is what I've come up with:

py file:

url = "my.url.com"
data = {'sample': 'data'}

encodeddata = urllib.parse.urlencode(data).encode('UTF-8')
req = urllib.request.Request(url, encodeddata)
response = urllib.request.urlopen(req)

and in my asp file, I used json2.js:

jsondata = request.form("data")
jsondata = replace(jsondata,"'","""")
SET jsondata = JSON.parse(jsontimecard)

Note: use requests instead. ;)

Solution 2

Use firebug with Firefox and watch the network traffic when the page is loaded. If it is actually an HTTP POST, which I suspect it is, check the post parameters on that post and do something like this:

from BeautifulSoup import BeautifulSoup
import urllib

post_params = {
              'param1' : 'val1',
              'param2' : 'val2',
              'param3' : 'val3'
              }
post_args = urllib.urlencode(post_params)

url = 'http://www.sample.com/myASP.asp'
fp = urllib.urlopen(url, post_args)
soup = BeautifulSoup(fp)

If its actually HTTP POST, this will work.

Comments

[Eduard]

Yes you are correct. The asp file has a Request.Querystring(..) that gets the sample data parameter string which is '{%22mykey%22:%20[{%22idno%22:%20%22id123%22,%20%22name%22:%‌​20%22ej%22}]}'.. Btw, I believe that I did not provide a data parameter on my urlopen.. or am I wrong? Because from what I've read, the structure when passing data is something like this: response = urllib.request.urlopen(full_url, data)

[Eduard]

will this affect the asp file? Because I am currenty using request.querystring.. should I change it to Request.form?

[Flakes]

from your post: full_url = url + "?data='" + str(sentData)...

[That1Guy]

I'm not sure about that, to be honest. It couldn't hurt. Give it a try and let me know how it works out! =)

[Eduard]

yes but the syntax used on urllib.request.urlopen() contains only 1 parameter, which is the url. response = urllib.request.urlopen(full_url)

[Eduard]

btw, could you please explain the last part soup = BeautifulSoup(fp) ? I can't seem to understand what this does..

[That1Guy]

Oh, sorry. BeautifulSoup is an html parsing library for python. It makes parsing web pages a lot easier. The documentation can be found at 'crummy.com/software/BeautifulSoup'. Also, if this helped, I'd like it if you accepted this answer as correct. No pressure, but it helps out my rep =)

[Eduard]

Yes I'll do that once I've verified the code. Thanks a ton bro

[That1Guy]

No problem. I'm here to help. If there is anything else I can do, just leave a comment and I'll do my best.

[Eduard]

It is a GET, and even though I changed it to POST, still no data is processed. I even tried changing the charset upon retrieving the values as utf-8 on my asp file

[Flakes]

but your full_url contains the data . Just change the Request.Querystring(..) to Request(..) and see.

[Eduard]

Request()? Doesn't ASP use Request.Querystring for GET and Request.Form for POST ? Sorry I'm not really familiar with ASP..

[Flakes]

just Request(..) will also work. It first looks in the querystring, then the form , then the cookies ... .It's just a shortcut.

[Flakes]

from this link : strName = Request("name") returns the value of the "name" key regardless of the collection in which it's located, because IIS searches all collections. When you specify a value in this manner, ASP looks through each Request object collection in the following order:...

[That1Guy]

Hmmm...If this was a page I was parsing, at this point I would use selenium. Its hard to figure out what is actually going on without viewing the actual page. If I knew what was going on, I'd be more likely to figure it out. I'd recommend using Selenium for this, though. See my answer on this question: stackoverflow.com/questions/13039530/… for some insight as to how to proceed. Also, the Selenium documentation can be found here: seleniumhq.org This WILL do the job if you use it correctly! =)