urllib3.urlencode googlescholar url from string

Question

I am trying to encode a string to url to search google scholar, soon to realize, urlencode is not provided in urllib3.

>>> import urllib3
>>> string = "https://scholar.google.com/scholar?" + urllib3.urlencode( {"q":"rudra banerjee"} )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'urlencode'

So, I checked urllib3 doc and found, I possibly need request_encode_url. But I have no experience in using that and failed.

>>> string = "https://scholar.google.com/scholar?" +"rudra banerjee"
>>> url = urllib3.request_encode_url('POST',string)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'request_encode_url'

So, how I can encode a string to url?

NB I don't have any particular fascination to urllib3. so, any other module will also do.

Solution 1

(Edit: I assumed you wanted to download the URL, not simply encode it. My mistake. I'll leave this answer as a reference for others, but see the other answer for encoding a URL.)

If you pass a dictionary into fields, urllib3 will take care of encoding it for you. First, you'll need to instantiate a pool for your connections. Here's a full example:

import urllib3
http = urllib3.PoolManager()

r = http.request('POST', 'https://scholar.google.com/scholar', fields={"q":"rudra banerjee"})
print(r.data)

Calling .request(...) will take care of figuring out the encoding for you based on the method.

Getting started examples are here: https://urllib3.readthedocs.org/en/latest/index.html#usage

Solution 2

To simply encode fields in a URL, you can use urllib.urlencode.

In Python 2, this should do the trick:

import urllib
s = "https://scholar.google.com/scholar?" + urllib.urlencode({"q":"rudra banerjee"})
print(s)
# Prints: https://scholar.google.com/scholar?q=rudra+banerjee

In Python 3, it lives under urllib.parse.urlencode instead.

Comments

[BaRud]

hi, thanks for the reply. but here r is <class 'urllib3.response.HTTPResponse'>. but I need to open the url in webbrowser module, as url. while trying, getting error: TypeError: Can't convert 'HTTPResponse' object to str implicitly

[shazow]

Ah I see, you don't want to download the website, you just want to encode the URL. I'll add a separate answer for that.