How do I set headers using python's urllib?
157,958
Solution 1
adding HTTP headers using urllib2:
from the docs:
import urllib2
req = urllib2.Request('http://www.example.com/')
req.add_header('Referer', 'http://www.python.org/')
resp = urllib2.urlopen(req)
content = resp.read()
Solution 2
For both Python 3 and Python 2, this works:
try:
from urllib.request import Request, urlopen # Python 3
except ImportError:
from urllib2 import Request, urlopen # Python 2
req = Request('http://api.company.com/items/details?country=US&language=en')
req.add_header('apikey', 'xxx')
content = urlopen(req).read()
print(content)
Solution 3
Use urllib2 and create a Request object which you then hand to urlopen. http://docs.python.org/library/urllib2.html
I dont really use the "old" urllib anymore.
req = urllib2.Request("http://google.com", None, {'User-agent' : 'Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5'})
response = urllib2.urlopen(req).read()
untested....
Solution 4
For multiple headers do as follow:
import urllib2
req = urllib2.Request('http://www.example.com/')
req.add_header('param1', '212212')
req.add_header('param2', '12345678')
req.add_header('other_param1', 'sample')
req.add_header('other_param2', 'sample1111')
req.add_header('and_any_other_parame', 'testttt')
resp = urllib2.urlopen(req)
content = resp.read()
Related videos on Youtube
Author by
ewok
Software engineer in the Greater Boston Area. Primary areas of expertise include Java, Python, web-dev, and general OOP, though I have dabbled in many other technologies.
Updated on March 28, 2020Comments
-
ewok about 4 years
I am pretty new to python's urllib. What I need to do is set a custom header for the request being sent to the server. Specifically, I need to set the Content-type and Authorizations headers. I have looked into the python documentation, but I haven't been able to find it.
-
user3378649 almost 9 yearsCan we do the same thing with requests q.add_header('apikey', 'xxx')
-
Cees Timmerman almost 9 yearsWhat do you mean, @user3378649?
-
WeizhongTu about 8 years@user3378649 may be you means use
requests
python package custom headers -
Beorn Harris over 4 yearsTHIS answer - a thousand times YES (thanks!). I have been struggling for hours trying to find a common interface for python 2 and 3 (between urllib, urllib2 and urllib3).
-
Raleigh L. almost 2 yearsDo not do this, simply pass in a headers dictionary if you have multiple header fields to use.
-
Gil Allen almost 2 yearsguess you are right - but hey, this was 2015 for Python 2.7 Theses days - I'm populating a dictionary.