Sending "User-agent" using Requests library in Python

Question

I want to send a value for "User-agent" while requesting a webpage using Python Requests. I am not sure is if it is okay to send this as a part of the header, as in the code below:

debug = {'verbose': sys.stderr}
user_agent = {'User-agent': 'Mozilla/5.0'}
response  = requests.get(url, headers = user_agent, config=debug)

The debug information isn't showing the headers being sent during the request.

Is it acceptable to send this information in the header? If not, how can I send it?

Solution 1

The user-agent should be specified as a field in the header.

Here is a list of HTTP header fields, and you'd probably be interested in request-specific fields, which includes User-Agent.

If you're using requests v2.13 and newer

The simplest way to do what you want is to create a dictionary and specify your headers directly, like so:

import requests

url = 'SOME URL'

headers = {
    'User-Agent': 'My User Agent 1.0',
    'From': '[email protected]'  # This is another valid field
}

response = requests.get(url, headers=headers)

If you're using requests v2.12.x and older

Older versions of requests clobbered default headers, so you'd want to do the following to preserve default headers and then add your own to them.

import requests

url = 'SOME URL'

# Get a copy of the default headers that requests would use
headers = requests.utils.default_headers()

# Update the headers with your custom ones
# You don't have to worry about case-sensitivity with
# the dictionary keys, because default_headers uses a custom
# CaseInsensitiveDict implementation within requests' source code.
headers.update(
    {
        'User-Agent': 'My User Agent 1.0',
    }
)

response = requests.get(url, headers=headers)

Solution 2

It's more convenient to use a session, this way you don't have to remember to set headers each time:

session = requests.Session()
session.headers.update({'User-Agent': 'Custom user agent'})

session.get('https://httpbin.org/headers')

By default, session also manages cookies for you. In case you want to disable that, see this question.

Comments

[here]

You can also access the headers you sent with response.request.headers , this works because the original request object is an attribute of the response object. See also http://docs.python-requests.org/en/latest/user/advanced/#req‌​uest-and-response-ob‌​jects

[nealmcb]

The default value is also available as requests.utils.default_user_agent() if you want to just augment that with your own info.

[Chad Miller]

It's not correct. It clobbers the rest of the headers. He should get a copy of defaults from requests.utils.default_user_agent() and update it, and send those.

[m3nda]

for easyness, on httpbin.org/headers (downloadable stuff) you can get the browser headers then make your query appear you

[Jmills]

At least in 2.13.0, the headers are not clobbered and the docs just tell you to use the headers kwarg.