Change user-agent for Selenium web-driver
Solution 1
There is no way in Selenium to read the request or response headers. You could do it by instructing your browser to connect through a proxy that records this kind of information.
Setting the User Agent in Firefox
The usual way to change the user agent for Firefox is to set the variable "general.useragent.override"
in your Firefox profile. Note that this is independent from Selenium.
You can direct Selenium to use a profile different from the default one, like this:
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override", "whatever you want")
driver = webdriver.Firefox(profile)
Setting the User Agent in Chrome
With Chrome, what you want to do is use the user-agent
command line option. Again, this is not a Selenium thing. You can invoke Chrome at the command line with chrome --user-agent=foo
to set the agent to the value foo
.
With Selenium you set it like this:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
opts = Options()
opts.add_argument("user-agent=whatever you want")
driver = webdriver.Chrome(chrome_options=opts)
Both methods above were tested and found to work. I don't know about other browsers.
Getting the User Agent
Selenium does not have methods to query the user agent from an instance of WebDriver
. Even in the case of Firefox, you cannot discover the default user agent by checking what general.useragent.override
would be if not set to a custom value. (This setting does not exist before it is set to some value.)
Once the browser is started, however, you can get the user agent by executing:
agent = driver.execute_script("return navigator.userAgent")
The agent
variable will contain the user agent.
Solution 2
To build on Louis's helpful answer...
Setting the User Agent in PhantomJS
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
...
caps = DesiredCapabilities.PHANTOMJS
caps["phantomjs.page.settings.userAgent"] = "whatever you want"
driver = webdriver.PhantomJS(desired_capabilities=caps)
The only minor issue is that, unlike for Firefox and Chrome, this does not return your custom setting:
driver.execute_script("return navigator.userAgent")
So, if anyone figures out how to do that in PhantomJS, please edit my answer or add a comment below! Cheers.
Solution 3
This is a short solution to change the request UserAgent on the fly.
Change UserAgent of a request with Chrome
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
driver = webdriver.Chrome(driver_path)
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent":"python 2.7", "platform":"Windows"})
driver.get('http://amiunique.org')
then return your useragent:
agent = driver.execute_script("return navigator.userAgent")
Some sources
The source code of webdriver.py from SeleniumHQ (https://github.com/SeleniumHQ/selenium/blob/11c25d75bd7ed22e6172d6a2a795a1d195fb0875/py/selenium/webdriver/chrome/webdriver.py) extends its functionalities through the Chrome Devtools Protocol
def execute_cdp_cmd(self, cmd, cmd_args):
"""
Execute Chrome Devtools Protocol command and get returned result
We can use the Chrome Devtools Protocol Viewer to list more extended functionalities (https://chromedevtools.github.io/devtools-protocol/tot/Network#method-setUserAgentOverride) as well as the parameters type to use.
Related videos on Youtube
xralf
Updated on January 16, 2021Comments
-
xralf over 3 years
I have the following code in
Python
:from selenium.webdriver import Firefox from contextlib import closing with closing(Firefox()) as browser: browser.get(url)
I would like to print the user-agent HTTP header and possibly change it. Is it possible?
-
xralf about 9 yearsNote that I have
from selenium.webdriver import Firefox
. I'm trying to figure out, how to set the profile forFirefox
, I imported fromselenium.webdriver
. -
xralf about 9 yearsuser-agent is not request or response header, but it is general header.
-
Louis about 9 yearsI deal with Firefox in my answer in the first code snippet. Also,
User-Agent
absolutely is a request header. See section 14.43: "The User-Agent request-header field contains information about the user agent originating the request." (Emphasis mine.) -
xralf about 9 yearsOK, I combined it with my code. So, for setting user-agent is there a set_preference method. Is there something like get_preference as well, to know what it was before?
-
xralf about 9 yearsYou're right, it is a request-header field. In the book HTTP essentials (page 56) is mistake.
-
Louis about 9 yearsI've edited my answer with a section about one way to get the user agent.
-
JJC over 7 yearsNice. This was very helpful. I added an answer for how to do this in PhantomJS too.
-
oldboy almost 6 yearsSo we MUST change the profile to change the user agent string? the reason i'm asking is because i need to maintain the session to avoid having to log in for every single instance i create
-
Hamed Baziyad over 5 yearsHi, Could you tell me what's your mean from "your_path" in this part: driver = webdriver.PhantomJS(executable_path=r"your_path", desired_capabilities=dcap)
-
Hamed Baziyad over 5 yearsI guess you want to say address of "phantomjs.exe" in "bin file".
-
Hamed Baziyad over 5 yearsI do it but, this massage will be published for me: http.client.RemoteDisconnected: Remote end closed connection without response
-
Hamed Baziyad over 5 yearsI do it but this massage is published when I run it: http.client.RemoteDisconnected: Remote end closed connection without response
-
Hamed Baziyad over 5 yearsI do it but I can't see any change in browser is "under remote control" mood. Can you analyze it?
-
Hamed Baziyad over 5 yearsI do it but my browser (FF) is under remote controller yet. Is it correct?
-
X.C. over 4 yearsThere is a problem with this method that "Firefox Profile - Cannot open window as TAB.“ github.com/mozilla/geckodriver/issues/1139
-
migrant over 3 yearsThis would be useful when I want to change the user agent on every request, but not the whole driver session
-
user1871891 over 2 yearsBtw call me a newbie, though "whatever you want" is a bit misleading lol. I literally put a random string and well, didn't realise that there was more to it. Finally got a proper glimpse of the structure from here intoli.com/blog/making-chrome-headless-undetectable