Pass the user-agent through webdriver in Selenium

Question

I am working on a website scraping project using Selenium in Python. When I open the homepage through a browser, it opens properly.

But, when I try to open the webpage through webdriver() in Selenium, it opens a completely different page.

I think, it is able to detect the user-agent( not sure what it is called) and is able to check the properties of the browser or something.

Is it possible to pass the properties though the webdriver() so that the right homepage is loaded.

Thanks

Solution 1

Changing the user agent in the python version of webdriver is done by altering your browser's profile. I have only done this for webdriver.Firefox() by passing a profile parameter. You need to do the following:

from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override","your_user_agent_string")
driver=webdriver.Firefox(profile)

Every time you wish to change the user agent you will need to restart your web browser (i.e. call driver=webdriver.Firefox(profile) again)

If you are unsure to what your user agent string is do a search for "what is my user agent" on a browser that displays the page properly and just copy and paste that one.

Hope that sorts it.

Solution 2

Assuming the user-agent is the problem, in Java you can modify it like this:

FirefoxProfile profile = new FirefoxProfile();
profile.addAdditionalPreference("general.useragent.override", "some UA string");
WebDriver driver = new FirefoxDriver(profile);

See documentation here.

Comments

[oldboy]

So we MUST change the profile to change the user agent string? the reason i'm asking is because i need to maintain the session to avoid having to log in for every single instance i create

[oldboy]

So we MUST change the profile to change the user agent string? the reason i'm asking is because i need to maintain the session to avoid having to log in for every single instance i create