DeprecationWarning: use setter for headless property instead of set_headless opts.set_headless(headless=True) using Geckodriver and Selenium in Python

Question

I have a very basic Python script that runs perfectly on my local machine (Mint 19), and yet fails on a remote box (Ubuntu 16.04). Same files, both Python 3.7. I have geckodriver in /usr/local/bin and it checks out from path with gecko --version from the command line. I can't figure out what the problem is. The geckodriver.log file just says:

1541268536111   mozrunner::runner   INFO    Running command: "/usr/bin/firefox" "-marionette" "-headless" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofile.Mt6zAyZc7D01"
*** You are running in headless mode.
1541268546125   Marionette  INFO    Listening on port 33632

The error from the terminal is:

root@dev1:/home/krypterro/PycharmProjects/corbot# python3 test1.py
 2018-11-03 12:28:22,442 -  INFO -  Application - Start
test1.py:12: DeprecationWarning: use setter for headless property instead of set_headless
  opts.set_headless(headless=True)
Traceback (most recent call last):
  File "test1.py", line 21, in <module>
    main()
  File "test1.py", line 14, in main
    driver = webdriver.Firefox(options=opts)
  File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 174, in __init__
    keep_alive=True)
  File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: timed out

Here is the Python code:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)


def main():
    logging.info('Application - Start')
    # Operate in headless mode
    opts = Options()
    opts.set_headless(headless=True)
    assert opts.headless
    driver = webdriver.Firefox(options=opts)
    driver.get("https://www.krypterro.com")
    html_src = driver.page_source
    print(html_src)
    driver.close()
    driver.quit()
    logging.info('Application - End')
main()

I have port 4444 allowed in the firewall on the remote box, but as it's a local-to-local connection I am not sure that should even matter.

Solution

This information log...

INFO - Application - Start test1.py:12: DeprecationWarning: use setter for headless property instead of set_headless opts.set_headless(headless=True)

...implies that the set_headless opts.set_headless(headless=True) is deprecated and you have to use the setter for headless property as follows:

opts = Options()
opts.headless = True
driver = webdriver.Firefox(options=opts)
driver.get("https://www.krypterro.com")

You can find the detailed discussion in How to make firefox headless programmatically in Selenium with python?

Moving ahead as you are trying to retrive the Page Source and as the Web Application is JavaScript enabled you need to induce WebDriverWait and you can use the following solution:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

driver.get("https://www.krypterro.com")
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//h2[contains(.,'Products and Services')]")))
    html_src = driver.page_source
    print(html_src)
    driver.quit()

Note B: You don't need to invoke driver.close() and driver.quit() rather always invoke driver.quit() only within tearDown(){} method to close & destroy the WebDriver and Web Client instances gracefully.

Comments

[Corey Goldberg]

while it's a good idea to avoid deprecated code, the deprecation warning is just a warning printer to stderr. The OP's stacktrace shows an actual error occurring after that..

[krypterro]

Added that fix for the headless, the warning goes away, thanks for that. But I still get the same error, and it still crashes.

[undetected Selenium]

@krypterro Checkout my updated Answer and let me know the status

[krypterro]

I narrowed the problem down to some weird network issues on the specific VPS server, I spun up a new box, and the code above works, and now doesn't throw the error. Thank you.