Execute Python (selenium) script in crontab
Solution 1
A quick guess, your job fails to run because it requires an X session. You should setup your test script to run in a headless mode.
update:
Your trace says exactly what I meant, firefox does not run without X, or a display if you want.
The output was: Error: no display specified\n'
Save yourself more trouble, read the link I posted.
Solution 2
As seen here, you need to give cron a display:
30 5 * * * export DISPLAY=:0; /home/me/good_morning.sh
Solution 3
You can do it without crontab.
I found schedule modul: GIThub/schedule
Now your code should look like:
import schedule
import time
def job():
import pickle
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('http://www.google.com')
t=driver.current_url
pickle.dump(t,open('noreal','wb'))
schedule.every(1).minutes.do(job)
while True:
schedule.run_pending()
time.sleep(1)
It is not best solution but it works.
Solution 4
This issue has bugged me for near 6 hours.
So if I write /usr/bin/python3 -u /root/apps/bridge/cinkselenium.py
in terminal it is working fine.
But it doesn't work for crontab. Right?
Adding
from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()
will not help.
What you need is to make a shell script and write the following -
#!/usr/bin/env bash
export DISPLAY=:0
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
nohup /usr/bin/python3 -u /root/apps/bridge/cinkselenium.py > /root/apps/bridge/cinkselenium.log &y
Now run the shell script from Crontab -
* * * * * /root/apps/bridge/cinkselenium.sh
This is what works properly for selenium in crontab without headache in DigitalOcean servers. The paths will be different based on your server and program location.
Captain_Meow_Meow
Updated on June 15, 2022Comments
-
Captain_Meow_Meow almost 2 years
I have read most of the python/cron here in stackoverflow and yet couldn't make my script run. I understood that I need to run my script through shell (using zsh & ipython by the way), but really I have no idea what to do :/
My simple code:
In the crontab-
*/1 * * * * ipython /home/usr/Data/progs/cron_test.py
My python script-
import pickle from selenium import webdriver driver = webdriver.Firefox() driver.get('http://www.google.com') t=driver.current_url pickle.dump(t,open('noreal','wb'))
I have tried some things already but to no avail:
#!python ../python etc SHELL = /usr/bin/zsh PATH =/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
Simply because I don't know what the problem is, I assume something with running python script via interpreter, but I don't know what I am doing really :)
A working solution would be great, and I would really appreciate an explanation to accompany any solution so I could understand the why and how and not just be like 'It works! Thanks! Bye!'
Thanks for any help!
Update So far I have narrowed down the problem, and now python runs with the following settings:
*/3 * * * * /usr/local/bin/ipython /home/user/Data/progs/RF/cron_test.py
and I got a traceback:
[1;31m---------------------------------------------------------------------------[0m [1;31mWebDriverException[0m Traceback (most recent call last) [1;32m/usr/local/lib/python2.7/dist-packages/IPython/utils/py3compat.pyc[0m in [0;36mexecfile[1;34m(fname, *where)[0m [0;32m 176[0m [1;32melse[0m[1;33m:[0m[1;33m[0m[0m [0;32m 177[0m [0mfilename[0m [1;33m=[0m [0mfname[0m[1;33m[0m[0m [1;32m--> 178[1;33m [0m__builtin__[0m[1;33m.[0m[0mexecfile[0m[1;33m([0m[0mfilename[0m[1;33m,[0m [1;33m*[0m[0mwhere[0m[1;33m)[0m[1;33m[0m[0m [0m [1;32m/home/user/Data/progs/FB/cron_test.py[0m in [0;36m<module>[1;34m()[0m [0;32m 9[0m [1;33m[0m[0m [0;32m 10[0m [1;33m[0m[0m [1;32m---> 11[1;33m [0mdriver[0m [1;33m=[0m [0mwebdriver[0m[1;33m.[0m[0mFirefox[0m[1;33m([0m[1;33m)[0m[1;33m[0m[0m [0m[0;32m 12[0m [0mdriver[0m[1;33m.[0m[0mget[0m[1;33m([0m[1;34m'http://www.google.com'[0m[1;33m)[0m[1;33m[0m[0m [0;32m 13[0m [1;33m[0m[0m [1;32m/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.pyc[0m in [0;36m__init__[1;34m(self, firefox_profile, firefox_binary, timeout, capabilities, proxy)[0m [0;32m 58[0m RemoteWebDriver.__init__(self, [0;32m 59[0m command_executor=ExtensionConnection("127.0.0.1", self.profile, [1;32m---> 60[1;33m self.binary, timeout), [0m[0;32m 61[0m desired_capabilities=capabilities) [0;32m 62[0m [0mself[0m[1;33m.[0m[0m_is_remote[0m [1;33m=[0m [0mFalse[0m[1;33m[0m[0m [1;32m/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/extension_connection.pyc[0m in [0;36m__init__[1;34m(self, host, firefox_profile, firefox_binary, timeout)[0m [0;32m 45[0m [0mself[0m[1;33m.[0m[0mprofile[0m[1;33m.[0m[0madd_extension[0m[1;33m([0m[1;33m)[0m[1;33m[0m[0m [0;32m 46[0m [1;33m[0m[0m [1;32m---> 47[1;33m [0mself[0m[1;33m.[0m[0mbinary[0m[1;33m.[0m[0mlaunch_browser[0m[1;33m([0m[0mself[0m[1;33m.[0m[0mprofile[0m[1;33m)[0m[1;33m[0m[0m [0m[0;32m 48[0m [0m_URL[0m [1;33m=[0m [1;34m"http://%s:%d/hub"[0m [1;33m%[0m [1;33m([0m[0mHOST[0m[1;33m,[0m [0mPORT[0m[1;33m)[0m[1;33m[0m[0m [0;32m 49[0m RemoteConnection.__init__( [1;32m/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.pyc[0m in [0;36mlaunch_browser[1;34m(self, profile)[0m [0;32m 59[0m [1;33m[0m[0m [0;32m 60[0m [0mself[0m[1;33m.[0m[0m_start_from_profile_path[0m[1;33m([0m[0mself[0m[1;33m.[0m[0mprofile[0m[1;33m.[0m[0mpath[0m[1;33m)[0m[1;33m[0m[0m [1;32m---> 61[1;33m [0mself[0m[1;33m.[0m[0m_wait_until_connectable[0m[1;33m([0m[1;33m)[0m[1;33m[0m[0m [0m[0;32m 62[0m [1;33m[0m[0m [0;32m 63[0m [1;32mdef[0m [0mkill[0m[1;33m([0m[0mself[0m[1;33m)[0m[1;33m:[0m[1;33m[0m[0m [1;32m/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.pyc[0m in [0;36m_wait_until_connectable[1;34m(self)[0m [0;32m 98[0m raise WebDriverException("The browser appears to have exited " [0;32m 99[0m [1;34m"before we could connect. The output was: %s"[0m [1;33m%[0m[1;33m[0m[0m [1;32m--> 100[1;33m self._get_firefox_output()) [0m[0;32m 101[0m [1;32mif[0m [0mcount[0m [1;33m==[0m [1;36m30[0m[1;33m:[0m[1;33m[0m[0m [0;32m 102[0m [0mself[0m[1;33m.[0m[0mkill[0m[1;33m([0m[1;33m)[0m[1;33m[0m[0m [1;31mWebDriverException[0m: Message: 'The browser appears to have exited before we could connect. The output was: Error: no display specified\n'