How to capture network traffic with selenium
Solution 1
Browsermob is the right way.
I must understand how browsermob works and tor too. For Tor you must enable the HTTPTunnelPort configuration like this.
tor --HTTPTunnelPort 8088
And configure browsermob to use it.
proxy_params = {'httpProxy': 'localhost:8088', 'httpsProxy': 'localhost:8088'}
proxy_b = server.create_proxy(params=proxy_params)
Thanks.
Solution 2
You can use a proxy to catch the network traffic. browsermob-proxy works well with selenium in Python. You need to download browsermob executable before. This is the piece of code with Firefox :
from browsermobproxy import Server
from selenium import webdriver
server = Server('path_to_executable')
server.start()
proxy = server.create_proxy()
profile = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)
proxy.new_har("file_name", options={'captureHeaders': True, 'captureContent': True})
driver.get("your_url")
proxy.wait_for_traffic_to_stop(1, 60)
for ent in proxy.har['log']['entries']:
print(ent)
server.stop()
driver.quit()
Solution 3
Python has a package called selenium-wire. You can use that package to capture the network traffics and also validate them. selenium-wire is an extended version of selenium will all the capabilities of selenium along with extra API to capture the network and validate. following is a link of an article https://sensoumya94.medium.com/validate-network-requests-using-selenium-and-python-3da5be112f7b
following is the repository of the package
https://github.com/wkeeling/selenium-wire
Sample code -
from seleniumwire import webdriver # Import from seleniumwire
# Create a new instance of the Firefox driver
driver = webdriver.Firefox()
# Go to the Google home page
driver.get('https://www.google.com')
# Access requests via the `requests` attribute
for request in driver.requests:
if request.response:
print(
request.url,
request.response.status_code,
request.response.headers['Content-Type']
)
Related videos on Youtube
Neeko
Updated on June 04, 2022Comments
-
Neeko almost 2 years
I'm starting a new Django project, I'm trying to capture the network traffic with Selenium.
I have already achieved this objective with Selenium-wire (MITM Proxy), but Django doesn't like to work with selenium-wire ( must start the server with "--nothreading --noreload", connection bug... ). I'm looking for achieve this with modern solutions, like parsing the network devtools of firefox directly or with a firefox addons. I'm using Firefox Geckodriver for my test.
for x in range(0, 10): profile = profile_firefox() options = options_firefox() driver = webdriver.Firefox(firefox_profile=profile, options=options, executable_path='/Users/*****/Desktop/selenium-master/headless_browser/geckodriver') try: driver.set_window_position(0, 0) driver.set_window_size(randint(1024, 2060), randint(1024, 4100)) time.sleep(randint(3,10)) driver.get(url) wait = ui.WebDriverWait(driver, 10) time.sleep(randint(8,10)) if driver.find_element_by_xpath("//*[@id=\"container\"]/main/div/div/div[1]/div[2]/form/div/div[2]/div[1]/button"): driver.find_element_by_xpath("//*[@id=\"container\"]/main/div/div/div[1]/div[2]/form/div/div[2]/div[1]/button").click() del driver.requests time.sleep(randint(8,10)) driver.find_element_by_xpath("//*[@id=\"container\"]/main/div/div/div[1]/div[2]/form/div/div[2]/div[1]/button").click() time.sleep(randint(10,20)) for request in driver.requests: if request.path == "https://api.*********.**/*******/*******": request_api = request raw = str(request_api.body) request_api = raw.split(('b\'')) payload_raw = request_api[1] payload = payload_raw[:-1] if payload: header = request.headers time.sleep(8) break except: print("Houston on a eu un probleme") firefox_closing(driver)
Edit :
def profile_firefox(): profile = FirefoxProfile() profile.set_preference('permissions.default.image', 2) profile.set_preference('dom.ipc.plugins.enabled.libflashplayer.so', 'false') profile.set_preference("general.useragent.override", firefox_init()) profile.set_preference('network.proxy.type', 1) profile.set_preference('network.proxy.socks', 'localhost') profile.set_preference('network.proxy.socks_port', 9050) profile.set_preference("network.proxy.socks_remote_dns", False) profile.set_preference("driver.privatebrowsing.autostart", True) profile.update_preferences() return profile
Test 2 with Socks,HTTP,SSL configuration :
server = Server('/Users/*****/Desktop/selenium-master/browsermob-proxy-2.1.4/bin/browsermob-proxy') server.start() proxy = server.create_proxy() proxy.selenium_proxy()#Dont understand what it does ??? port = int(proxy.port) profile = FirefoxProfile() profile.set_preference('permissions.default.image', 2) profile.set_preference('dom.ipc.plugins.enabled.libflashplayer.so', 'false') profile.set_preference('general.useragent.override', firefox_init()) profile.set_preference('network.proxy.type', 1) profile.set_preference('network.proxy.socks', 'localhost') profile.set_preference('network.proxy.socks_port', 9050) profile.set_preference('network.proxy.ssl', 'localhost') profile.set_preference('network.proxy.ssl_port', port) profile.set_preference('network.proxy.http', 'localhost') profile.set_preference('network.proxy.http_port', port) profile.set_preference('network.proxy.socks_remote_dns', False) profile.set_preference('driver.privatebrowsing.autostart', True) profile.update_preferences()
It seems Http proxy override the socks configuration...
Thanks a lot if you have any clue or advice about my code or solutions.
-
Neeko almost 5 yearsThanks for your answer. I have edited my first post, I'm using Tor to rotate my IP through the 9050 port, in my Firefox profile. Is it possible to combine your solution with my configuration ?
-
F Blanchet almost 5 yearsI don't know if it is compatible I never used Tor with this. Maybe there will be some port/host redirections to do between the proxies.
-
Neeko almost 5 yearsI'm stuck, when I set manually in the profile configuration a socks,http,ssl proxy the socks proxy doesn't work.... I'm looking for a solution like this one, if profile configuration can't work link
-
ebeland almost 5 yearsThe BrowserMob Proxy is no longer maintained since 2016. I'd suggest using the BrowserUp proxy fork at github.com/browserup/browserup-proxy. It is a drop-in replacement and works with the python browsermob client.