How to move to the next page on Python Selenium?

Question

I am trying to build a proxy scraper for a specific site, but I'm failing on move to next page.

This is the code that I'm using.

If you answer my question, please, explain me a bit about what you used and if you can, please, if there are any good tutorials for over this kind of code, provide me some:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time

options = Options()
#options.headless = True     #for headless
#options.add_argument('--disable-gpu') #for headless and os win

driver = webdriver.Chrome(options=options)

driver.get("https://hidemyna.me/en/proxy-list/")
time.sleep(10) #bypass cloudflare


tbody = driver.find_element_by_tag_name("tbody")
cell = tbody.find_elements_by_tag_name("tr")

for column in cell:
    column = column.text.split(" ")
    print (column[0]+":"+ column[1]) #ip and port

nxt = driver.find_element_by_class_name('arrow_right')
nxt.click()

Solution

To move to the next page you can try the following solution:

• Code Block:

  from selenium import webdriver
  from selenium.webdriver.chrome.options import Options
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.common.exceptions import TimeoutException, WebDriverException
  
  options = Options()
  options.add_argument("start-maximized")
  options.add_argument("disable-infobars")
  options.add_argument("--disable-extensions")
  driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
  driver.get('https://hidemyna.me/en/proxy-list/')
  while True:
      try:
          driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//li[@class='arrow__right']/a"))))
          driver.find_element_by_xpath("//li[@class='arrow__right']/a").click()
          print("Navigating to Next Page")
      except (TimeoutException, WebDriverException) as e:
          print("Last page reached")
          break
  driver.quit()
  

• Console Output:

  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  Navigating to Next Page
  .
  .
  .
  Navigating to Next Page
  Last page reached
  

Comments

[Andersson]

Try nxt = driver.find_element_by_css_selector('.arrow__right>a'). Note that there are two underscores in class name

[JeffC]

Post the relevant HTML here.

[undetected Selenium]

@CodeIt Can you explain why this will not work on page 2?

[Andersson]

@DebanjanB , on second page your selector will match Previous button instead of Next

[Haunter]

tried the above but it is not working on this site. the error output is selenium.common.exceptions.WebDriverException: Message: unknown error: Element <a href="/en/proxy-list/?start=64#list"></a> is not clickable at point (874, 563). Other element would receive the click: <jdiv class="hoverl_6R"></jdiv>

[undetected Selenium]

@Haunter Checkout my answer update and let me know the status.