How to scroll down in Python Selenium step by step

Question

Hi guys I am new to Selenium and Python. I was just scraping the site pagalguy website. I know how to scroll down to the bottom of the page but what I need is to scroll down step by step so that the Selenium will click all the readmore buttons,but I don't know how to scroll down step by step like that so I hard coded it like following one

browser.execute_script("window.scrollTo(0,300);")
browser.find_element_by_link_text("Read More...").click()

browser.execute_script("window.scrollTo(300,600);")
browser.find_element_by_link_text("Read More...").click()

browser.execute_script("window.scrollTo(600,900);")
browser.find_element_by_link_text("Read More...").click()

browser.execute_script("window.scrollTo(900,1200);")
browser.find_element_by_link_text("Read More...").click()

browser.execute_script("window.scrollTo(1200,1500);")
browser.find_element_by_link_text("Read More...").click()

browser.execute_script("window.scrollTo(1500,1800);")
browser.find_element_by_link_text("Read More...").click()

browser.execute_script("window.scrollTo(1800,2100);")
browser.find_element_by_link_text("Read More...").click()

browser.execute_script("window.scrollTo(2100,2500);")
browser.find_element_by_link_text("Read More...").click()
it goes on .......

I tried automating it using a while loop but it resulted in error, the above one works but I want it short and looped so that I can use it for all the other pages with different page length.

initial_value = 0
next_value = 300
while next_value<300000: 
  browser.execute_script("window.scrollTo(initial_value,next_value);")
  browser.find_element_by_link_text("Read More...").click()
  initial_value=next_value
  next_value+=300

JavascriptException: Message: ReferenceError: initial_value is not defined

But I have defined the value, I think I explained what I am actually trying to do, I want to automatically scroll down and click all the readmore buttons and then I will get the full text content

Solution 1

Agree with answer of @Rahul Chawla.

But adding one change. You can try this one

driver = webdriver.Chrome()

read_mores = driver.find_elements_by_xpath('//a[text()="Read More..."]')

for read_more in read_mores:
    driver.execute_script("arguments[0].scrollIntoView();", read_more)
    driver.execute_script("$(arguments[0]).click();", read_more)

Solution 2

Use loop with javascript window.scrollBy(0, Y) method with coordinates selecting movement step and iterations number.

   for i in range(20): # adjust integer value for need
       # you can change right side number for scroll convenience or destination 
       driver.execute_script("window.scrollBy(0, 250)")
       # you can change time integer to float or remove
       time.sleep(1)

Solution 3

We can do this by finding all read more buttons using find_elements_by_xpath() and looping over them while scrolling them into view one by one.

driver = webdriver.Chrome()

read_mores = driver.find_elements_by_xpath('//a[text()="Read More..."]')

for read_more in read_mores:
    driver.execute_script("arguments[0].scrollIntoView();", read_more)
    read_more.click()
    # your code here

Comments

[venkat]

Thank you for the answer, it is scrolling down to all the readmore clickables but it is not clicking i dont no the reason and there is no error i even added time.sleep(2) between the scrollintoview and click() still it is not cliscking can you give any other suggestions

[venkat]

Thanks a lot it worked but i have a doubt why is there a $ in the script driver.execute_script("$(arguments[0]).click();", read_more) it throwed an error then i removed it and now it works

[Chanda Korat]

It's all about javascript coding conventions. You can refer this