Selenium clicking to next page until on last page

Question

I am trying to keep clicking to next page on this website, each time appending the table data to a csv file and then when I reach the last page, append the table data and break the while loop

Unfortunately, for some reason it keeps staying on the last page, and I have tried several different methods to catch the error

while True:
  try :
      WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.LINK_TEXT, 'Next'))).click()
  except :
      print("No more pages left")
      break
driver.quit()

I also tried this one:

try:
    link = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,'.pagination-next a')))
    driver.execute_script('arguments[0].scrollIntoView();', link)
    link.click()
except:
    keep_going = False

I've tried putting print statements in and it just keeps staying on the last page.

Here is the HTML for the first page/last page for the next button, I'm not sure if I could do something utilizing this: HTML for first page:

<li role="menuitem" ng-if="::directionLinks" ng-class="{disabled: noNext()||ngDisabled}" class="pagination-next" style=""><a href="" ng-click="selectPage(page + 1, $event)" ng-disabled="noNext()||ngDisabled" uib-tabindex-toggle="">Next</a></li>
    <a href="" ng-click="selectPage(page + 1, $event)" ng-disabled="noNext()||ngDisabled" uib-tabindex-toggle="">Next</a>
</li>

HTML for last page:

<li role="menuitem" ng-if="::directionLinks" ng-class="{disabled: noNext()||ngDisabled}" class="pagination-next disabled" style=""><a href="" ng-click="selectPage(page + 1, $event)" ng-disabled="noNext()||ngDisabled" uib-tabindex-toggle="" disabled="disabled" tabindex="-1">Next</a></li>
    <a href="" ng-click="selectPage(page + 1, $event)" ng-disabled="noNext()||ngDisabled" uib-tabindex-toggle="" disabled="disabled" tabindex="-1">Next</a>
</li>

Solution 1

The next_page_btn.index(0).click() wasn't working, but checking the len of next_page_btn worked to find if it was the last page, so I was able to do this.

while True:
    next_page_btn = driver.find_elements_by_xpath("//li[@class = 'pagination-next']/a")
    if len(next_page_btn) < 1:
        print("No more pages left")
        break
    else:
        WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.LINK_TEXT, 'Next'))).click()

Thanks so much for the help!

Solution 2

You can solve the problem as below,

Next Button Will be enabled until the Last Page and it will be disabled in the Last Page.

So, you can create two list to find the enabled button element and disabled button element. At any point of time,either enabled element list or disabled element list size will be one.So, If the element is disabled, then you can break the while loop else click on the next button.

I am not familiar with python syntax.So,You can convert the below java code and then use it.It will work for sure.

Code:

    boolean hasNextPage=true;

    while(hasNextPage){
        List<WebElement> enabled_next_page_btn=driver.findElements(By.xpath("//li[@class='pagination-next']/a"));
        List<WebElement> disabled_next_page_btn=driver.findElements(By.xpath("//li[@class='pagination-next disabled']/a"));

        //If the Next button is enabled/available, then enabled_next_page_btn size will be one.
        // So,you can perform the click action and then do the action
        if(enabled_next_page_btn.size()>0){
            enabled_next_page_btn.get(0).click();
            hasNextPage=true;
        }else if(disabled_next_page_btn.size()>0){
            System.out.println("No more Pages Available");
            break;
        }
    }

Solution 3

How about using a do/while loop and just check for the class "disabled" to be included in the attributes of the next button to exit out? (Excuse the syntax. I just threw this together and haven't tried it)

string classAttribute

try :

     do
     {
          IWebElement element = driver.findElement(By.LINK_TEXT("Next"))
          classAttribute = element.GetAttribute("class")
          element.click()
     }
     while(!classAttribute.contains("disabled"))

except :

     pass

driver.quit()

Solution 4

xPath to the button is:

//li[@class = 'pagination-next']/a

so every time you need to load next page you can click on this element:

next_page_btn = driver.find_elements_by_xpath("//li[@class = 'pagination-next']/a")
next_page_btn.index(0).click()

Note: you should add a logic:

while True:
    next_page_btn = driver.find_elements_by_xpath("//li[@class = 'pagination-next']/a")
    if len(next_page_btn) < 1:
        print("No more pages left")
        break
    else:
        # do stuff

View more solutions

Comments

[WIT]

converted this to python and I don't think it's working... adding some print statements to see what's going on... thank you for your help though