python selenium find child elements in loop

10,175

The XPath expression in the loop has to start with a dot to be context-specific:

for article in article_elements:
    title = article.find_element_by_xpath('.//article/h2').text
    share_count = article.find_element_by_xpath('.//footer/div/a/span').text
    poinst = article.find_element_by_xpath('.//footer/div[2]/div[1]/div[3]').text
    meta_info_list.append({'title':title, 'share count':share_count, 'points':poinst})

As a side note, you can shorten the code by using a list comprehension:

meta_info_list = [{
    'title': article.find_element_by_xpath('.//article/h2').text,
    'share count': article.find_element_by_xpath('.//footer/div/a/span').text,
    'points': article.find_element_by_xpath('.//footer/div[2]/div[1]/div[3]').text
} for article in article_elements]
Share:
10,175
Konstantin Rusanov
Author by

Konstantin Rusanov

Updated on June 04, 2022

Comments

  • Konstantin Rusanov
    Konstantin Rusanov almost 2 years

    I need to parse some child elements in all parent elements on page.

    Create list of all articles on page

    article_elements = driver.find_elements_by_tag_name('article')
    

    And after tying to get child elements in for loop and append all results to list

    for article in article_elements:
        title = article.find_element_by_xpath('//article/h2').text
        share_count = article.find_element_by_xpath('//footer/div/a/span').text
        poinst = article.find_element_by_xpath('//footer/div[2]/div[1]/div[3]').text
        meta_info_list.append({'title':title, 'share count':share_count, 'points':poinst})
    

    After loop ends i got 40 times one same article meta (of first article)

    {'share count': u'66', 'points': u'53 points', 'title': u'25+ Random Acts Of Genius Vandalism'}
    {'share count': u'66', 'points': u'53 points', 'title': u'25+ Random Acts Of Genius Vandalism'}
    {'share count': u'66', 'points': u'53 points', 'title': u'25+ Random Acts Of Genius Vandalism'}
    {'share count': u'66', 'points': u'53 points', 'title': u'25+ Random Acts Of Genius Vandalism'}
    ... 40 times
    

    My whole code

     # coding: utf8
    from selenium import webdriver
    from selenium.webdriver.common.keys import Keys
    import time
    
    driver = webdriver.Chrome()
    driver.set_window_size(1024,768)
    driver.get('http://www.boredpanda.com/')
    
    time.sleep(2)
    
    meta_info_list = []
    
    article_elements = driver.find_elements_by_tag_name('article')
    
    for article in article_elements:
        title = article.find_element_by_xpath('//article/h2').text
        share_count = article.find_element_by_xpath('//footer/div/a/span').text
        poinst = article.find_element_by_xpath('//footer/div[2]/div[1]/div[3]').text
        meta_info_list.append({'title':title, 'share count':share_count, 'points':poinst})
    
    for list in meta_info_list:
        print(list)