How to interact with the elements within #shadow-root (open) while Clearing Browsing Data of Chrome Browser using cssSelector

Question

I had been following the discussion How to automate shadow DOM elements using selenium? to work with #shadow-root (open) elements.

While in the process of locating the Clear data button within the Clear browsing data popup, which appears while accessing the url chrome://settings/clearBrowserData through Selenium I am unable to locate the following element:

#shadow-root (open)
<settings-privacy-page>

Snapshot:

Using Selenium following are my code trials and the associated errors encountered:

• Attempt 1:

  WebElement root5 = shadow_root4.findElement(By.tagName("settings-privacy-page"));
  

  • Error:

    Exception in thread "main" org.openqa.selenium.JavascriptException: javascript error: b.getElementsByTagName is not a function
    

• Attempt 2:

  WebElement root5 = shadow_root4.findElement(By.cssSelector("settings-privacy-page"));
  

  • Error:

    Exception in thread "main" org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"css selector","selector":"settings-privacy-page"}
    

• Attempt 3:

  WebElement root5 = (WebElement)((JavascriptExecutor)shadow_root4).executeScript("return document.getElementsByTagName('settings-privacy-page')[0]");
  

  • Error:

    Exception in thread "main" java.lang.ClassCastException: org.openqa.selenium.remote.RemoteWebElement cannot be cast to org.openqa.selenium.JavascriptExecutor
    

Incase if it is helpful the initial code block (till the above line) works perfect:

driver.get("chrome://settings/clearBrowserData");
WebElement root1 = driver.findElement(By.tagName("settings-ui"));
WebElement shadow_root1 = expand_shadow_element(root1);

WebElement root2 = shadow_root1.findElement(By.cssSelector("settings-main#main"));
WebElement shadow_root2 = expand_shadow_element(root2);

WebElement root3 = shadow_root2.findElement(By.cssSelector("settings-basic-page[role='main']"));
WebElement shadow_root3 = expand_shadow_element(root3);

WebElement root4 = shadow_root3.findElement(By.cssSelector("settings-section[page-title='Privacy and security']"));
WebElement shadow_root4 = expand_shadow_element(root4);

PS: expand_shadow_element() works flawless.

Solution 1

If you are trying to get 'Clear Data' element then you can use the below js to get the element and then perform.

return document.querySelector('settings-ui').shadowRoot.querySelector('settings-main').shadowRoot.querySelector('settings-basic-page').shadowRoot.querySelector('settings-section > settings-privacy-page').shadowRoot.querySelector('settings-clear-browsing-data-dialog').shadowRoot.querySelector('#clearBrowsingDataDialog').querySelector('#clearBrowsingDataConfirm')

Here is the sample script.

driver.get("chrome://settings/clearBrowserData");
driver.manage().window().maximize();
JavascriptExecutor js = (JavascriptExecutor) driver; 
WebElement clearData = (WebElement) js.executeScript("return document.querySelector('settings-ui').shadowRoot.querySelector('settings-main').shadowRoot.querySelector('settings-basic-page').shadowRoot.querySelector('settings-section > settings-privacy-page').shadowRoot.querySelector('settings-clear-browsing-data-dialog').shadowRoot.querySelector('#clearBrowsingDataDialog').querySelector('#clearBrowsingDataConfirm')");
// now you can click on clear data button
clearData.click();

Edit 2: Explanation

Problem: Selenium does not provide explicit support to work with Shadow DOM elements, as they are not in the current dom. That's the reason why we will get NoSuchElementException exception when try to access the elements in the shadow dom.

Shadow DOM:

Note: We will be referring to the terms shown in the picture. So please go through the picture for better understanding.

Solution:

In order to work with shadow element first we have to find the shadow host to which the shadow dom is attached. Here is the simple method to get the shadow root based on the shadowHost.

private static WebElement getShadowRoot(WebDriver driver,WebElement shadowHost) {
    JavascriptExecutor js = (JavascriptExecutor) driver;
    return (WebElement) js.executeScript("return arguments[0].shadowRoot", shadowHost);
}

And then you can access the shadow tree element using the shadowRoot Element.

// get the shadowHost in the original dom using findElement
WebElement shadowHost = driver.findElement(By.cssSelector("shadowHost_CSS"));
// get the shadow root
WebElement shadowRoot = getShadowRoot(driver,shadowHost);
// access shadow tree element
WebElement shadowTreeElement = shadowRoot.findElement(By.cssSelector("shadow_tree_element_css"));

In order to simplify all the above steps created the below method.

public static WebElement getShadowElement(WebDriver driver,WebElement shadowHost, String cssOfShadowElement) {
    WebElement shardowRoot = getShadowRoot(driver, shadowHost);
    return shardowRoot.findElement(By.cssSelector(cssOfShadowElement));
}

Now you can get the shadowTree Element with single method call

WebElement shadowHost = driver.findElement(By.cssSelector("shadowHost_CSS_Goes_here));
WebElement shadowTreeElement = getShadowElement(driver,shadowHost,"shadow_tree_element_css");

And perform the operations as usual like .click(), .getText().

shadowTreeElement.click()

This Looks simple when you have only one level of shadow DOM. But here, in this case we have multiple levels of shadow doms. So we have to access the element by reaching each shadow host and root.

Below is the snippet using the methods that mentioned above (getShadowElement and getShadowRoot)

// Locate shadowHost on the current dom
WebElement shadowHostL1 = driver.findElement(By.cssSelector("settings-ui"));

// now locate the shadowElement by traversing all shadow levels
WebElement shadowElementL1 = getShadowElement(driver, shadowHostL1, "settings-main");
WebElement shadowElementL2 = getShadowElement(driver, shadowElementL1,"settings-basic-page");
WebElement shadowElementL3 = getShadowElement(driver, shadowElementL2,"settings-section > settings-privacy-page");
WebElement shadowElementL4 = getShadowElement(driver, shadowElementL3,"settings-clear-browsing-data-dialog");
WebElement shadowElementL5 = getShadowElement(driver, shadowElementL4,"#clearBrowsingDataDialog");
WebElement clearData = shadowElementL5.findElement(By.cssSelector("#clearBrowsingDataConfirm"));
System.out.println(clearData.getText());
clearData.click();

You can achieve all the above steps in single js call as at mentioned at the beginning of the answer (added below just to reduce the confusion).

WebElement clearData = (WebElement) js.executeScript("return document.querySelector('settings-ui').shadowRoot.querySelector('settings-main').shadowRoot.querySelector('settings-basic-page').shadowRoot.querySelector('settings-section > settings-privacy-page').shadowRoot.querySelector('settings-clear-browsing-data-dialog').shadowRoot.querySelector('#clearBrowsingDataDialog').querySelector('#clearBrowsingDataConfirm')");

Screenshot:

Solution 2

I had to do a similar test which required clearing browsing the chrome history. A minor difference was that I was clearing the data after going to the advanced section of the pop-up. As you are struggling to click only the "Clear data" button, I'm quite sure that you've missed one or two hierarchy elements mistakenly. Or got confused between sibling and parent elements probably. As per seeing your code, I assume that you already know that to access a particular shadow DOM element you need proper sequencing and it has been explained also quite nicely above.
Coming right at your problem now, here is my code snippet which is working correctly. The code waits until the data is cleaned and then will proceed to your next action-

public WebElement expandRootElement(WebElement element) {
WebElement ele = (WebElement) ((JavascriptExecutor) driver).executeScript("return arguments[0].shadowRoot",
        element);
return ele;
}

public void clearBrowsingHistory() throws Exception {

    WebDriverWait wait = new WebDriverWait(driver, 15);
    driver.get("chrome://settings/clearBrowserData");
// Get shadow root elements
    WebElement shadowRoot1 = expandRootElement(driver.findElement(By.xpath("/html/body/settings-ui")));

    WebElement root2 = shadowRoot1.findElement(By.cssSelector("settings-main"));
    WebElement shadowRoot2 = expandRootElement(root2);

    WebElement root3 = shadowRoot2.findElement(By.cssSelector("settings-basic-page"));
    WebElement shadowRoot3 = expandRootElement(root3);

    WebElement root4 = shadowRoot3
        .findElement(By.cssSelector("#advancedPage > settings-section > settings-privacy-page"));
    WebElement shadowRoot4 = expandRootElement(root4);

    WebElement root5 = shadowRoot4.findElement(By.cssSelector("settings-clear-browsing-data-dialog"));
    WebElement shadowRoot5 = expandRootElement(root5);

    WebElement root6 = shadowRoot5
        .findElement(By.cssSelector("cr-dialog div[slot ='button-container'] #clearBrowsingDataConfirm"));

    root6.click();
    wait.until(ExpectedConditions.invisibilityOf(root6));
}

It should work properly in your case too if you don't intend to change any of the options selected by default in the pop-up (In that case, you will have to add a few more codes regarding selecting those checkboxes). Please tell me if this solves your issue. Hope this is helpful I've added a snapshot of the the screen here too- image

Comments

[supputuri]

hmm. Can you try in the chrome console to check the js is correct. I am able to run the snippet without any error from eclipse. Btw, I added the chrome console screenshot for reference.

[supputuri]

@DebanjanB Added detailed explanation. Please let me know if you need any clarifications.

[supputuri]

Btw we can also add the Expected Conditions to make sure the shadowHost/shadowRoot/shadowElement are present before moving to the next step.

[undetected Selenium]

A couple of questions: While traversing from one #shadow-root (open) to another you have used document.querySelector('element').shadowRoot.querySelector('‌​element'), no issues with that. But when traversing from settings-section to settings-privacy-page you haven't used .shadowRoot.querySelector('element') instead you have used .shadowRoot.querySelector('settings-section > settings-privacy-page'). Can you help me out with the reason please?

[supputuri]

settings-privacy-page is part of the same shadow-tree where settings-section is and settings-section is the parent. Let me update the screenshot. Also updated the screenshot to point the right 4th shadow root.

[undetected Selenium]

Sounds great. One last question, while traversing through the last #shadow-root (open) from #clearBrowsingDataDialog why skiping the shadowRoot() method and directly invoking querySelector() worked?

[supputuri]

The Clear Data button is directly under #clearBrowsingDataDialog but not the last shadow-root marked in the screenshot.

[supputuri]

Updated the screenshot for more clarification and highlighted the part that helps finding the shadow element path.

[supputuri]

Are you still getting org.openqa.selenium.JavascriptException: javascript error: Cannot read property 'shadowRoot' of null message when run from selenium. This happens when you try to access the shadowRoot when it's still not loaded. If you want I can provide the logic to wait_for_shadow_root method to make sure the shadowroot is there before accessing the clear data button.

[grantr]

The accepted answer no longer works, here's an update: clearData = driver.execute_script('''return document.querySelector('settings-ui').shadowRoot.querySelect‌​or('#container').que‌​rySelector('settings‌​-main').shadowRoot.q‌​uerySelector('settin‌​gs-basic-page').shad‌​owRoot.querySelector‌​('#basicPage').query‌​Selector('settings-s‌​ection > settings-privacy-page').shadowRoot.querySelector('settings-c‌​lear-browsing-data-d‌​ialog').shadowRoot.q‌​uerySelector('#clear‌​BrowsingDataDialog')‌​.querySelector('#cle‌​arBrowsingDataConfir‌​m');''')