WatiN or Selenium?

Question

I'm going to start coding some automated tests of our presentation soon. It seems that everyone recommends WatiN and Selenium. Which do you prefer for automated testing of ASP.NET web forms? Which of these products work better for you?

As a side note, I noticed that WatiN 2.0 has been in CTP since March 2008, is that something to be concerned about?

Solution 1

Just want to say that I'm currently working hard on a beta release of WatiN 2.0 somewhere in Q1 of 2009. It will be a major upgrade to the current CTP 2.0 versions and will basically give you the same functionality to automate FireFox and IE as version 1.3.0 offers for automating IE.

So no concerns there.

Hope this helps in making your choice Jeroen van Menen Lead dev WatiN

Solution 2

If you're looking to make a serious long-term investment in a framework that will continue to be improved and supported by the community, Selenium is probably your best bet. For example, I just came across this info on Matt Raible's blog:

As of Friday, Google has over 50 teams running over 51K tests per day on internal Selenium Farm. 96% of these tests are handled by Selenium RC and the Farm machines correctly. The other 4% are partly due to RC bugs, partly to test errors, but isolating the cause can be difficult. Selenium has been adopted as the primary technology for functional testing of web applications within Google. That's the good news.

I also went to one of the Selenium meetups recently and learned that Google is putting serious resources into improving Selenium and integrating it with WebDriver, which is an automated testing tool developed by Simon Stewart. One of the major advantages of WebDriver is that it controls the browser itself rather than running inside the browser as a Javascript application, which means that major stumbling blocks like the "same origin" problem will no longer be an issue.

Solution 3

We've tested both and decided to go with WaTiN. As others have pointed out, Selenium does have some nice features not found in WaTiN, but we ran into issues getting Selenium working and once we did it was definitely slower when running tests than WaTiN. If I remember correctly, the setup issues we ran into stemmed from the fact that Selenium had a separate app to control the actual browser where WaTiN did everything in process.

Solution 4

I've been trying 'em both out and here are my initial thoughts...

---

WatiN

The Good

• Fast execution.
• Script creation tools are independent projects; there are 2 that I know of: Wax (Excel based, hosted on CodePlex) and WatiN Test Record (hosted on SourceForge). Neither is as robust as Selenium IDE.
• Very good IE support. Can attach and detach to/from running instances. Can access native window handles etc. (See script example below).
• NuGet packaged, easy to get running in .NET, Visual Studio style environments and keep updated.

The Bad

• Googling WatiN (watin xyz) often causes Google to recommend "watir xyz" instead. Not that much documentation out there.
• What little there is (documentation), it is confusing; for example: at first blush it would appear that there is no native support for CSS selectors. Especially since there are extensions libraries like 'WatiNCssSelectorExtensions' and many blog articles about alternative techniques (such as injecting jQuery/sizzle into the page). On Stack Overflow, I found a comment by Jeroen van Menen which suggests that there is native support. At least the lead-developer spends time on Stack Overflow :)
• No native XPath support.
• No out-of-the-box remote execution/grid based execution.

Script Example (C#). You can't do this with Selenium (not that I know off, at least):

class IEManager
{
    IE _ie = null;
    object _lock = new object();

    IE GetInstance(string UrlFragment)
    {
        lock (_lock)
        {
            if (_ie == null)
            {
                var instances = new IECollection(true);  //Find all existing IE instances
                var match = instances.FirstOrDefault(ie=>ie.Url.Contains(UrlFragment));
                _ie = match ?? new IE();
                if (match==null)  //we created a new instance, so we should clean it up when done!
                    _ie.AutoClose = true;
            }
        }

        return _ie;
    }
}

---

Selenium

• Slower than WatiN (especially since a new process has to be created).
• Built-in CSS selectors/XPath support.
• Selenium IDE is good (can't say great, but it’s the best in class!).
• Feels more Java-ish than .NET-ish...but really, it's programming language agnostic; all commands are sent to an out-of-process 'Driver'. The driver is really a 'host' process for the browser instance. All communication must be serialised in/out across process boundaries, which might explain the speed issues relative to WatiN.
• Decoupled processes - "Driver" and "Control" mean more robustness, more complexity, etc., but also easier to create grids/distributed test environments. Would have really liked it if the "distribution" mechanism (i.e. the communication between Driver & Control) were across WebSphere or other existing, robust, message queue manager.
• Support chrome and other browsers out of the box.

Despite everything, I went with WatiN in the end; I mainly intend to write small screen-scraping applications and want to use LINQPad for development. Attaching to a remote IE instance (one that I did not spawn myself) is a big plus. I can fiddle around in an existing instance...then run a bit of script...then fiddle again etc. This is harder to do with Selenium, though I suppose "pauses" could be embedded in the script during which time I could fiddle directly with the browser.

Solution 5

The biggest difference is that Selenium has support for different browsers (not just IE or FF, see http://seleniumhq.org/about/platforms.html#browsers.

Also, Selenium has a remote control server (http://seleniumhq.org/projects/remote-control/), which means that you don't need to run the browser on the same machine the test code is running. You can therefore test your Web app. on different OS platforms.

In general I would recommend using Selenium. I've used WatiN few years ago, but I wasn't satisfied with its stability (it has probably improved by now). The biggest plus for Selenium for me is the fact that you can test the Web app. on different browsers.

View more solutions

Comments

[Maxim Eliseev]

I do not think this question should be closed. It is useful for me and other developers (see upvotes). Such question is one of reasons why I need Stackoverflow. I wish I could downvote the admin's decisions.

[marcelo-ferraz]

I wonder why this question was closed. It is very constructive. I am studying both, and would like to know their diferences

[Ronald McDonald]

Not constructive??? ...this site is being overrun by idiots with too much control.

[marcumka]

"You will have to do both if you need to do IE and FF testing" - can't Selenium do both?

[redsquare]

selenium rc can do ie safari and ff

[Jeff Martin]

you can record only in FF but you can use the RC to control both IE and FF (and probably others)

[Jeff Martin]

i want to give a +1 for bringing up a 3rd alternative but then I say -1 for not providing any experiences with the product and how it compares to the other two.

[Igor Brejc]

Not "probably": seleniumhq.org/about/platforms.html#browsers.

[Jeremy McGee]

+1 for performance notes, and real-world usage.

[Peter Gfader]

I realized the same issues: #1 Performance is not so good and #1 the tests are running on a Java server (that needs to be set up in [TestSetup]).

[Tony Ennis]

Selenium has support for different browsers - Extra important given now we have to support Chrome, Safari, FF, and IE 6, 7, and 8.

[Igor Brejc]

This isn't an issue any longer - Selenium 2.0 comes with WebDriver library, which enables direct control of the browser, not just through a Java server.

[Henry99]

@jcollum Sorry, but I disagree to give a -1. Jeroen only responded to the second question "As a side note...". And who would be better qualified to respond that, if not the Lead Developer of the product. Only marking this response as best answer may be questionable.

[jcollum]

@Henry99 would've been more appropriate as a comment under the question or a separate question. The core question here is "A or B". The author of A or B shouldn't be responding to questions like that since it's pretty obvious they will be biased.

[jcollum]

Windows dialogs = js alert boxes? If so, selenium has support for that. I've never tested it since I find those alert boxes obnoxious.

[sonstabo]

Being able to have the QA team create 'manual' tests with the FF plugin and have developers port the generated C# tests our infrastructure made the conclusion to go for Selenium pretty easy. WaitIn looks fair enough - but the 'painstaking' process of building tests (according to the video on the WaitIn project page) was not an option for our customer in this case.

[jcollum]

@sonstabo: that's the direction I'm hoping to go. Someday when we have a QA department :puppydogeyes:

[Grinn]

@jcollum Jeroen made no mention of the quality of his product v.s. Selenium, or said anything that could be considered biased one direction or the other. Perhaps you didn't read second part of the question, but the lead developer on the project is without question absolutely the most qualified person to answer that question.

[jcollum]

@Grinn: I addressed that in my comment above yours, did you read it? He's not answering the main question (Watin or Selenium) he's addressing something that should've been in a separate question completely.

[John Doe]

do you know what version of FF is supported? I tried the IDE with FF6 and FF5 with no love. I expected no support for 6, but for 5 I was surprised.

[Oliver]

This answer needs MUCH MORE ATTENTION: Coypu is the missing link between YOU and automated browser testing! Amazing! Go look at it now! If you've ever struggled with Selenium (or maybe WatiN), trying to get AJAX or element finding right - Coypu is the answer to your prayers ;-)

[Oliver]

Thanks, @penderi, for adding more details :-)

[Sam]

Thanks for the detailed comparison.

[Piotr Owsiak]

I haven't tried Selenium, but I had issues with Watin. I had test suddenly stop for no apparent reason plus I've had COM errors being thrown randomly (at least I couldn't find any pattern).

[Piotr Owsiak]

Selenium seems like a more mature project at the moment plus the fact that Google is using it is a pretty solid recommendation (also I've tried Watin and had problems - never tried Selenium though)

[Mohsin Awan]

Nice Answer ...