Cucumber/Capybara vs Selenium?

Question

The other day I was showing one of the testers at my company some tests I had written in cucumber (2 features, 5 scenarios). Then he asked me question that I could not answer:

How is this better than selenium or any other functionality test recording tool?

I understand that cucumber is a different technology and it's placed at a different level of testing, but I can't understand why I should bother to write and maintain Cucumber/Capybara tests.

Can someone give me a reasonable explanation for using Cucumber/Capybara instead of just Selenium?

Solution 1

This question is borderline asking for an opinion. Your question actually reads to me, "What tool is right for me?" I say this because you don't give a reason for why you chose Cucumber and Capybara. I believe to answer that tester's question, you need to answer a couple more questions first:

1.) What stage in the process are you going to be writing these tests?

Cucumber may not be the right choice for unit tests, depending on the language you're using. But it can be used for any level of testing, from unit to integration to end-user.

2.) Who is going to maintaining your tests? You? Other developers? Testers? Business Analysts? Project Managers?

Automated tests must be maintained, and knowing who will be doing that can help you decide on a tool - as some will be too technical for certain users.

3.) Who is going to be defining new tests?

Cucumber is meant to be used collaboratively between development, QA and business owners. It is the perfect tool for leveraging everyone's knowledge into the automated testing process. It requires the development of an ubiquitous language to be effect however. You can read up on that on James Shore's Art of Agile page.

Once you've answered these questions, you're ready to address the tester's question.

However, there are a couple of points to keep in mind when comparing recording tools (such as Selenium IDE, HP Quick Test Pro, IBM Rational Functional Tester) vs. development tools (nUnit, jUnit, RSpec, Selenium webdriver, Capybara) is that they are targeted towards different audiences. They also have different plusses and minuses.

Recording tools are easy for anyone to use, but the scripts they create are fragile. They break easily and require more maintenance. They are great for one-off automated testing, where you need to get it done quickly and have non-technical manpower.

Development tools have a larger learning curve and require programming (or at the least scripting) experience. The scripts are generally more robust, but require more technical knowledge to maintain. They are a good solution when you want repeatability and plan to use tests for a long time.

I strongly suggest you read The Cucumber Book. It will really help you decide if Cucumber is the right choice for you.

Solution 2

Cucumber isn't only a testing tool. Besides testing Cucumber features also take a role of documentation, a mechanism to collaborate with stakeholders and requirements storage (if you write them in declarative style).

You don't have to use Cucumber with Capybara. You can use selenium directly. But Capybara has the same high-level API for all supporting drivers. It's more high-level than Selenium's and allows to write tests a bit faster. You don't have to change code when you switch from one driver to another.

Tests built using test recording tools are generally much worse. Selenium IDE may produce valid programming code but it's not good-looking and thus quiet difficult to maintain.

Solution 3

Selenium ide is good for testing features that have mostly visual elements (links, text and etc.). But often web apps have features that don't represent itself as visual elements, like sending emails, queueing jobs, communicating with 3rd party services and etc. You could, for example, test if an 'Email has been sent' message is present after submitting a form. But it doesn't really tell you if an email is actually sent and therefore you aren't really testing the whole feature here.

Solution 4

Cucumber is tool used to make tests readable to business users. It consists of plain English sentences that are matched using regex to your Capybara steps.

Using recording tools won't do you any good in the long run. They were meant for beginners and aren't that powerful so I recommend you go straight to coding.

You can use Selenium alone for your tests, but I would recommend you continue to use Cucumber for documentation purposes, if you find them useful and easy to work with. After all, Cucumber can use Capybara or the Selenium web driver.

View more solutions

Comments

[Doug Noel]

Fair enough. But that also assumes that you can do both. As a QA engineer, I have no control on the coding practices of the developers. However I can leverage Cucumber much more effectively than Unit tests to promote change.

[treaz]

Are business users really interested in reading/writing tests? From my experience they just want things to work... Recording tools were meant for beginner -> I would anyone bother with learning cucumber/capybara if there are easier to use tools?

[treaz]

Actually, the question is: "Why would I use cucumber?"

[Andrei Botalov]

@treaz Cucumber "takes a role of documentation, a mechanism to collaborate with stakeholders and requirements storage"

[treaz]

@AndreyBotalov that's some hard to maintain documentation. However, I would assume that once the step definitions are defined it is a lot easier to write the scenarios, right? Does anyone have numbers relating to this topic?

[Andrei Botalov]

@treaz You write features first and only then you write step defintions

[Doug Noel]

treaz I think that's a valid question. I think @AndreyBotalov had a nice succinct answer. But once you know what Cucumber can do, you should still go back and ask yourself "What am I trying to accomplish with automation?" I've seen a lot of people pick a tool because they used it before, rather than because it was the right solution to their problem.

[gerky]

Yup, I agree that most if not all business users do not even look at the feature files, but that is what cucumber is for. While recording tools are great for newbies, learning a high level language such as capybara is extremely easy, you can get around with only a handful of methods, so why not?