Difference between automation testing tool and automation framework

Question

I am trying to understand the difference between automation frameworks and automation testing tools. According to Test Automation in wikipedia a tool is dependent on the environment whereas a framework provides the structure for running the tool.

However, we have White which interacts with windows applications, then Selenium which works on browsers for web based applications. We call both of these "frameworks" but they both need to be run using a testing framework like NUnit or JUnit. So I feel they should be called tools because of this.

How exactly do we differentiate between an automation testing tool and an automation framework?

Solution 1

I now have better working experience on this and feel I can answer this myself. Basically a framework:

• provides a list of common library of functions,
• generates logs,
• provides the use of config files to make typical setting changes,
• and separates user test data from the code so that we are not hardcoding the actual test code,
• serves as a layer between the actual testing code and the raw language code. This allows makes maintaining the actual testing code much easier and simpler so that the engineer can make changes/fixes much more efficiently,
• separates the raw code from the testing code so that if general changes occur in the product then only selected functions can be modified instead of having it affect all the tests.

Now, in my original query, White was indeed a framework.

1. The reason is that if we try to "run" White it won't do anything by itself. It is just providing us some functions that we can call and get the job done.
   
   
2. I mentioned that

but they both need to be run using a testing framework

thus implying that White needs to use NUnit or it is useless. I was wrong. White need NOT use a testing framework necessarily. We can also write basic procedural code that makes use of only white and nothing else and still it could do some meaninful task.

The testing framework that we added (NUnit) is only used to accomplish another separate goal (to keep track of different parts of the code (called tests) and to generate a final report of which tests passed/failed.

3. I also mentioned that

but they both need to be run using a testing framework like NUnit or JUnit. So I feel they should be called tools because of this.

Here the misconception is the idea that "if it's using a framework then it must be a tool". The correct idea is that a framework can allow us to do some task and it can still use another framework to allow us to do even more tasks. For example, I'm using White framework to automate the windows dialogs. Now I add Log4j framework to add functionality for my own logs. Next I add a unit testing framework so that we can run the code (which uses both the previous frameworks) in the form of tests and generate results. Next, I add PNUnit framework and can now run the tests in parallel. See the idea?

Now the tools - these are nothing but the utilities that we are using while using the actual code - the code that will make use of all the above frameworks to do some actual worthwhile task. It could be the IDE or the platform which allows executing the code.

Solution 2

Automation tool is a platform to perform automation action on an application. Automation tool is a generic tool that can implement any types of test cases. Some automation tool will not have an option to separate test script or module or select multiple test scripts for execution.

Automation framework is an addon to the automation tool. Automation framework will do nothing without the automation tool which is created on top of automation tool. We are creating set of roles and standards of test execution like test cases selection, execution flow, report generation, test management tool integration etc.

• We can do automation only with Tool but not possible to perform automation only with automation framework
• Automation framework is depends on the organisation standard, and project structure, so we can create an automation framework i f tool itself is providing a framework (like UFT, Guage, etc.)
• We can create new features which is not support by the tool or club multiple tools with single automation framework like Selenium and Sikuli
• Automation Freemason will provide easy maintenance

Solution 3

In a "Tool" you use "Framework" to link them up :)

Solution 4

A tool would do the actual testing for you, given the right inputs. Whereas a framework gives you the necessary help to do the testing. Like in the case of White, it just helps you get controls and provide mouse clicks and stuff. It doesn't actually run anything on its own. Nunit is another framework at a slightly higher level, you run your test cases, it just tells you whether it passed or not.

An example for a tool can be a defragmenter, it actually performs the action.

Just tying to explain my point of view.. Hope it helps!

Solution 5

I have been using automation testing tools and have developed automation frameworks, giving you the details based on my experience -

Automation testing Tool -

Its a utility which can have features, functions and capabilities to operate on a specific or variety of Application under test (AUT)- such as - it can search and uniquely identify AUT objects, panes, button, links etc.

the underlying Libraries, utilities, methods and object identifying model work as a core framework for tool, which provides features, functions and capabilities to operate it. But this underlying core system is mostly propitiatory /static code and cant be changed as per custom needs.

most of the market standard tools focus on their object recognition model with a vast list technologies like .NET, JAva, Struts, etc. but may not be equally strong or capable in -

Creating test scripts / scenarios

Data input and usage of data files such as text, excel, Json, XML etc.

Managing scripts execution with a Basic result reporting capabilities.

most of the time the lack the customization and adaptation option - as every project or work has different need.

Automation framework -

Framework are as a word means - set of rules or best practices. even though a Automated testing tool has a features and capabilities, it requires a set of rules and driver to initiate, manage and report a complete test execution.

A Framework can be a set of scripts ( libs, jars, methods etc.) which are developed with a customized and tailored client / project / work needs. It can use the core / best features and functionalities of automation tool's core functionalities which then wrapped with the framework - to create a -

more seamless multi script and platform execution.

test data management - creation, input and reset if requires.

test orchestration - what triggers the build, where the inputs comes from, what all to execute, where to execute, how to integrate, who / how to control execution, where to reporting and whom to alert.

Hope this give a bit more prospect to this question.

View more solutions

Comments

[Mugen]

When we say white helps us to click around an application isn't it finally a tool? I feel it is the same thing for Selenium also. It helps you click around in your browser, enter characters, press keystrokes, read data etc. Combine with with Nunit and you can create a working project with them.