When you begin to learn about test automation, you must come across the term “test automation framework”. Maybe some of you get uncomfortable with this term and begin to feel that it is something which is difficult to understand and even more difficult to implement.
This tutorial is written with the aim to help you understand test automation frameworks as simply as possible. Read all tutorials in this ‘Automation Testing Tutorials’ series here.
Test automation framework (in very simple language) is “set of rules.” Rules help us to write scripts in such a manner that result in “lower maintenance”.
To completely understand the concept of the framework, we first have to learn how we write simple scripts and then how to implement a framework on them.
In test automation, we write scripts. Scripting is basically about three ‘A’s:
- ARRANGEMENT
- ACTION
- ASSERTION
Below are the details of each A, with examples:
#1. ARRANGEMENT or Object Identification
We identify objects (buttons, dropdowns etc.) either by their ids, names or by their Window Titles etc.
In case of web application, we identify by user ID, or By XPath or By CSS or By Class Name etc. If nothing works, we then identify objects by using mouse coordinates (But it is not a reliable method of object identification)
Take this example of Selenium WebDriver (with C#) in which we identify objects using the id. (Web application)
IWebElement txtServer = _webDriver.FindElement(By.Id("tfsServerURL")); |
Another example from MS Coded UI (desktop application)
WinButton btnAdd = new WinButton(calWindow);btnAdd.SearchProperties[WinButton.PropertyNames.Name] = "Add"; |
After identification, we arrange or store these objects into UIMaps or Object Repository to reuse them in our scripts. That is why this step is called ARRANGEMENT.
#2. ACTION on the Identified Object
When the objects are identified, we perform some kind of actions on it either by mouse or by keyboard. For example, either we click, or we double-click, or we mouse hover over it or sometimes we drag-drop. Sometimes we write on text boxes. So any kind of action we perform on these objects are covered in this second step.
Example 1: (Selenium WebDriver with C#)
txtServer.Clear();txtServer.SendKeys(“Some sample text”); |
Example 2: (MS Coded UI with C#)
Mouse.Click(buttonAdd); |
#3. ASSERTION
The assertion is basically checking the object with some expected result. For example, if we press 2+3 on the calculator, the screen should show 5. In this case, our expected result is 5. This concept is already explained in our first tutorial.
Here we give an example of assertion:
Assert.AreEqual("5", txtResult.DisplayText); |
Nearly every script written in test automation contains these three things: Arrangement, Action and Assertion.
Now take a look at a complete script which contains all these steps. The script will open a calculator, press 1 + 6 and then check whether screen shows 7 or not.
Example A:
[TestMethod][TestMethod]public void TestCalculator(){ var app = ApplicationUnderTest.Launch("C:\\Windows\\System32\\calc.exe");//Object identification part (ARRANGEMENT) //----*Calculator Window----*//WinWindow calWindow = new WinWindow(app);calWindow.SearchProperties[WinWindow.PropertyNames.Name] = "Calculator";calWindow.SearchProperties[WinWindow.PropertyNames.ClassName] = "CalcFrame"; //----*Button1 ----*//WinButton btn1 = new WinButton(calWindow);btn1.SearchProperties[WinButton.PropertyNames.Name] = "1"; //----*Button Add ----*//WinButton btnAdd = new WinButton(calWindow);btnAdd.SearchProperties[WinButton.PropertyNames.Name] = "Add"; //----*Button 6 ----*//WinButton btn6 = new WinButton(calWindow);btn6.SearchProperties[WinButton.PropertyNames.Name] = "6"; //----*Button Equals ----*//WinButton btnEquals = new WinButton(calWindow);btnEquals.SearchProperties[WinButton.PropertyNames.Name] = "Equals"; //----*Text Box Results----*//WinText txtResult = new WinText(calWindow);txtResult.SearchProperties[WinText.PropertyNames.Name] = "Result"; //(ACTIONS Part)// Click '1' buttonMouse.Click(btn1); // Click 'Add' buttonMouse.Click(btnAdd); // Click '6' buttonMouse.Click(btn6); // Click 'Equals' buttonMouse.Click(btnEquals); //evaluate the results (ASSERTIONS)Assert.AreEqual("7", txtResult.DisplayText, “Screen is not displaying 7); //close the applicationapp.Close(); } |
What You Will Learn: [show]
What’s wrong with that script?
The script is easy to understand and I hope you get the concept of three ‘A’s in the above example. But all is not well with that script.
This script does not allow easy maintenance. Take the example of calculator again, if we have to write test cases of each function of the calculator, there will be many test cases. If there are 10 test cases and in each test, we have to define the same object, then if any change occurs in the name or id of the object, we have to change the object identification part in 10 test cases.
For example, take the example of button ADD in the script.
WinButton btnAdd = new WinButton(calWindow);btnAdd.SearchProperties[WinButton.PropertyNames.Name] = "Add"; |
Let’s say, this line is used in 10 test cases. Now in the next version of the calculator, the developer has changed the name of the button from “Add” to “Plus”. Now when we run our test cases, they will be failed and we have to change the above line to this in 10 test cases.
btnAdd.SearchProperties[WinButton.PropertyNames.Name] = "Plus"; |
So we have to improve this test case. We should follow the famous DRY principle in our coding. DRY stands for “Do not Repeat Yourself”. We should write the object identification part in such a manner that the object should be identified only in one place and should be called everywhere.
Take a look at the improved script.
Example B:
//defining the objects outside the script and only once.ApplicationUnderTest app = null;public WinWindow calWindow{ get {WinWindow _calWindow = new WinWindow(app);_calWindow.SearchProperties[WinWindow.PropertyNames.Name] = "Calculator";_calWindow.SearchProperties[WinWindow.PropertyNames.ClassName] = "CalcFrame";return _calWindow;}} public WinText txtResult{get{WinText _txtResult = new WinText(calWindow);_txtResult.SearchProperties[WinText.PropertyNames.Name] = "Result";return _txtResult;}} //making functions for similar kind of taskspublic void ClickButton(string BtnName){WinButton button = new WinButton(calWindow);button.SearchProperties[WinButton.PropertyNames.Name] = BtnName ;Mouse.Click(button);} public void AddTwoNumbers(string number1, string number2){ClickButton(number1);ClickButton("Add");ClickButton(number2);ClickButton("Equals");} //Test case becomes simple and easy to maintain.[TestMethod] public void TestCalculatorModular(){app = ApplicationUnderTest.Launch("C:\\Windows\\System32\\calc.exe"); //do all the operationsAddTwoNumbers("6", "1"); //evaluate the resultsAssert.AreEqual("7", txtResult.DisplayText, “screen is not displaying 7”); //close the applicationapp.Close(); } |
In the above example, we have separated the calWindow and txtResult objects and move them to the top so that they can be used across different test methods. We have defined them only once and we can use them in as many test cases as we want.
We have also created two functions. ClickButton() which accepts a button name and click on it and AddTwoNumbers() which takes any two number and add them using the click button function inside it.
The moment we start “improving” our code and making it reusable and maintainable, it means, we are making use of any automation framework. Now it gets interesting.
See Also => Why do we need the framework for test automation?
There are five popular frameworks in test automation:
- Linear
- Modularity
- Data Driven
- Keyword Driven
- Hybrid
We will now explain each framework with the help of its characteristics.
#1. Linear Framework:
Characteristics
- Everything related to a script is defined inside the scripts.
- Does not care about abstraction and code duplication
- Record and playback normally generate linear code
- Easy to get started
- Maintenance Nightmare.
By reading the above 5 characteristics of Linear Framework, we can easily relate our Example A to them. This example is basically using Linear framework, Ever thing related to a script is defined inside the script. The call window and TxtResult are defined inside the script. The script does not care about abstraction and code duplication. It is also a maintenance Nightmare as I have explained earlier.
So why should we use this framework?
This framework can be used in small-scale projects where there are not many UI screens. Also, when we use any automation tool for the first time, it normally generates code in Linear Form. So we can learn about what code is generated by Automation tool for specific actions. Other than these reasons, this framework should be avoided in your scripting.
=> See here the example of Linear and Keyword Framework with QTP example.
#2. Modularity Framework:
Characteristics
- The objects are defined once and reusable in all test methods.
- Small and to-the-point methods are created for individual functionalities
- The test case is the collection of these small methods and reusable objects
- This allows us to write maintainable code.
By Reading the above characteristics, we can relate our Example B to these characteristics. In that example, we have created an abstraction by moving the calWindow to the top and define it inside a property which can be used everywhere. We have created two small and independent functions called ClickButton() and AddTwoNumbers() . We combine these two small functions to create our final script which tests the “Add” functionality of calculator.
This results in easier maintenance. If any change occurs in the calculator UI, we have to change only in functions. Our scripts will remain intact. This framework is highly used in automation. The famous Page Object Framework (which is used with Selenium) is also a kind of modularity framework. We distribute the whole web application into separate pages. The buttons, dropdowns and checkboxes of each page are defined inside the class of that page. If any change occurs on the website, we have to change only in that page class and other pages remain intact. This results in better maintenance and easier readability of scripts.
The only downside of this framework is that it requires good Object Oriented concepts and strong development skills. If you have those, this framework is highly recommended.
#3. Data Driven Framework:
Characteristics:
- Test Data (input and output values) are separated from the script and stored in External Files. This could be a.CSV file, an Excel spreadsheet or a Database.
- When the script is executed, these values are picked from external files, stored in variables and replace the hard-coded values if present.
- Really useful in places where the same test case has to be run with different inputs.
Example C:
We want to run the add test case with three different inputs.
The data is
7 + 2 = 9
5 + 2 = 7
3 + 2 = 5
We stored this data (both input and output) in an external CSV file.
[DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", "|DataDirectory|\\data.csv", "data#csv", DataAccessMethod.<strong>Sequential</strong>), DeploymentItem("TestCalc\\data.csv"), TestMethod] public void TestCalculatorDataDrivsen(){app = ApplicationUnderTest.Launch("C:\\Windows\\System32\\calc.exe"); //do all the operationsAddTwoNumbers(FromCSV.ADD1, FromCSV.ADD2); //evaluate the resultsAssert.AreEqual(FromCSV.Sum, txtResult.DisplayText); //close the applicationapp.Close(); } |
In the above script, we define our data source on the top of the script, which is a .csv file.
We have given the path of that.CSV file and tell the script to parse it “Sequentially”. That means the script will run as many times as there are rows present in the CSV file. In our case, the script will run 3 times. In each run, it will add the two numbers defined in first two columns and verify that the sum of these two numbers matches the number present in the third column.
There are various advantages of this framework. All the values are stored outside the script, so if any change will occur in the next build, we just have to change the data in the external file and the script will remain intact.
The second advantage is that the same script can be run for different inputs. Take the example of an ERP in which you have to test the registration of 100 employees. You can write one script and store the names and other data related to employees in an external file. You will execute one script and it will run 100 times. Each time with different employee’s data. You can easily detect, on what data the script fails to register the employee. It will be an added advantage when you are doing negative testing.
=> See here the example of Data driven and Hybrid framework with QTP example.
#4. Keyword-Driven Framework:
Characteristics:
- Both data and actions are defined outside the script.
- It required the development of keywords for different types of actions.
- The functionality which we have to test is written in a step by step manner in tabular form using the keywords we develop and the test data. We store this table in external files just like data driven framework.
- The script will parse this table, and perform the corresponding actions.
- Allows manual tester who does not know about coding to be part of automation to some extent.
Example D:
We defined the data (e.g. 1 + 3 = 4) as well as actions (e.g. Click, Clear etc.) in an excel file in tabular form.
The script will become something like this (the below code is just written for understanding purpose)
[TestMethod]public void TestCalculator(){app = ApplicationUnderTest.Launch("C:\\Windows\\System32\\calc.exe"); Table tb = ReadFromExcel();Foreach(WinRow row in tb){ WinCell Window = row.Cells[“Window”];WinCell Control = row.Cells[“Control”];WinCell Action = row.Cells[“Action”];WinCell Arguments = row.Cells[“Arguments”];UITestControl c = GetControl(Control.Text,Window.Text); If(Action.Text == “Click”)Mouse.Click (c); If (Action.Text == “Clear”)c.Clear(); if(Action.Text == “Verify Result”)Assert.AreEqual(c.Text, Arguments.Text) //….and so on} } |
The above script is just a parser of the excel file. It parses the excel file line by line and looks for keywords to perform respective actions. If it finds the keyword “Click”, it will click on the defined object. If it finds “Verify Result”, it will perform the assertion.
There are various advantages of using the keyword driven framework.
The first advantage is that this framework is very helpful in those scenarios where there are great chances of changes in test cases. If any step changes in a test case, we don’t need to touch the code. We just have to update the excel file and script will be updated.
You can define all your scripts in an excel file and hand over this excel file to manual testers to add new scripts or update the existing ones. In this way, manual testers can also become the part of test automation because they do not need to code anything. They will just update this excel file when there are a need and scripts will automatically be updated.
The second advantage is that your script becomes tool independent. You can maintain your scripts in an excel file and if you need to change your automation tool at some point, you can easily change it by writing an excel parser in another tool.
The downside of this framework is that you need to invent keywords for various types of actions. In large-scale projects, there will be so many keywords that you need to remember and organize your scripts and keywords. This itself becomes a cumbersome task at a point.
In some complex scenarios, where objects cannot easily be identified and we need to use mouse coordinates and other techniques, this framework is not very helpful.
Keyword driven is still a favourite framework for many automation testers. Robot framework by Google is a popular keyword driven framework which is supported by an active community.
#5. Hybrid Test Automation Framework:
Characteristics:
- The combination of two or more of the above techniques, taking from their strengths and minimizing their weaknesses.
- The framework can use the modular approach along with either data-driven or keyword-driven framework.
- The framework can use scripts to perform some tasks that might be too difficult to implement in a pure keyword driven approach.
In simple words, Hybrid framework, use the combination of the above-mentioned techniques. We can use a data-driven framework which is also modular in nature. For some test cases, we can use keyword-driven approach and for remaining we can use modular. So whenever we mix two or more techniques mentioned in this article, we are actually using a Hybrid approach.
Conclusion
I hope that test automation framework is no longer a scary term for you now. I tried to explain the most popular frameworks as simply as possible.
Frameworks are here to make your life easier. They help you to write maintainable and reliable scripts. Without using frameworks, the test automation field is a nightmare. For every small change in the application, you have to change your code in hundreds of places.
So an understanding of these frameworks is a must for every tester who wants a taste of test automation.
In our next tutorial in this series, we will learn ‘Execution and reporting of Test Automation’.
