Regression tests - tools, approaches and when to use

When it comes to software testing, it’s rarely about testing only one version of it. Is it even possible to test the very first version of application which is just complete, functional and ready to be released into the world? I think that if you want to test something properly, you have to be a part of a developing team.

Development and testing

Development is a process and if I have to use just one word to describe it, it would be “changes”. Basically, the changes you are about to make in the software, can be minor or big (or even huge). It depends on complexity and difficulty. Sometimes it could mean how much this new task affects the others, that are already developed. Nowadays, we have so many programs, applications etc., that client’s expectations are growing high and therefore the application itself is more complicated. Many things can result from each other and that makes the dependence network very dense. We should always remember that the change we are introducing into the software may affect existing functionality. So, the tester role is not only to test the change, but to make sure that it didn’t “spoil” anything.

Regression

Regression testing is about re-executing tests in order to check whether the previous functionalities of the application is working fine. If not, that would be called a regression. Consequently, this type of testing is “regression testing”. There are also so-called re-tests. However, they are not the same as regression tests. The re-test is a confirmatory test that concern checking a specific error. The reported error is assigned to the developer who repairs it. Later, we have to check whether it has really been repaired and this action is called the re-test. It does not focus on general tests of the entire application like in case of regression.

Regression testing - advantages

What are the advantages of regression testing which make it so important and different from just re-testing?

• it ensures that any change or bug fix that is done does not impact the existing functionality of the product
• it makes sure that issues which are already fixed do not occur again
• it improves the quality of the product

Regression testing - when to use?

When to use regression testing? The simplest answer is always after a change. Firstly, before any testing, you should perform smoke tests. This type of tests allows you to make sure that the new version of the application is "testable" and whether it is worth start working with it. What next? It seems like you don’t have to perform regression tests after every change. Specially if the change seems irrelevant. But if you know anything about developing a software, you should already know, that some changes appear to others as simple, minor. The reality is that some changes can affect places one may not think of. In any new software growth, there may be a regress - something that was previously tested and worked properly could have been corrupted. That is why regression testing is important. Of course, we should not go to the extreme and after each change of graphics or text change, carry out a whole application’s regression tests. It helps if you know your software well.

Mind map

With each new release, you can list out or create a mind map (“mind map” shows relationships among pieces of the whole) for affected areas. For example, if you make change in the log-in functionality, try to recall all of the places that are connected with logging in or out. Don’t forget about collateral functions such as forgetting/changing a password.

How often to perform regression tests

Regression tests should be performed as frequently as possible. The more often regression testing can occur, the more issues can be discovered and resolved. That leads to the more stable product. However, the reality is that regression tests are extremely arduous and time-consuming. Sometimes the budget or little time is the cause that the full regression tests of the application can’t be performed. If it’s the case, the test cases should be selected based on:

• what errors have been corrected/what changes have been made
• which areas of the application these changes concern the most?
• what is the impact of the changes on other parts of the software?

Test automation

We know that the regression testing is valuable, one can even say essential. It also takes a lot of time. What can we do? Fortunately, regression testing is a good candidate for automation due to its repeatability. We do regression testing after (almost) every deployment, so it would make life easier to automate test cases instead of running them manually every time. Specially if we have hundreds of test cases, it’s better to create automation test scripts for the test cases which we do after every change. Most of the regression test tools are record and playback type. You will record the test cases and verify whether the expected results are coming or not. Some of the regression test tools are:

• Selenium
• Winrunner
• vTest
• QTP/UFT
• SahiPro
• Ranorex
• TestComplete
• Regression Tester
• Watir
• IBM Rational Functional Tester

Potential problems of automation

The automation of regression tests seems like a great idea, answer for everything, free of defects. But we should also be aware of problems related to test automation. The company may be having an unrealistic expectation about the automation. Firstly, it also takes time and could be very expensive at the beginning. Then, the cost of maintaining tests must be included in the cost of the product. In a situation where the application changes very dynamically and the tester have to constantly improve test scenarios and automated tests based on these scenarios, it may turn out that these tests are unprofitable. Another thing is a false sense of a good product. The automated tests are as good as the test cases designed for them and how they were automated. The fact that we have thousands of automated case test, which application always passes, does not mean that we have a great product. We also need to think it through when tests do not detect any errors.

Automation – conclusion

Remember that automation is only a supplement to manual tests. In case we have a badly defined test process or even a lack of it, we do not have good programming practices, no automation will help us.

What is good to remember?

• Regression testing is one of the most important tests performed in the software development process. Its aim is to control the quality and quickly detect potential software regress
• If you can’t do a whole package of regression tests, it is essential to do an impact analysis of the changes to identify areas of the software that have the highest probability of being affected by the change and that have the highest impact to users. That requires a good knowing of the application that you’re testing
• It is highly recommended to do regression testing very often, which is also very time-consuming. Good idea is to automate regression tests
• Once regression tests (specially automated tests) are written, they won’t work forever, remember about maintaining and adapting them to new versions of products