Expert knowledge | June 13, 2019

Test automation – common myths

There are several myths regarding the implementation, maintenance, costs and types of tests that can be automated. Discussions on test automation have kept coming up during multiple IT conferences and meetups.The number of tools, and thereby the possibilities of using them, is increasing; however vague knowledge of test automation does not help to popularize such tests and therefore to deliver products of the highest quality. In the article I shall examine common misunderstandings around automated tests to clear up doubts in this area.

Test automation – how does it work?

Before I discuss the test automation process further, let’s see how such tests are created.

Firstly, we specify areas requiring automated testing, focusing on relevant issues from a business perspective. We set up the goal and terms depending on the tested system, bearing in mind one of the principles of testing: “Testing is context-dependent”.

Once the goal is set up, we proceed to technical aspects. We choose a testing tool, and create the testing environment and testing scripts. Having run tests, we analyze the results and verify whether we have managed to obtain business value as assumed earlier on.

Deterministic automated tests work in a quite simple way. First of all, we put the application in the initial state to test a desired action, for example how the application acts while being used by, say, 1000 users. The aim of the verification is to compare the assumed results with the actual state.

Test automation – myths

1.      Each test can be automated

According to one of the widespread myths, each test can be automated. Let’s think it over: is it really possible? As per the International Software Testing Qualifications Board – the software testing certification organization – the testing process is infinite, which is why we cannot forecast and test everything.

Test automation is still incomparable to efficiency tests, volume testing, and stress testing. Automated testing has its limitations, though. For example, it cannot be used for non-functional tests. These are related to our personal perception of esthetics, legibility and transparency. In the abovementioned area we are unable to define the initial requirements and the expected final results. Some of the aspects, for example the appearance and transparency of the application, are perceptible only to the human eye. The same thing applies to accessibility testing, which aims to determine whether users with disabilities can use the system with ease.

All in all, knowing that some of the tests cannot be automated, we understand that it is impossible to entirely replace manual testing with automated testing – as per the second common myth. There will always be areas requiring manual testing.

2.      Automated tests can replace manual testing

This misunderstanding brings to mind a common myth from the past, that one day television would entirely replace radio. Nowadays, as we can see, radio, television and the Internet coexist and complement each other. It works similarly when it comes to automated and manual tests.

The first ones are advantageous due to their repeatability. With appropriately implemented tests we can be sure that every test will be run in the same, replicable manner. Additionally, contrary to human nature, automatons do not need coffee, the weekend or sleep, so they can work perpetually. On the other hand, humans are better when it comes to intelligence, intuition and imagination, which are essential to running an effective test of the application.

The majority of tests related to functional and non-functional requirements can be performed manually. There will always be a need for manual tests, calling for the engagement and knowledge of a tester – namely usability tests, accessibility tests and security tests. Test automation supports manual testing, but cannot replace exploratory testing or the technique of error guessing, wherein the tester’s knowledge in the context of the product and tools is essential.

3.      Automated tests help to catch a lot of defects

Contrary to popular opinion, automated tests will not help to detect a lot of defects. One can ask: “How can that be? Isn’t that what we run them for?” Of course, it is one of the goals we are trying to attain; however, automated tests are quality-oriented, not quantity-oriented. To give an example, automated tests are useful in catching functionalities’ change-related defects in the event that software code has been amended. These are called regression tests and are used to ensure that changes implemented to either software, configuration or the environment will not impact the developed product itself. Automated regression tests can successfully replace manual testing carried out by entire development teams, which – depending on the software’s complexity – would normally take hours, days or even weeks. Our trust increases with every single automated test. Even though they do not catch a lot of defects, they allow us to save time and money, enabling us to focus on the quality of the product instead.

4.      Each automatable case should be automated

Knowing that not all tests can be automated and that there are areas in which we will still need manual testing – is it worth automating every automatable testing case?

There are situations in which swift software testing is required and engaging additional resources, tools or funds (which is often related to test automation) is not an option. Let’s paraphrase a quote by Roger Needham, Microsoft’s Research Laboratory Director, to clear up doubts: automation is a process of replacing something that works with something that almost works, but is quicker and cheaper.

This reminds us of the two principal ways of defining whether we should automate tests or not:

  • automated tests should be faster than manual verification
  • the cost of programming tests cannot exceed the income gained thanks to test automation

From a business perspective, the cost of tools and time needed for the implementation of automated tests in comparison with manual testing should be the decisive factor. This should facilitate the decision-making process in terms of automating testing cases.

Summary

Despite many myths related to test automation, automated tests are gaining greater understanding and popularity. Myths related to automated tests keep coming up, which proves that a lot still needs to be done to clarify doubts. I hope that I have managed to clear up some of them and that the article will help to define further discussion on automated tests. This is priceless help for testers in the process of delivering a high quality product.

Daria Bajda-Czyżyńska

Daria Bajda-Czyżyńska

Senior QA Engineer, Technical Leader

Technical leader and software tester, specializes in automated testing. Her adventure with testing began quite unexpectedly – on the production line of the Slovak car factory. Then she entered the world of embedded software testing and fell in love with the diverse world of mobile and web applications.