What is our automation test coverage?

- What is our automation test coverage?
- 42
- What is “42”?
- What is automation test coverage?

Asking about test coverage you have to be ready to have such conversation. In general the question about test coverage is very slippery. The person who is asking it has to realize all issues related to testing:

Testing can be endless if it is not limited. It can be limited by time frame (Session based exploratory testing) or by cutting the scope (scripted testing). You can cut the scope by limiting the number of features to be tested or by using test design techniques.

Due to the fact that we cannot automate exploratory testing the definition of automation test coverage will be treated to the scripted testing and it will show the percentage of automated manual tests.

Let’s look at one small example:

There is a simple application consists of a couple of pages, forms, linked between each other entities, functions and so on (I'm not talking about infrastructure). To perform testing very close to the full (exhausting) testing we will need days, weeks, months.

Cut the scope of testing by using the well-known test design techniques (equivalent partition, boundary value analysis, state flow control and etc.) leaving only the most critical data sets and scenarios. Having this we are getting coverage on the level of "a drop of water in the sea". Thereby on the question about automation test coverage you can answer that we have for example 50% of all automated manual tests what corresponds to undefined percentage from the full test coverage (which we are not going to do). In the result it means that our automation test coverage equals to “42”.

In case if you limit testing by time frame where you don’t have manual tests the answer “42” will also be valid since the scope of performing testing anyway infinitesimal relative to the scope of full testing.