The million dollar question which every Manager asks to testing teams is that ‘Whether they have executed all the test cases ? “. If you are a software test practitioner, you would be feeling really frustrated. How on earth, can you ever execute all the ‘Test Cases’ for a particular feature.
Well, there is no silver bullet but there is a way which you can use to get a good enough answer for the above question. Its called ‘Code Coverage’ analysis.
How does it work
There are certain software tools in the market which can be used to do ‘Code Coverage’. One of the popular software is ‘Bullseye’. The process is simple. While building binaries for your software program, configure ‘Bullseye’ with the IDE. When binaries are made, bullseye inserts some of its own signatures. This is called ‘Instrumentation’ of the code.
Once a binary has been instrumented, just exercise the features of the program. As you use the program, a ‘coverage’ file gets created and updated. This ‘coverage’ file has the information on the lines of code which are getting executed. After you are done, analyze this coverage file using Bullseye and you would get details on ‘how much code has been covered’. So if you run your whole test-suite then you would get the coverage for that test-suite.
How to interpret results
If you have a 100% coverage then something is wrong because then you are doing something which is so perfect. For a real life full blown software application, 100 % is not possible and not desirable as well. Code coverage reports helps you in two ways.
1. It helps you to find the ‘dead code’, the code which is never executed.
2. It helps you to find the areas within the code for which no testcase has been executed.
To keep it short, I would stop here but feel free to ask anything by writing a comment.
Nice article !!!
I have not used “Bullseye” tool but I did used Java Junit code coverage tool integrated with WSAD (Websphere Application Development IDE) sometime back. It is a good code coverage testing tool.
When integrated with build script you can create cruise control dashboard . You can track unit test code coverage for each java package in the code. It also points to the missing and dead code. I really liked it and just wanted to share my experience.
Thanks Manish.
Would you want to write a guest article about ‘Junit CC tool’ which you mentioned. It would be useful to fellow readers.
Thanks
You have to know how to write code in order to use this tool, right?
No you do not need to write code to use this tool, but knowing about various structures (loops, switches, decision statements etc) of a programming language helps you in analysis of results.
Dear Nandan,
Thanks for the update on the Bulls eye code coverage. I need a clarification wrt bullseye CC tool for one of the scenario. I have done the testing for one of executable which is code coverage enabled and even the .cov file is also updated with the necessary data about my testing. But while testing i come across a bug and a patch will be given by the respective team as a fix to the cosmetic bug. In this scenario the executable is again tested and found working fine.
I like to know in the initial round of testing by bullseye testing statistics i came to know we have covered 90% of testing. During the 2nd round testing even though i have tested for only the cosmetic bug the test coverage tool will give only 1% testing instead of 91%.Here i like to know whether there is any mechanism of getting the previous result getting appended with the recent one tested scenario.
Please let me know.
@ Hari – If I am able to follow it right then you are looking at a way to merge two .cov files from two separate test runs. Yes, this is possible. I have not used the tool for a long but look for an executable (search *covmerge* ) within the CC installations and then one can pass on parameters like exename.exe -c finalmergedfile.cov testrun1.cov testrun2.cov
If you are able to do this, then requesting you to post the solution here so that other readers can benefit. Wishes. – Nandan
@Nandan, Is their any possibility to merge two .cov fiiles genarated out of two separate test runs(of diffrent builds).
@ Manju – Yes, that is what I remember. One can merge the two .cov file, even though they are on two separate builds. I am also sending the link to a colleague who has done more work here and can advice better.
Here is my understanding. It’s possible to merge the 2 cov files from different builds but may not be very useful. After merging the latest coverage is retained and you get the latest coverage rather than the merged.
Using the same source version:
First coverage file shows function A has 30% coverage and function B has 50% coverage.
The second coverage file shows function B has 75% and function C has 80% coverage. Then the merged file should show: A-30%, B-75%, C-80%.
Using different source version: In the next build if function B is changed whereas A and C are not changed and we run the Code coverage again to create a new cov file.
The first cov file from build#1 shows function A has 30% coverage and function B has 50% coverage .
The second cov file from build #2 shows function A has 0% coverage, function B has 25% coverage and C has 80% coverage. Here the merged coverage report would show the coverage for the recent version of the source. Hence the merged cov file would have: A-0%, B-25% , C-80%