Software Testing is still more in realm of ‘Arts’ than Science, especially when we are talking about testing software products, something like MS-Office or Mercury WinRunner. Unless you are a part of a highly evolved and organized form of testing, in other words, a defined set of test cases on a defined set of test-bed to be executed, you can safely assume that you wont be able to find all the bugs. Being paid to find bugs and still not able to find all is an oxymoron arrangement and as Test Engineers and Test Managers, we need to live in the reality (of not being able to find all bugs) and still do business (find all bugs).
In this post, I am going to talk about a exercise which lots of organizations are started to get into their test-process as an effective tool for finding bugs, which typically act elusive to the ‘Test Engineer’ who is looking at a specific area. This exercise is called by different names like Bug Hunt, Bug Bash, Focus Day, Bug Harvest and so on. We would refer it as ‘Bug Hunt’, just a personal preference, they all work same / similar.
What is a Bug Hunt ?
A Bug Hunt is an intense bug finding mission over a small period of time, from couple of hours to a day, done by people who are not testing that area or that product as normal day-to-day work or may not be ‘testers’ at all, allows all kinds of bugs (some bugs are feature-requests) to be reported and is done to sort of give one big shake to an area or a product before its being shipped.
Identify a functionality area which needs to be hunted for bugs. Ideally this functionality area should be devoid of routine bugs and must have gone through 2-4 rounds of bug finding and fixing. There is no point in getting an obvious bug from 10 different people. Once you identify the area then we need to evaluate on how many hours of testing would this area benefit from. There could be cases where you might want to test a feature like CD Writing and there the goals could be to test it on as many test-beds as possible. So choose the test coverage criteria like no of hours, test-beds, x no of fresh pair of eyes etc . After that choose the audience. An e-mail program could possible be bug-hunted by any one but a complex image processing program may need a specific set of skills to really test and find bugs. After identifying the test coverage needs (either in terms of time or test-beds or just plain no of people), and the intended audience, choose an apropriate time window and invite folks.
In certain cases, I have seen that getting a set of 12 odd people in a test-lab has proven to be much more effective. In such a set-up bug filtering can happen right at the time of bug-finding, since you can seek an instant verbal feedback. It may not be possible in all set-ups to have the hunters do it at one place so take your pick.
While the bug-hunting is on, few people should be around to help folks answer queries, so that they find better bugs. They can find better bugs only if they truly understand the feature so bug-hunt-owner needs to be around to hand-hold.
All and any bugs are allowed, so dont judge before they get reported. After all the bugs have been reported, do a bug scrubbing to remove the unwanted ones and duplicates. Also move the ‘Feature Requests’ to a different place. Now you have the ones which are true bugs and thats your harvest.
Cookies and Awards
To make this whole thing successful, you would need to identify a right tool to motivate people. Usually a recognition like ‘Great Catch’ or ‘Most No of Valid Bugs Reported’ works better, cookies while the hunt is on is definitely a plus.
There are minor versions of this hunt as well where you may want to do it with a small no of focused people, say 3 people spending 2 hours on a small area. These can be called ‘Mini Bug Hunts‘. The usefulness of these bug hunts is not only in terms of bug-reporting but also a big re-assurance which you can tell your self that if x no of people could use the software for a couple of hours without bumping into anything big then probably we are close to shipping, or going home by six.